mightymandel v16

GPU-based Mandelbrot set explorer
TESTING

Test Suite

The test suite renders all the examples. There is a timeout of 5m for each file. Not all examples render correctly at this time, this file keeps track of the current status. To run the test suite: 

make -C src test

After the test is run you will have a directory with all the images rendered, plus a CSV file containing detailed test results.

To exclude files from testing (perhaps known timeouts or other failures), list them in test/EXCLUDE.md.

The test suite needs these programs in addition to those for building:

  • find
  • grep
  • xargs
  • sort
  • timeout
  • pnmtopng
  • tee
  • ts

You might want to disable new windows getting focus so you can do something else while the test suite is in progress. For XFCE4 window manager the option is:

  • Menu
    • Settings
      • Window Manager
        • Focus tab
          • New window focus section
            • Automatically give focus to newly created windows checkbox

Corruption

Some images are corrupt, with undetected glitches or wrong appearance, etc. The only way to check is by visual inspection. After running the test, create a subdirectory 'corrupt' in the test output directory and move all the corrupt images to it. Then fix the test output CSV by doing something like this (inside test/): 

version=v14-135-g8a2fc7d
ls -1 "${version}/corrupt/" |
sed 's/.png$//' |
while read file
do
  sed -i "s|^${file},ok,|${file},corrupt,|" "${version}.csv"
done

Summarizing

Summary results for inserting into the results table can be gathered like this: 

version=v14-135-g8a2fc7d
for result in ok timeout glitch corrupt parse deep crash excluded
do
  grep -F ",${result}," "${version}.csv" | wc -l
done

Results

Detailed results are in archived in test/.

version pass time taken tests ok timeout glitch corrupt parse deep crash excluded
v14 81% 27m28.262s 119 96 5 14 4 0 0
v14-unknown 75% 92m59.418s 527 142 11 0 27 333 9 5
v14-125-g452a227 79% 133m10.896s 577 192 15 2 33 333 2 0 0
v14-132-gba03239 82% 57m47.065s 586 191 0 2 30 0 2 9 352
v14-135-g8a2fc7d 80% 60m16.515s 586 188 0 2 33 2 2 11 352
v14-164-g9cd6fc7 91% 77m53.602s 587 231 8 1 1 333 4 9 0
v14-199-g3b65956 94% 75m02.306s 589 241 6 0 4 333 0 0 3
v15-23-g929b6ba 95% 74m28.954s 590 245 7 0 4 333 0 0 1
v15-29-g9b25aa0 99% 30m47.903s 590 132 1 0 0 200 0 0 257
v15-31-g5cbe332 95% 21m26.985s 790 190 1 0 2 7 0 0 590
v15-38-gc34ab52 98% 106m16.045s 790 564 6 0 3 0 0 0 217
v15-181-gdcf4582 86% 43m03.129s 791 157 3 22 0 0 0 0 609
v15-182-g635bd87 99% 40m43.350s 791 180 2 0 0 0 0 0 609
v15-183-g583e89e 99% 93m01.208s 791 572 3 0 0 0 0 0 216
v15-202-gc979b39 99% 98m36.565s 791 573 3 0 0 0 0 0 215

Columns:

  • version: the version of mightymandel tested.
  • pass: the overall test pass rate.
  • time taken: the wall clock real time taken by make test.
  • tests: total number of parameter files tested.
  • ok: The rendering completed and the image is fine.
  • timeout: Rendering didn't complete within the time limit.
  • glitch: The image has an unacceptably high proportion of detected glitches.
  • corrupt: The image is bad: undetected glitches, wrong appearance, other.
  • parse: The parameter file failed to parse.
  • deep: the parameter is too deep.
  • crash: mightymandel crashed
  • excluded: not tested

Notes:

  • v14
    • unknown number of tests were excluded
  • v14-unknown
    • pass percentage is excluding both parse failures and excluded tests
    • time taken if timeouts were excluded would be 37m
  • v14-125-g452a227
    • pass percentage is excluding parse failures
    • time taken if timeouts were excluded would be 58m
  • v14-132-gba03239
    • pass percentage is excluding excluded tests
    • unfortunately the detailed log is missing due to programmer error
  • v14-135-g8a2fc7d
    • pass percentage is excluding excluded tests
    • only significant change was adding randomization to the glitch correction
    • two of the crashes were segfaults, the others were assertion failures
    • results may not be valid as the test might have been run without making sure to clean up after an aborted test run first
  • v14-164-g9cd6fc7
    • pass percentage is excluding parse failures
  • v14-199-g3b65956
    • pass percentage is excluding parse failures
    • v15 has no rendering engine changes since this version
  • v15-23-g929b6ba
    • pass percentage is excluding parse failures
    • renderer increased FP___STEP_ITERS
  • v15-29-g9b25aa0
    • pass rate is excluding parse failures and excluded tests
    • parser added for .ppar center-mag variant (the purpose of this test)
    • some images are off (because of unsupported stretch/rotation/skew)
    • note that some of the reference images are for different locations at http://reocities.com/CapeCanaveral/5003/par.htm
  • v15-31-g5cbe332
    • pass rate is excluding excluded tests
    • parser added for FractInt .gif (the purpose of this test)
    • some images are off (because of unsupported stretch/rotation/skew)
    • all parse failures were due to view radius being 0 in double precision
      • the parser doesn't yet handle any additional high precision chunks
    • the timeout failure was due to the view being 100% interior
    • the corrupt failures are multiple copies of the same image
      • possibly lack of precision to blame for the view being 100% far exterior
    • see examples/gif/README for instructions to obtain the source images
  • v15-38-gc34ab52
    • pass rate is excluding excluded tests (see Excluded Tests, mostly ppar parsing)
    • corrupt images are near-blank, possibly due to limited precision parsing
  • v15-181-gdcf4582
    • excluded tests include Excluded Tests, plus all gif and ppar
    • significant renderer changes in glitch correction reference finding (two pass blob extraction)
    • all glitched results have invisible glitches
  • v15-182-g635bd87
    • as previous test, but with –max-blob 4 instead of the default –max-blob 1
    • no visible glitches
  • v15-183-g583e89e
    • as previous test, but gif and ppar included again
  • v15-202-gc979b39
    • v16 has no rendering engine changes since this version

Reference Images

Reference images can be found online here:

Of particular note: