Technical aspects

Structure

A simulator structure contains a tests subfolder. To test a simulator, add .IN and .REF directories inside tests folder:

.IN directories are OpenFLUID datasets.
.REF directories are OpenFLUID reference outputs that are expected to be generated by the dataset, they are not mandatory but useful to check value regressions.

Example:

SimulatorA
├─ README.md
├─ openfluid-ware.json
├─ CMakeLists.txt
├─ doc/
├─ src/
│  ├─ WareMain.cpp
|  └─ CMakeLists.txt
└─ tests/
   ├─ test1.IN
   ├─ test1.REF
   ├─ test2.IN
   ├─ test2.REF
   ├─ test3.IN
   └─ CMakeLists.txt

Command

The following OpenFLUID command in tests/CMakeLists.txt will search for all datasets inside tests subdirectory.
For each dataset, a test is added, which will run a simulation with corresponding test dataset.

OPENFLUID_ADD_WARETESTS(DISCOVER)

After configuration and build, running ctest command in the build directory or clicking on the "run test" button of DevStudio will run all tests.

A simulation test output is located in the build directory at path tests-output/<test-name>.OUT.
If a reference is found for a dataset, the test will also compare output and reference directories.

If we refer to the given structure of SimulatorA above, 3 tests will be created:

dataset-test1-with-ref, will run the simulation based on test1 dataset with OUT data comparison to REF folder content
dataset-test2-with-ref, as above for test2
dataset-test3-no-ref, will run the simulation based on test3 dataset without data comparison since there is no ref folder for that dataset

Best practice

Reference data

We highly recommand to provide a reference output for all datasets, otherwise the test will only detect a fraction of potential impact caused by a change in source code, missing most of quantitative change.

This REF folder can be built by copying the OUT folder produced by a test run at a given time (considered as valid) and renaming it as dataset.REF.

When the comparison with REF fails based on value difference, it is crucial to pinpoint the underlying root. If it is a legitimate value change, REF should be regenerated based on the new OUT.

Stochasticity

For simulators with stochasticity, we advise to set a seed (cf Usage of random in ware : setup and example) to be able to compare outputs.

Focus on CTest

ctest is a CMake-related program to run tests (cf official page).

Several options are available with ctest, we will here only speak about three of them:

-R <string> for test name filtering: ctest -R with-ref will run any test containing the string with-ref (so test1 and test2 in our case)
-V/--output-on-failure for test verbosity: ctest -V will display the test output in shell, --output-on-failure will only display the test output if it fails (used in DevStudio test run)
-j <number> for parallel jobs: ctest -j 4 will run up to 4 tests in parallel (depending on system capacities)