@@ -392,11 +392,87 @@ Contact us if you would like more information about this and some support and he
## Disabling tests
_Since CamiTK 4.2_
In some specific cases, some automatic tests have to be disabled, for instance when a test depends on a specific platform or OS or version of a library and you know, _by design_ that it can only fail if some environment requirement are not met.
There is two ways to disable some of the automatically generated tests:
- disable test unconditionnally
- set specific requirements to run tests (if the requirements are not met, the tests will be disabled)
### Unconditionnally disable some tests
In order to unconditionnally disable some tests, use the `camitk_disable_tests` CamiTK CMake macro. For instance, in an action `CMakeLists.txt`, you can do:
REASON "This test is expected to fail until such and such is implemented. Disable for now"
)
```
This will disable test named `action-myaction-level1-10` (this test was automatically generated by the CamiTK test framework). You can of course use this macro for any test even tests that you added manually.
Basically, `camitk_disable_tests` macro encapsulates CMake `set_tests_properties` in order to manage version older that CMake 3.9 and to force you to give a (good) reason for bypassing the corresponding tests.
If CMake version is lower than 3.9, the test(s) is/are executed but with the `WILL_FAIL` (i.e., a failed test is considered as passed).
You can list as many tests as required in the `TESTS` argument.
### Define specific requirement for some tests
Some of your tests (manually generated or automatically generated by the CamiTK test framework) might need to be disabled due to the environment/testbed (e.g., if a test can only pass for a specific version of VTK, or if not run on a virtual desktop...). Use the `camitk_tests_requirement` macro to specify the requirement of specific tests. For instance:
```cmake
camitk_extension(ACTION_EXTENSION
...
ENABLE_AUTO_TEST
...
)
...
# Specific test management
# disable one test on Win32 platform (this is a two-parts requirement statement)
The default test files are written using the currently supported version of VTK (VTK 6.3).
This test will therefore fail when comparing the input to the output if another version of VTK is used."
)
```
`camitk_tests_requirement` will disable one or more tests if a specific requirement is not met.
It allows for specific test management in case the developper knows that a given test requires a given environment or can not be run in a given environment.
It encapsulates CMake `set_tests_properties`, allows for specifying a requirement and force you to give a (good) reason for bypassing the tests.
The requirement statement is a string that looks like a CMake test statement. It defines a subset of possible test that can check the current environment (OS, library/dependencies version...etc...)
If the requirements are not met, the listed tests are disabled.
`REQUIRES` statement looks like a CMake test statement but is restricted to a subset of CMake `if` statement. It can have one, two or three parts.
There is three types of `REQUIRES` statement:
- one-part statement are used to check the OS. Possible values are (as of Feb 2019) `"WIN32"`, `"WIN64"`, `"MSVC"`, `"APPLE"` or `"UNIX"`
- two-parts statement are used to check the opposite of the one-part statement using the `NOT` prefix. In the example given above, the test `action-itkfilters-integration-test` is ran only if the requirement `"NOT WIN32"` is met (e.g., if the platform is not `WIN32`). Possible values are therefore (as of Feb 2019) `"NOT WIN32"`, `"NOT WIN64"`, `"NOT MSVC"`, `"NOT APPLE"` or `"NOT UNIX"`
- three-parts statement are used to check the version of a library/dependency. It is composed of a left value (that is evaluated by the caller CMakeList, not inside the macro) followed by a test statement and a right value (also evaluated by the caller `CMakeList.txt`). Possible test statements are (as of Feb 2019) `VERSION_LESS`, `VERSION_EQUAL` or `VERSION_GREATER`. In the example above, tests `action-myaction-level1-3``action-myaction-level1-6``action-myaction-level1-11` are ran only if the variable `VTK_VERSION` (as set at configuration time depending of what version of VTK is detected) is equals to `6.3` (`VERSION_EQUAL` is a CMake test that is aware of major, minor and patch version, in this case the requirement is met for VTK version 6.3.x, whatever the value of x).
<!--
OLD Documentation: it is still in place, but is it worth describing?
