Commit 95d5eeae authored by Emmanuel Promayon's avatar Emmanuel Promayon
Browse files

Update docs/Highlighted Features/Automatic-Tests.md

parent 2a93dc1d
Pipeline #20058 passed with stages
in 26 seconds
......@@ -392,11 +392,87 @@ Contact us if you would like more information about this and some support and he
## Disabling tests
_Since CamiTK 4.2_
In some specific cases, some automatic tests have to be disabled, for instance when a test depends on a specific platform or OS or version of a library and you know, _by design_ that it can only fail if some environment requirement are not met.
There is two ways to disable some of the automatically generated tests:
- disable test unconditionnally
- set specific requirements to run tests (if the requirements are not met, the tests will be disabled)
### Unconditionnally disable some tests
In order to unconditionnally disable some tests, use the `camitk_disable_tests` CamiTK CMake macro. For instance, in an action `CMakeLists.txt`, you can do:
```cmake
camitk_extension(ACTION_EXTENSION
...
ENABLE_AUTO_TEST
...
)
...
# Specific test management
camitk_disable_tests(TESTS action-myaction-level1-10
REASON "This test is expected to fail until such and such is implemented. Disable for now"
)
```
This will disable test named `action-myaction-level1-10` (this test was automatically generated by the CamiTK test framework). You can of course use this macro for any test even tests that you added manually.
Basically, `camitk_disable_tests` macro encapsulates CMake `set_tests_properties` in order to manage version older that CMake 3.9 and to force you to give a (good) reason for bypassing the corresponding tests.
If CMake version is lower than 3.9, the test(s) is/are executed but with the `WILL_FAIL` (i.e., a failed test is considered as passed).
You can list as many tests as required in the `TESTS` argument.
### Define specific requirement for some tests
Some of your tests (manually generated or automatically generated by the CamiTK test framework) might need to be disabled due to the environment/testbed (e.g., if a test can only pass for a specific version of VTK, or if not run on a virtual desktop...). Use the `camitk_tests_requirement` macro to specify the requirement of specific tests. For instance:
```cmake
camitk_extension(ACTION_EXTENSION
...
ENABLE_AUTO_TEST
...
)
...
# Specific test management
# disable one test on Win32 platform (this is a two-parts requirement statement)
camitk_tests_requirement(TESTS action-itkfilters-integration-test
REQUIRES "NOT WIN32"
REASON "WIN32 OpenGL failure on a VM
This test will always fail when run inside a VM due to OpenGL crash.
This test will pass when run directly from a physical windows machine."
)
# disable three tests if VTK version is not a specific version (this is a three-parts requirement statement)
camitk_tests_requirement(TESTS action-myaction-level1-3 action-myaction-level1-6 action-myaction-level1-11
REQUIRES "${VTK_VERSION} VERSION_EQUAL 6.3"
REASON "VTK version is not equals than 6.3
The default test files are written using the currently supported version of VTK (VTK 6.3).
This test will therefore fail when comparing the input to the output if another version of VTK is used."
)
```
`camitk_tests_requirement` will disable one or more tests if a specific requirement is not met.
It allows for specific test management in case the developper knows that a given test requires a given environment or can not be run in a given environment.
It encapsulates CMake `set_tests_properties`, allows for specifying a requirement and force you to give a (good) reason for bypassing the tests.
The requirement statement is a string that looks like a CMake test statement. It defines a subset of possible test that can check the current environment (OS, library/dependencies version...etc...)
If the requirements are not met, the listed tests are disabled.
`REQUIRES` statement looks like a CMake test statement but is restricted to a subset of CMake `if` statement. It can have one, two or three parts.
There is three types of `REQUIRES` statement:
- one-part statement are used to check the OS. Possible values are (as of Feb 2019) `"WIN32"`, `"WIN64"`, `"MSVC"`, `"APPLE"` or `"UNIX"`
- two-parts statement are used to check the opposite of the one-part statement using the `NOT` prefix. In the example given above, the test `action-itkfilters-integration-test` is ran only if the requirement `"NOT WIN32"` is met (e.g., if the platform is not `WIN32`). Possible values are therefore (as of Feb 2019) `"NOT WIN32"`, `"NOT WIN64"`, `"NOT MSVC"`, `"NOT APPLE"` or `"NOT UNIX"`
- three-parts statement are used to check the version of a library/dependency. It is composed of a left value (that is evaluated by the caller CMakeList, not inside the macro) followed by a test statement and a right value (also evaluated by the caller `CMakeList.txt`). Possible test statements are (as of Feb 2019) `VERSION_LESS`, `VERSION_EQUAL` or `VERSION_GREATER`. In the example above, tests `action-myaction-level1-3` `action-myaction-level1-6` `action-myaction-level1-11` are ran only if the variable `VTK_VERSION` (as set at configuration time depending of what version of VTK is detected) is equals to `6.3` (`VERSION_EQUAL` is a CMake test that is aware of major, minor and patch version, in this case the requirement is met for VTK version 6.3.x, whatever the value of x).
<!--
OLD Documentation: it is still in place, but is it worth describing?
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment