Commit b6d20c7d authored by Jean-Loup Haberbusch's avatar Jean-Loup Haberbusch
Browse files

Update Testing-Framework-Whitepaper.md

parent 5d90a7f2
Pipeline #17999 passed with stages
in 26 seconds
......@@ -12,6 +12,7 @@ CamiTK provides some tools to test your code at different granularity levels. In
Testing a software code is always a good practice to make sure that the code behave accordingly to the specification, to detect bugs and errors and regression. It can also be very helpful to check that code integration is operational.
Several types of tests exist depending on the granularity level of the tests. There are classically three categories of tests:
- unit testing
- functionnal testing
- integration testing
......@@ -31,6 +32,7 @@ Please refer to [the CamiTK automatic tests framework](https://camitk.gricad-pag
## Cross-platform testing C++ code using CMake
To create any type of test (unit, functional ...), three basic steps are always followed:
- Implement the C++ code that will test your C++ class method / application (you may use some dedicated tools like Qt testing framework or any other cross-platform testing framework)
- Configure your project to create some cross-platform test targets that run the tests on all the platforms.
- Check the test results.
......@@ -38,6 +40,7 @@ To create any type of test (unit, functional ...), three basic steps are always
### C++ code to test your class / program
Assuming we have some C++ code to test gathered In a library (or extension), we have to basically create a **C++ application** (using a specific framework or a home made code) in order to:
- Load the library containing the C++ classes to tests.
- Do some tests on the class methods
- Show the test results (on standard output or in a file)
......@@ -52,10 +55,12 @@ Once the test C++ application is built, it should be ran each time a modificatio
CMake comes with a specific module to add, configure, and run tests. Using CMake makes it possible to define and
configure tests once for all platforms. To define and configure tests using CMake, two specific CMake macros should be used:
- `enable_testing()` to enable the testing functionality on our project
- `add_test(...)` to create a new test using the testing application
More specifically:
- The test executions are done using [CTest](https://cmake.org/Wiki/CMake/Testing_With_CTest), which manages which tests are run
- The test results can be sent to a dashboard using [CDash](http://www.cdash.org/)
......@@ -89,6 +94,7 @@ system using gitlab. They are called “automatic” tests as they do not focus
### Integration testing
In the CamiTK framework, we consider that integration testing consists of:
- **Checking that CamiTK extensions can be loaded**, can work together and can be used in a specific context. At the moment, this relies on the `camitk-config` application. The test uses a bash script to check the CamiTK Community Edition version number and verify that the expected number of extension for this version can be found and loaded in the application.
- **Checking that the code generator can automatically creates valid C++ code, that can be configured, built and integrated in a specific context (mimicking the CEP developer context)**. At the moment, this relies on the `camitk-cepgenerator` application. The test uses a bash script and starts from document written in the `cepcoreschema` XML language to apply the whole build chain: generate the code using the `cepgenerator` library and the `camitk-cepgenerator` application, configure the code using CMake and the CamiTK macros, build the code using the specific compiler available on the platform, and run the `camitk-config` application to check that the extensions can be loaded and available in any CamiTK application
- **Checking that different actions can be executed in a pipeline**. At the moment, this relies on the `camitk-actionstatemachine` application. The developer can create action pipelines writing an action state machine XML description language. During the test phase, the pipeline is ran automatically and the output files (i.e., the component that are saved to file during the pipeline execution) can be compared to expected files in order to verify that everything went according to plan.
......@@ -106,6 +112,7 @@ Not only the test results are visible, but CamiTK also offers a way to check the
## CamiTK testing framework
The CamiTK testing framework is defined As a layer on top of CMake/CTest, and provides _five_ different type of tests:
1. Automatic test for action and component extensions
2. Automatic test using a pipeline of actions
3. A set of specific CMake macros that help defining custom tests and test coverage
......@@ -121,6 +128,7 @@ The following sections describes these five different types of tests.
As CamiTK defines an API for action and component extensions, this API can be used to check the default/expected basic behaviour of component and action extensions. In order to do so, the CamiTK testing framework
provide two extra C++ applications: `camitk-testactions` and `camitk-testcomponents`:
- `camitk-testactions` provides one level of automatic test for a given action extension.
- `camitk-testcomponents` offers three different levels of automatic test for a given component extension.
......@@ -131,6 +139,7 @@ _additional component extension automatic tests_).
#### Testing action extensions
To test each action provided in the CamiTK framework, we insure that calling it on the proper components it has been defined on, works correctly. The `camitk-testactions` application has five main steps:
1. Load all the available CamiTK components the action can work on
2. Load the action extension itself
3. Open a component file and set it as the action input component
......@@ -139,6 +148,7 @@ To test each action provided in the CamiTK framework, we insure that calling it
The test passes if the `apply()` methods returns `Action::ApplyStatus::SUCCESS` or `Action::ApplyStatus::ABORT` (in this case it the developer responsability ot check that the `ABORT` status is _really_ what is expected). The test fails otherwise.
For technical details and implementation, please check:
- `CamiTKExtension.cmake` (in `sdk/cmake/modules/macros/camitk`) the script that creates the tests thanks to the `camitk_init_test` and `camitk_add_test` macro (resp. defined in `CamiTKInitTest.cmake` and `CamiTKAddTest.cmake` in `sdk/cmake/modules/macros/camitk/test`, see [the dedicated section below](#set-of-specific-cmake-macros-for-testing))
- the C++ source code for `testactions` (in `sdk/applications/testactions`)
......@@ -147,6 +157,7 @@ For example, check any actions in the SDK.
#### Testing component extensions
To test each component in the CamiTK framework, we insure that **opening** a file associated with this component and (when possible) **saving** it work. To do so, the `action-testcomponents` application goes trough four steps for each available (or specified) files in the component’s `testdata` subdirectory:
1. Load the component extension
2. Open the given file (test level 1)
3. Save the file using a different name (optional test level 2)
......@@ -155,6 +166,7 @@ To test each component in the CamiTK framework, we insure that **opening** a fil
The level of the test depends on the `AUTO_TEST_LEVEL` option specified in the `camitk_extension` macro.
For technical details and implementation, please check:
- `CamiTKExtension.cmake` (in `sdk/cmake/modules/macros/camitk`) the script that creates the tests thanks to the `camitk_init_test` and `camitk_add_test` macro (resp. defined in `CamiTKInitTest.cmake` and `CamiTKAddTest.cmake` in `sdk/cmake/modules/macros/camitk/test`, see [the dedicated section below](#set-of-specific-cmake-macros-for-testing))
- the C++ source code for `testactions` (in `sdk/applications/testactions`)
......@@ -173,6 +185,7 @@ Any file saved during the pipeline execution is going to be compared to the expe
In order to automatically add a test, you need to add the following line at the end of the action or component extension `CMakeLists.txt`:
This process is generalized thanks to some normalization: all the required following files should be available in a `integration-testdata` subdirectory:
- An CamiTK SCXML document `input-asm.scxml`
- Any number of `input-#.xxx` input files
- Any number of `output-#.xxx` output files (this are the expected output files)
......@@ -194,6 +207,7 @@ TODO : use this for memory leak check (and performance check?)
### Set of specific CMake macros for testing
CamiTK testing framework provides three specific CMake macros in order to more efficiently configure tests inside the CamiTK environment:
- `camitk_init_test(...)` helps to initialize all the common CMake configuration for all the test of the current CMake project.
- `camitk_add_test(...)` adds a new test for the current CamiTK project. This macro encapsulates the default `add_test(...)` function provided by CMake.
- `setup_target_for_coverage(...)`, entirely based on [Lars Bilke macro](https://github.com/bilke/cmake/modules/blob/master/CodeCoverage.cmake), simplifies the generation of test coverage report.
......@@ -213,6 +227,7 @@ The CamiTK Community Edition project tests if the valid number of expected actio
Once installed, CamiTK should be ready to be used to develop a CEP.
The CamiTK Community Edition project tests that the CamiTK development framework is ready to use doing the following steps (all described in bash script, see `sdk/applications/cepgenerator/testing/cepgenerator-test.sh`):
1. Test that the C++ code generation from a cep core schema XML is possible: this tests the availability of the `camitk-cepgenerator` application.
2. Test that the generated code can be configured: this tests that the CamiTK CMake macros and environment is correctly installed and accessible
3. Test that the generated code can be compiled: this tests that the core library and other dependencies are available and linkable with all the C++ compilation tool chain
......@@ -227,6 +242,7 @@ You can also create your own example from a tweaked core schema XML if your CEP
You can easily generate and check a test coverage report for the CamiTK Community Edition or your own extension.
There are two requirements to generate the test coverage report:
1. you need to use the GNU cxx compiler
2. you need to have `gcov` and `lcov` installed (`apt-get install` is your friend\!)
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment