Commit 8372bbd7 authored by Jean-Loup Haberbusch's avatar Jean-Loup Haberbusch
Browse files

Merge branch 'master' of gricad-gitlab.univ-grenoble-alpes.fr:CamiTK/docs

parents ad024f3d 203661cb
Pipeline #19339 passed with stages
in 33 seconds
......@@ -12,6 +12,7 @@ CamiTK provides some tools to test your code at different granularity levels. In
Testing a software code is always a good practice to make sure that the code behave accordingly to the specification, to detect bugs and errors and regression. It can also be very helpful to check that code integration is operational.
Several types of tests exist depending on the granularity level of the tests. There are classically three categories of tests:
- unit testing
- functionnal testing
- integration testing
......@@ -31,6 +32,7 @@ Please refer to [the CamiTK automatic tests framework](https://camitk.gricad-pag
## Cross-platform testing C++ code using CMake
To create any type of test (unit, functional ...), three basic steps are always followed:
- Implement the C++ code that will test your C++ class method / application (you may use some dedicated tools like Qt testing framework or any other cross-platform testing framework)
- Configure your project to create some cross-platform test targets that run the tests on all the platforms.
- Check the test results.
......@@ -38,6 +40,7 @@ To create any type of test (unit, functional ...), three basic steps are always
### C++ code to test your class / program
Assuming we have some C++ code to test gathered In a library (or extension), we have to basically create a **C++ application** (using a specific framework or a home made code) in order to:
- Load the library containing the C++ classes to tests.
- Do some tests on the class methods
- Show the test results (on standard output or in a file)
......@@ -52,10 +55,12 @@ Once the test C++ application is built, it should be ran each time a modificatio
CMake comes with a specific module to add, configure, and run tests. Using CMake makes it possible to define and
configure tests once for all platforms. To define and configure tests using CMake, two specific CMake macros should be used:
- `enable_testing()` to enable the testing functionality on our project
- `add_test(...)` to create a new test using the testing application
More specifically:
- The test executions are done using [CTest](https://cmake.org/Wiki/CMake/Testing_With_CTest), which manages which tests are run
- The test results can be sent to a dashboard using [CDash](http://www.cdash.org/)
......@@ -89,6 +94,7 @@ system using gitlab. They are called “automatic” tests as they do not focus
### Integration testing
In the CamiTK framework, we consider that integration testing consists of:
- **Checking that CamiTK extensions can be loaded**, can work together and can be used in a specific context. At the moment, this relies on the `camitk-config` application. The test uses a bash script to check the CamiTK Community Edition version number and verify that the expected number of extension for this version can be found and loaded in the application.
- **Checking that the code generator can automatically creates valid C++ code, that can be configured, built and integrated in a specific context (mimicking the CEP developer context)**. At the moment, this relies on the `camitk-cepgenerator` application. The test uses a bash script and starts from document written in the `cepcoreschema` XML language to apply the whole build chain: generate the code using the `cepgenerator` library and the `camitk-cepgenerator` application, configure the code using CMake and the CamiTK macros, build the code using the specific compiler available on the platform, and run the `camitk-config` application to check that the extensions can be loaded and available in any CamiTK application
- **Checking that different actions can be executed in a pipeline**. At the moment, this relies on the `camitk-actionstatemachine` application. The developer can create action pipelines writing an action state machine XML description language. During the test phase, the pipeline is ran automatically and the output files (i.e., the component that are saved to file during the pipeline execution) can be compared to expected files in order to verify that everything went according to plan.
......@@ -106,6 +112,7 @@ Not only the test results are visible, but CamiTK also offers a way to check the
## CamiTK testing framework
The CamiTK testing framework is defined As a layer on top of CMake/CTest, and provides _five_ different type of tests:
1. Automatic test for action and component extensions
2. Automatic test using a pipeline of actions
3. A set of specific CMake macros that help defining custom tests and test coverage
......@@ -121,6 +128,7 @@ The following sections describes these five different types of tests.
As CamiTK defines an API for action and component extensions, this API can be used to check the default/expected basic behaviour of component and action extensions. In order to do so, the CamiTK testing framework
provide two extra C++ applications: `camitk-testactions` and `camitk-testcomponents`:
- `camitk-testactions` provides one level of automatic test for a given action extension.
- `camitk-testcomponents` offers three different levels of automatic test for a given component extension.
......@@ -131,6 +139,7 @@ _additional component extension automatic tests_).
#### Testing action extensions
To test each action provided in the CamiTK framework, we insure that calling it on the proper components it has been defined on, works correctly. The `camitk-testactions` application has five main steps:
1. Load all the available CamiTK components the action can work on
2. Load the action extension itself
3. Open a component file and set it as the action input component
......@@ -139,6 +148,7 @@ To test each action provided in the CamiTK framework, we insure that calling it
The test passes if the `apply()` methods returns `Action::ApplyStatus::SUCCESS` or `Action::ApplyStatus::ABORT` (in this case it the developer responsability ot check that the `ABORT` status is _really_ what is expected). The test fails otherwise.
For technical details and implementation, please check:
- `CamiTKExtension.cmake` (in `sdk/cmake/modules/macros/camitk`) the script that creates the tests thanks to the `camitk_init_test` and `camitk_add_test` macro (resp. defined in `CamiTKInitTest.cmake` and `CamiTKAddTest.cmake` in `sdk/cmake/modules/macros/camitk/test`, see [the dedicated section below](#set-of-specific-cmake-macros-for-testing))
- the C++ source code for `testactions` (in `sdk/applications/testactions`)
......@@ -147,6 +157,7 @@ For example, check any actions in the SDK.
#### Testing component extensions
To test each component in the CamiTK framework, we insure that **opening** a file associated with this component and (when possible) **saving** it work. To do so, the `action-testcomponents` application goes trough four steps for each available (or specified) files in the component’s `testdata` subdirectory:
1. Load the component extension
2. Open the given file (test level 1)
3. Save the file using a different name (optional test level 2)
......@@ -155,6 +166,7 @@ To test each component in the CamiTK framework, we insure that **opening** a fil
The level of the test depends on the `AUTO_TEST_LEVEL` option specified in the `camitk_extension` macro.
For technical details and implementation, please check:
- `CamiTKExtension.cmake` (in `sdk/cmake/modules/macros/camitk`) the script that creates the tests thanks to the `camitk_init_test` and `camitk_add_test` macro (resp. defined in `CamiTKInitTest.cmake` and `CamiTKAddTest.cmake` in `sdk/cmake/modules/macros/camitk/test`, see [the dedicated section below](#set-of-specific-cmake-macros-for-testing))
- the C++ source code for `testactions` (in `sdk/applications/testactions`)
......@@ -173,6 +185,7 @@ Any file saved during the pipeline execution is going to be compared to the expe
In order to automatically add a test, you need to add the following line at the end of the action or component extension `CMakeLists.txt`:
This process is generalized thanks to some normalization: all the required following files should be available in a `integration-testdata` subdirectory:
- An CamiTK SCXML document `input-asm.scxml`
- Any number of `input-#.xxx` input files
- Any number of `output-#.xxx` output files (this are the expected output files)
......@@ -194,6 +207,7 @@ TODO : use this for memory leak check (and performance check?)
### Set of specific CMake macros for testing
CamiTK testing framework provides three specific CMake macros in order to more efficiently configure tests inside the CamiTK environment:
- `camitk_init_test(...)` helps to initialize all the common CMake configuration for all the test of the current CMake project.
- `camitk_add_test(...)` adds a new test for the current CamiTK project. This macro encapsulates the default `add_test(...)` function provided by CMake.
- `setup_target_for_coverage(...)`, entirely based on [Lars Bilke macro](https://github.com/bilke/cmake/modules/blob/master/CodeCoverage.cmake), simplifies the generation of test coverage report.
......@@ -213,6 +227,7 @@ The CamiTK Community Edition project tests if the valid number of expected actio
Once installed, CamiTK should be ready to be used to develop a CEP.
The CamiTK Community Edition project tests that the CamiTK development framework is ready to use doing the following steps (all described in bash script, see `sdk/applications/cepgenerator/testing/cepgenerator-test.sh`):
1. Test that the C++ code generation from a cep core schema XML is possible: this tests the availability of the `camitk-cepgenerator` application.
2. Test that the generated code can be configured: this tests that the CamiTK CMake macros and environment is correctly installed and accessible
3. Test that the generated code can be compiled: this tests that the core library and other dependencies are available and linkable with all the C++ compilation tool chain
......@@ -227,6 +242,7 @@ You can also create your own example from a tweaked core schema XML if your CEP
You can easily generate and check a test coverage report for the CamiTK Community Edition or your own extension.
There are two requirements to generate the test coverage report:
1. you need to use the GNU cxx compiler
2. you need to have `gcov` and `lcov` installed (`apt-get install` is your friend\!)
......
......@@ -8,7 +8,7 @@ The CamiTK team recommends `Visual Studio 2015` IDE.
Install Qt 5.6.1, Visual Studio 2015, CMake and git:
* [Visual Studio ](https://visualstudio.microsoft.com/fr/vs/older-downloads/)
* [Visual Studio ](https://visualstudio.microsoft.com/fr/vs/older-downloads/) please take a look at this [issue](https://gricad-gitlab.univ-grenoble-alpes.fr/CamiTK/CamiTK/issues/78)
* [CMake 3.7](https://cmake.org/files)
* For git, follow the [git starter guide page on the old wiki](https://camitk.gricad-pages.univ-grenoble-alpes.fr/Docs/Getting%20Started/Examples%20and%20Tutorials/Git_recipes/)
* [Qt 5.6.1](http://download.qt.io/archive/qt/5.6/5.6.1/qt-opensource-windows-x86-msvc2015_64-5.6.1.exe)
......
......@@ -4,6 +4,8 @@ id: "camitk-whitepaper-style"
# The CamiTK Automatic Test framework
Since: CamiTK 4.1
__or how to easily add integration test to your extension__
## Introduction
......@@ -15,6 +17,7 @@ It is very simple to set up a testing environment for the development of your ex
The CamiTK automatic testing module is built with three
[Kitware](http://www.kitware.com/) tools:
- [CMake](http://www.cmake.org/)
- [CTest](https://cmake.org/cmake/help/latest/manual/ctest.1.html)
- [CDash](http://www.cdash.org/)
......@@ -23,7 +26,7 @@ The CamiTK automatic testing module is built with three
Based on the common API for `Component` and `Action` extension, the CamiTK testing framework can automatically test *some aspects* of any extension.
You can refer to the technical overview [whitepaper about testing in CamiTK](/Docs/design/Testing-Framework-Whitepaper) to learn how testing is done in the CamiTK framework and in CamiTK Community Edition.
You can refer to the technical overview [whitepaper about testing in CamiTK](https://camitk.gricad-pages.univ-grenoble-alpes.fr/Docs/Getting%20Started/CamiTK%20Overviews/Testing-Framework-Whitepaper/) to learn how testing is done in the CamiTK framework and in CamiTK Community Edition.
### Components tests
......@@ -35,6 +38,7 @@ Testing a component extension requires some data. The files to test are expected
#### Level 1
This level performs two steps:
- Load the component extension
- Open a file
......@@ -43,6 +47,7 @@ It checks that the component extension can be loaded by a CamiTK application and
#### Level 2
Level 2 is the same as level 1, but it adds an extra step:
- Save the loaded component to another file
It checks that the component can be saved. This level can only be run if the component extension implements the `save(..)` method.
......@@ -50,6 +55,7 @@ It checks that the component can be saved. This level can only be run if the com
#### Level 3
Level 3 is the same as level 2, but it adds one other extra step:
- Make a diff between the input file and the output file.
It checks that the input and output data are exactly the same.
......@@ -71,6 +77,7 @@ For each action in the action extension, the action is tested using a test file.
Testing an action extension requires some data. The files to test are expected to be in the `testdata` build directory (i.e., the files used to tests action should be present in one of the `testdata` directory of one of the component extensions).
At the time of writing, there is only one level of test for actions, it consists in the following steps:
- Load all the component extensions
- Open a test file (should be found in one of the component extensions `testdata` directory)
- Run the action on the instantiated component
......@@ -83,6 +90,7 @@ Since CamiTK 4.1, you can also define a _pipeline_ of actions to perform using t
In this case, the CamiTK action state machine application is used to perform the actions defined in your pipeline one after the other automatically. Any file saved during the pipeline execution is going to be compared to the expected file and the test fails if any of the output files differ from the expected output files.
For instance, you can define a pipeline to:
- open an medical image from `input-1.mha`
- perform one filter
- save the result to `output-1.mha`
......@@ -100,6 +108,7 @@ You can record a first version of your pipeline using the "Save History" feature
## Step-by-step: How to automatically test your extension
You can easily add one or all of the following tests to your extension:
- automatic test of your extension (as described at step #2)
- additional test for existing actions that are already installed (as described at step #3)
- pipelined integration tests (as described at step #4)
......@@ -131,14 +140,20 @@ camitk_extension(COMPONENT_EXTENSION
```
As you can see above, there are two other option you can set to match specific requirements for your test:
- the `TEST_FILES` option
- the `AUTO_TEST_LEVEL` option (for component extension)
The option `TEST_FILES` let you specify the list of files to use for testing. If provided, only the filenames are required (not the absolute paths), each of them should be found in the `testdata` build dir. If no test files are provided then:
- For component extension: all the files in the component `testdata` subdirectory are used by default
- For action extension: all the files available in the `testdata` build dir (i.e., **all** the test files found in **all** the component `testdata` subdirectories) are used by default (note that it might be a lot of files!).
!!! Note
**Specific case** if you do not have any component in your CEP or if your component extensions do no provide any test files, you need to ensure that the test files used uniquely for testing your action are present in the `testdata` directory of the build directory. The `testdata` directory is in the subfolder `share/camitk-x.y` of the build directory (where `x.y` is the major.minor version of CamiTK)
For component extension, the parameter `AUTO_TEST_LEVEL` let you specify the level of automatic test to perform (if not specified, the default is 3, see also [above](#Components_tests)):
- Level 1: load the component extension, open and close a test component file
- Level 2: same as level 1, but save the component before closing it.
- Level 3 (default): same as level 2, but also compare the saved component to the input.
......@@ -165,6 +180,7 @@ camitk_additional_action_test(ACTION_EXTENSIONS action1 action2 ...
```
Where:
- `ACTION_EXTENSIONS` is the list of the action extension that will also be tested with the given test files.
Only the action extension name (not the full library/dll name nor the path) should be provided.
This macro checks the existence of the extension library/dll in the following directories (in this order): build dir; user install dir and global install dir. The first action that is found is used for the tests.
......@@ -176,7 +192,8 @@ Default value is the name of the current component extension. The resulting test
### Step 4: Add pipelined integration tests
To add a test that will run a pipeline of actions, apply them and check the result against resulting output, you just need to:
- create a `integration-test` directory in your component or action extension source directory
- create a `integration-testdata` directory in your component or action extension source directory
- add the `ENABLE_INTEGRATION_TEST` option to the `camitk_extension` macro:
```cmake
camitk_extension(...
......@@ -184,20 +201,23 @@ camitk_extension(...
ENABLE_INTEGRATION_TEST
)
```
- create a `asm-input.xml` SCXML document that describes the action pipeline. To create a SCXML file, the best way to start is to run `camitk-imp`, select, set the parameters and apply the actions you want to perform in your test (do not forget to use the `Save` action to output some or all of the resulting component). Then go to the `File` menu and choose `Save History`. You will then have to manually edit the resulting XML file in order to fullfil the requirements (see below)
- rename all the input files needed to run your pipeline to the normalized form `input-#.xxx` and copy them to the `integration-test` subdirectory
- rename all the expected output files to compare with actual output to the normalized form `output-#.xxx` and copy them to the `integration-test` subdirectory
- create a `asm-input.scxml` SCXML document that describes the action pipeline. To create a SCXML file, the best way to start is to run `camitk-imp`, select, set the parameters and apply the actions you want to perform in your test (do not forget to use the `Save` action to output some or all of the resulting component). Then go to the `File` menu and choose `Save History`. You will then have to manually edit the resulting XML file in order to fullfil the requirements (see below)
- rename all the input files needed to run your pipeline to the normalized form `input-#.xxx` and copy them to the `integration-testdata` subdirectory
- rename all the expected output files to compare with actual output to the normalized form `output-#.xxx` and copy them to the `integration-testdata` subdirectory
- run the CMake configuration again, et voilà, a new test called `extensiontype-integration-test` should be available
You can find an example in the `itkfilters` action extension of the `imaging` CEP (provided in the CamiTK Community Edition), and in the `image/reconstruction` action extension of the SDK.
!!! Note
In order to transform a saved history to an functional `asm-input.xml`, edit the XML files with an XML aware editor in order to:
1. Move all your input files to the `integration-testdata` subdirectory and rename them using the normalized form `input-#.xxx` where `#` is the unique index
2. Move the `asm-input.xml` in the `integration-testdata` subdirectory
3. Change (if any) `Open` action into `Open File` instead (`Open File` does not require user interaction or the use of an open dialog):
In order to transform a saved history to an functional `asm-input.xml`, you will need to edit the XML files with an XML aware editor. Just follow the step-by-step below.
Here are the steps required to transform a saved history SCXML to an functional `asm-input.scxml`:
1. Move all your input files to the `integration-testdata` subdirectory and rename them using the normalized form `input-#.xxx` where `#` is the unique index
2. Move the `asm-input.scxml` in the `integration-testdata` subdirectory
3. Change (if any) `Open` action into `Open File` instead (`Open File` does not require user interaction or the use of an open dialog):
```xml
...
...
<camitk:action>
<camitk:name>Open</camitk:name>
<camitk:parameters/>
......@@ -214,18 +234,18 @@ Becomes
```
4. For each action that produce a new component that will be later saved and compared to the expected result, you need to **rename** the component output name throughout the SCXML document to the normalized form `output-#.xxx`, where `#` is a unique index and `xxx` is the proper extension to save. For instance:
```xml
<camitk:action>
<camitk:name>An Action That Creates a New Component</camitk:name>
<camitk:parameters>
...
</camitk:parameters>
<camitk:inputs>
<camitk:action>
<camitk:name>An Action That Creates a New Component</camitk:name>
<camitk:parameters>
...
</camitk:parameters>
<camitk:inputs>
...
</camitk:inputs>
<camitk:outputs>
<camitk:component name="named created automatically by the action (sometimes without extension)" type="ComponentClass"/>
</camitk:outputs>
...
</camitk:inputs>
<camitk:outputs>
<camitk:component name="named created automatically by the action (sometimes without extension)" type="ComponentClass"/>
</camitk:outputs>
...
```
Becomes:
```xml
......@@ -244,7 +264,7 @@ Becomes:
```
5. Change (if any) `Save As` action into `Save` instead. Also make sure that the input parameter of the `Save` action is using a component name corresponding to one of the `output-#.xxx` that you renamed in the previous step.
6. You may also rename the state's `id` attributes in order to use more explicit names (do not forget to modify the corresponding transition's `target` attributes)
7. test the `asm-input.xml` using the action state machine:
7. test the `asm-input.scxml` using the action state machine:
```bash
# On the command line, go to the directory where you stored all input files and asm-input.xml documents
camitk-actionstatemachine --file asm-input.xml --output-dir .
......@@ -317,6 +337,7 @@ ctest -VV -R action-myactionname # this will execute all the tests automatically
### Step 6: Check test coverage of your CEP
A report on test coverage can also be generated automatically for any CamiTK Extension Project. The requirements are that:
1. you need to use the GNU cxx compiler
2. you need to have **gcov** and **lcov** installed (**apt-get install** is your friend !)
......@@ -349,10 +370,12 @@ If your project is hosted on a gitlab server, you can quickly add continuous int
CI is very useful for mature code to set high level of code quality in order to protect your code from any regression.
In order to do that you need:
- to install gitlab runners (they are in charge of executing the pipeline on the VM/OS of your choice)
- to add a specific `.gitlab-ci.yml` configuration file in the root directory of your CEP.
CamiTK Community Edition has continuous integration in place, so that:
- each push to the gitlab hosted code (in any branch) triggers the CI pipeline
- each day a nightly pipeline is executed on the `develop` branch.
......
......@@ -11,14 +11,14 @@ theme:
logo: 'assets/logo_CamiTK.png'
favicon: 'assets/logo_CamiTK.png'
feature:
tabs: true
tabs: false
markdown_extensions:
- tables
- admonition
- codehilite:
linenums: true
guess_lang: false
guess_lang: true
- toc:
permalink: true
- footnotes
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment