Commit d2a0202d authored by Emmanuel Promayon's avatar Emmanuel Promayon
Browse files

FIXED testdata location and update md so that list items appear as such

parent 073ff06a
Pipeline #17955 canceled with stage
in 0 seconds
......@@ -4,6 +4,8 @@ id: "camitk-whitepaper-style"
# The CamiTK Automatic Test framework
Since: CamiTK 4.1
__or how to easily add integration test to your extension__
## Introduction
......@@ -35,6 +37,7 @@ Testing a component extension requires some data. The files to test are expected
#### Level 1
This level performs two steps:
- Load the component extension
- Open a file
......@@ -43,6 +46,7 @@ It checks that the component extension can be loaded by a CamiTK application and
#### Level 2
Level 2 is the same as level 1, but it adds an extra step:
- Save the loaded component to another file
It checks that the component can be saved. This level can only be run if the component extension implements the `save(..)` method.
......@@ -50,6 +54,7 @@ It checks that the component can be saved. This level can only be run if the com
#### Level 3
Level 3 is the same as level 2, but it adds one other extra step:
- Make a diff between the input file and the output file.
It checks that the input and output data are exactly the same.
......@@ -71,6 +76,7 @@ For each action in the action extension, the action is tested using a test file.
Testing an action extension requires some data. The files to test are expected to be in the `testdata` build directory (i.e., the files used to tests action should be present in one of the `testdata` directory of one of the component extensions).
At the time of writing, there is only one level of test for actions, it consists in the following steps:
- Load all the component extensions
- Open a test file (should be found in one of the component extensions `testdata` directory)
- Run the action on the instantiated component
......@@ -83,6 +89,7 @@ Since CamiTK 4.1, you can also define a _pipeline_ of actions to perform using t
In this case, the CamiTK action state machine application is used to perform the actions defined in your pipeline one after the other automatically. Any file saved during the pipeline execution is going to be compared to the expected file and the test fails if any of the output files differ from the expected output files.
For instance, you can define a pipeline to:
- open an medical image from `input-1.mha`
- perform one filter
- save the result to `output-1.mha`
......@@ -100,6 +107,7 @@ You can record a first version of your pipeline using the "Save History" feature
## Step-by-step: How to automatically test your extension
You can easily add one or all of the following tests to your extension:
- automatic test of your extension (as described at step #2)
- additional test for existing actions that are already installed (as described at step #3)
- pipelined integration tests (as described at step #4)
......@@ -131,14 +139,20 @@ camitk_extension(COMPONENT_EXTENSION
```
As you can see above, there are two other option you can set to match specific requirements for your test:
- the `TEST_FILES` option
- the `AUTO_TEST_LEVEL` option (for component extension)
The option `TEST_FILES` let you specify the list of files to use for testing. If provided, only the filenames are required (not the absolute paths), each of them should be found in the `testdata` build dir. If no test files are provided then:
- For component extension: all the files in the component `testdata` subdirectory are used by default
- For action extension: all the files available in the `testdata` build dir (i.e., **all** the test files found in **all** the component `testdata` subdirectories) are used by default (note that it might be a lot of files!).
!!! Note
**Specific case** if you do not have any component in your CEP or if your component extensions do no provide any test files, you need to ensure that the test files used uniquely for testing your action are present in the `testdata` directory of the build directory. The `testdata` directory is in the subfolder `share/camitk-x.y` of the build directory (where `x.y` is the major.minor version of CamiTK)
For component extension, the parameter `AUTO_TEST_LEVEL` let you specify the level of automatic test to perform (if not specified, the default is 3, see also [above](#Components_tests)):
- Level 1: load the component extension, open and close a test component file
- Level 2: same as level 1, but save the component before closing it.
- Level 3 (default): same as level 2, but also compare the saved component to the input.
......@@ -165,6 +179,7 @@ camitk_additional_action_test(ACTION_EXTENSIONS action1 action2 ...
```
Where:
- `ACTION_EXTENSIONS` is the list of the action extension that will also be tested with the given test files.
Only the action extension name (not the full library/dll name nor the path) should be provided.
This macro checks the existence of the extension library/dll in the following directories (in this order): build dir; user install dir and global install dir. The first action that is found is used for the tests.
......@@ -176,6 +191,7 @@ Default value is the name of the current component extension. The resulting test
### Step 4: Add pipelined integration tests
To add a test that will run a pipeline of actions, apply them and check the result against resulting output, you just need to:
- create a `integration-test` directory in your component or action extension source directory
- add the `ENABLE_INTEGRATION_TEST` option to the `camitk_extension` macro:
```cmake
......@@ -192,19 +208,22 @@ camitk_extension(...
You can find an example in the `itkfilters` action extension of the `imaging` CEP (provided in the CamiTK Community Edition), and in the `image/reconstruction` action extension of the SDK.
!!! Note
In order to transform a saved history to an functional `asm-input.xml`, edit the XML files with an XML aware editor in order to:
1. Move all your input files to the `integration-testdata` subdirectory and rename them using the normalized form `input-#.xxx` where `#` is the unique index
2. Move the `asm-input.xml` in the `integration-testdata` subdirectory
3. Change (if any) `Open` action into `Open File` instead (`Open File` does not require user interaction or the use of an open dialog):
In order to transform a saved history to an functional `asm-input.xml`, you will need to edit the XML files with an XML aware editor. Just follow the step-by-step below.
Here are the steps required to transform a saved history SCXML to an functional `asm-input.xml`:
1. Move all your input files to the `integration-testdata` subdirectory and rename them using the normalized form `input-#.xxx` where `#` is the unique index
2. Move the `asm-input.xml` in the `integration-testdata` subdirectory
3. Change (if any) `Open` action into `Open File` instead (`Open File` does not require user interaction or the use of an open dialog):
```xml
...
...
<camitk:action>
<camitk:name>Open</camitk:name>
<camitk:parameters/>
...
```
Becomes
```xml
```
Becomes
```xml
<camitk:action>
<camitk:name>Open File</camitk:name>
<camitk:parameters>
......@@ -214,18 +233,18 @@ Becomes
```
4. For each action that produce a new component that will be later saved and compared to the expected result, you need to **rename** the component output name throughout the SCXML document to the normalized form `output-#.xxx`, where `#` is a unique index and `xxx` is the proper extension to save. For instance:
```xml
<camitk:action>
<camitk:name>An Action That Creates a New Component</camitk:name>
<camitk:parameters>
...
</camitk:parameters>
<camitk:inputs>
<camitk:action>
<camitk:name>An Action That Creates a New Component</camitk:name>
<camitk:parameters>
...
</camitk:parameters>
<camitk:inputs>
...
</camitk:inputs>
<camitk:outputs>
<camitk:component name="named created automatically by the action (sometimes without extension)" type="ComponentClass"/>
</camitk:outputs>
...
</camitk:inputs>
<camitk:outputs>
<camitk:component name="named created automatically by the action (sometimes without extension)" type="ComponentClass"/>
</camitk:outputs>
...
```
Becomes:
```xml
......@@ -317,6 +336,7 @@ ctest -VV -R action-myactionname # this will execute all the tests automatically
### Step 6: Check test coverage of your CEP
A report on test coverage can also be generated automatically for any CamiTK Extension Project. The requirements are that:
1. you need to use the GNU cxx compiler
2. you need to have **gcov** and **lcov** installed (**apt-get install** is your friend !)
......@@ -349,10 +369,12 @@ If your project is hosted on a gitlab server, you can quickly add continuous int
CI is very useful for mature code to set high level of code quality in order to protect your code from any regression.
In order to do that you need:
- to install gitlab runners (they are in charge of executing the pipeline on the VM/OS of your choice)
- to add a specific `.gitlab-ci.yml` configuration file in the root directory of your CEP.
CamiTK Community Edition has continuous integration in place, so that:
- each push to the gitlab hosted code (in any branch) triggers the CI pipeline
- each day a nightly pipeline is executed on the `develop` branch.
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment