Automatic-Tests.md 21.6 KB
Newer Older
Jean-Loup Haberbusch's avatar
Jean-Loup Haberbusch committed
1
2
3
4
5
6
<!-- ---
id: "camitk-whitepaper-style"
--- -->

# The CamiTK Automatic Test framework

7
8
Since: CamiTK 4.1

Jean-Loup Haberbusch's avatar
Jean-Loup Haberbusch committed
9
10
11
12
13
14
15
16
17
18
19
__or how to easily add integration test to your extension__

## Introduction

Testing can be a time consuming task, this is why CamiTK comes with automatic tests tools directly implemented in the CamiTK framework.
Every action and component extensions of the CamiTK opensource framework and of any other CEP can be automatically tested thanks to the CamiTK testing framework.

It is very simple to set up a testing environment for the development of your extension, this page details the different steps and possibilities.

The CamiTK automatic testing module is built with three
[Kitware](http://www.kitware.com/) tools:
20

Jean-Loup Haberbusch's avatar
Jean-Loup Haberbusch committed
21
22
23
24
25
26
27
28
- [CMake](http://www.cmake.org/)
- [CTest](https://cmake.org/cmake/help/latest/manual/ctest.1.html)
- [CDash](http://www.cdash.org/)

## CamiTK automatic test framework overview

Based on the common API for `Component` and `Action` extension, the CamiTK testing framework can automatically test *some aspects* of any extension.

29
You can refer to the technical overview [whitepaper about testing in CamiTK](https://camitk.gricad-pages.univ-grenoble-alpes.fr/Docs/Getting%20Started/CamiTK%20Overviews/Testing-Framework-Whitepaper/) to learn how testing is done in the CamiTK framework and in CamiTK Community Edition.
Jean-Loup Haberbusch's avatar
Jean-Loup Haberbusch committed
30
31
32
33
34
35
36
37
38
39
40

### Components tests

For each component extension, the test procedure has three possible level from level 1 (basic testing) to level 3 (most advanced testing so far).
By default the component extension is tested at level 3. If for some (good) reason you don't want to run the test at level 3, you can downgrade the level to level 2 or level 1.

Testing a component extension requires some data. The files to test are expected to be in the `testdata` subdirectory of a component extension.

#### Level 1

This level performs two steps:
41

Jean-Loup Haberbusch's avatar
Jean-Loup Haberbusch committed
42
43
44
45
46
47
48
49
- Load the component extension
- Open a file

It checks that the component extension can be loaded by a CamiTK application and that a test file can be opened.

#### Level 2

Level 2 is the same as level 1, but it adds an extra step:
50

Jean-Loup Haberbusch's avatar
Jean-Loup Haberbusch committed
51
52
53
54
55
56
57
- Save the loaded component to another file

It checks that the component can be saved. This level can only be run if the component extension implements the `save(..)` method.

#### Level 3

Level 3 is the same as level 2, but it adds one other extra step:
58

Jean-Loup Haberbusch's avatar
Jean-Loup Haberbusch committed
59
60
61
62
63
64
65
66
67
- Make a diff between the input file and the output file.

It checks that the input and output data are exactly the same.

### Actions tests

For each action in the action extension, the action is tested using a test file. Basically the test consists in calling the `apply()` method of the action on the loaded component. The test pass if the `apply()` method returns `SUCCESS` or `ABORTED` and fails otherwise.


68
69
!!! Note
    when test is `ABORTED`, you need to make sure that this is what you expected.
Jean-Loup Haberbusch's avatar
Jean-Loup Haberbusch committed
70

71
72
!!! Note
    if the action cannot be applied to this type of component, the test pass. This is a limitation of this CamiTK test module.
Jean-Loup Haberbusch's avatar
Jean-Loup Haberbusch committed
73

74
75
!!! Note
    the `apply()` method is tested with default parameters defined in the action itself (unit testing). This is up to the action developer to configure its testing procedure with correct default values.
Jean-Loup Haberbusch's avatar
Jean-Loup Haberbusch committed
76
77
78
79

Testing an action extension requires some data. The files to test are expected to be in the `testdata` build directory (i.e., the files used to tests action should be present in one of the `testdata` directory of one of the component extensions).

At the time of writing, there is only one level of test for actions, it consists in the following steps:
80

Jean-Loup Haberbusch's avatar
Jean-Loup Haberbusch committed
81
82
83
84
85
86
87
88
89
90
91
92
- Load all the component extensions
- Open a test file (should be found in one of the component extensions `testdata` directory)
- Run the action on the instantiated component
- Check the action status

### Pipelined integration tests

Since CamiTK 4.1, you can also define a _pipeline_ of actions to perform using the automatic integration test framework.

In this case, the CamiTK action state machine application is used to perform the actions defined in your pipeline one after the other automatically. Any file saved during the pipeline execution is going to be compared to the expected file and the test fails if any of the output files differ from the expected output files.

For instance, you can define a pipeline to:
93

Jean-Loup Haberbusch's avatar
Jean-Loup Haberbusch committed
94
95
96
97
98
99
100
101
102
103
104
- open an medical image from `input-1.mha`
- perform one filter
- save the result to `output-1.mha`
- perform a mesh 3D reconstruction
- save the result to `output-2.vtk`

When the test is executed all these actions are performed automatically one after the other and the resulting `output-1.mha` and `output-2.vtk` are compared to expected files.
The test succeed if the output files are equals to the expected output files.

You can record a first version of your pipeline using the "Save History" feature of `camitk-imp`. You will have to edit the resulting SCXML file in order to format the names of the input and output files. The SCXML, input and output files should be placed in a `integration-test` subfolder of either your component or action extension.

105
106
!!! Note
    The main limitation for this automatic test is  of course that you cannot use actions that requires user interaction (but he! of course, this is automatic tests, so there is no user to interact with the test...).
Jean-Loup Haberbusch's avatar
Jean-Loup Haberbusch committed
107
108
109
110

## Step-by-step: How to automatically test your extension

You can easily add one or all of the following tests to your extension:
111

Jean-Loup Haberbusch's avatar
Jean-Loup Haberbusch committed
112
113
114
115
- automatic test of your extension (as described at step #2)
- additional test for existing actions that are already installed (as described at step #3)
- pipelined integration tests (as described at step #4)

116
117
!!! Note
    Tests are made in your own CEP context and do not take into account a possible installation. For instance, if you decide to proceed with the installation of your CEP components in CamiTK. It will be loaded with all other CamiTK component. This will lead you to an error message and a fail for your test.
Jean-Loup Haberbusch's avatar
Jean-Loup Haberbusch committed
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142

### Step 1: Enable the CamiTK testing framework

The first thing to do is to tell CMake that for this project you want to enable your test. In the top level `CMakeLists.txt` of your project (if you're working on a CamiTK Extenstion Project, open the top level `CMakeLists.txt` file and add the option `ENABLE_TESTING` to the `camitk_extension_project(...)` parameters:

```cmake
camitk_extension_project(...
                         ENABLE_TESTING
)
```

### Step 2: Enable automatic test of your extension

In the `CMakeLists.txt` of your component or action extension, add the `ENABLE_AUTO_TEST` option to automatically let the CamiTK test framework create some test. Tests will be automatically generated for your extension.

```cmake
camitk_extension(COMPONENT_EXTENSION
                 CEP_NAME MYCEP
                 DESCRIPTION "my extension description"
                 ENABLE_AUTO_TEST
                 [TEST_FILES file1 file2...]
                 [AUTO_TEST_LEVEL level] # only for component extension
```

As you can see above, there are two other option you can set to match specific requirements for your test:
143

Jean-Loup Haberbusch's avatar
Jean-Loup Haberbusch committed
144
145
146
147
- the `TEST_FILES` option
- the `AUTO_TEST_LEVEL` option (for component extension)

The option `TEST_FILES` let you specify the list of files to use for testing. If provided, only the filenames are required (not the absolute paths), each of them should be found in the `testdata` build dir. If no test files are provided then:
148

Jean-Loup Haberbusch's avatar
Jean-Loup Haberbusch committed
149
150
151
- For component extension: all the files in the component `testdata` subdirectory are used by default
- For action extension: all the files available in the `testdata` build dir (i.e., **all** the test files found in **all** the component `testdata` subdirectories) are used by default (note that it might be a lot of files!).

152
153
154
!!! Note
    **Specific case** if you do not have any component in your CEP or if your component extensions do no provide any test files, you need to ensure that the test files used uniquely for testing your action are present in the `testdata` directory of the build directory. The `testdata` directory is in the subfolder `share/camitk-x.y` of the build directory (where `x.y` is the major.minor version of CamiTK)
    
Jean-Loup Haberbusch's avatar
Jean-Loup Haberbusch committed
155
For component extension, the parameter `AUTO_TEST_LEVEL` let you specify the level of automatic test to perform (if not specified, the default is 3, see also [above](#Components_tests)):
156

Jean-Loup Haberbusch's avatar
Jean-Loup Haberbusch committed
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
- Level 1: load the component extension, open and close a test component file
- Level 2: same as level 1, but save the component before closing it.
- Level 3 (default): same as level 2, but also compare the saved component to the input.

### Step 3: Define additional test for existing actions

In some cases, it can be useful to test your component with actions that are already installed (e.g., with an action provided by the CamiTK Community Edition).
**For instance:** if you develop a new component extension that derived from `ImageComponent`, an additional test can be automatically generated by the CamiTK test framework in order to check that the existing ITK Otsu filter action (provided by the CamiTK Community Edition) works on your new component.

The macro `camitk_additional_action_test` is here to do just that: it creates an additional level 1 test for an existing (installed) CamiTK action using test data from the current component extension.

This macro can be used to check the quality of the integration level of a new component extension defined within a CEP introduces. It will take new test files and run the defined action.
If the new component class derives from an existing (installed) component class for which there are existing (installed) actions, these actions should indeed works using the new derived component class.
**For example:** if a CEP introduces a new `MeshComponent` derived class, then all the actions that works on `MeshComponent` should also work on the new component.

To define an additional test for an existing action, you only need to modify the component extension `CMakeLists.txt`, and add _one line_ just after the
`camitk_extension(COMPONENT_EXTENSION ...)` macro, at the end of the `CMakeLists.txt`:

```cmake
camitk_additional_action_test(ACTION_EXTENSIONS action1 action2 ...
                              [TEST_FILES  file1 file2 ...]
                              [TEST_SUFFIX name]
)
```

Where:
183

Jean-Loup Haberbusch's avatar
Jean-Loup Haberbusch committed
184
185
186
187
188
189
190
191
192
193
194
- `ACTION_EXTENSIONS` is the list of the action extension that will also be tested with the given test files.
Only the action extension name (not the full library/dll name nor the path) should be provided.
This macro checks the existence of the extension library/dll in the following directories (in this order): build dir; user install dir and global install dir. The first action that is found is used for the tests.
- `TEST_FILES` is an optional argument. It can specify the list of files to use for testing.
If not set, then all the files in the `testdata` subdirectories are used. If provided, only the filenames are required (not the absolute paths) and all the given files should be in the `testdata` subdirectory of the current component.
- `TEST_SUFFIX` is an optional argument. It provides the suffix name for the additional test.
Default value is the name of the current component extension. The resulting test will be called `action-A-additional-C-level1-i` (where `A` is the action name, `C` the component name and `i` the test order number depending of the filename order).

### Step 4: Add pipelined integration tests

To add a test that will run a pipeline of actions, apply them and check the result against resulting output, you just need to:
195

Jean-Loup Haberbusch's avatar
Jean-Loup Haberbusch committed
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
- create a `integration-test` directory in your component or action extension source directory
- add the `ENABLE_INTEGRATION_TEST` option to the `camitk_extension` macro:
```cmake
camitk_extension(...
				 ...
                 ENABLE_INTEGRATION_TEST
)
```
- create a `asm-input.xml` SCXML document that describes the action pipeline. To create a SCXML file, the best way to start is to run `camitk-imp`, select, set the parameters and apply the actions you want to perform in your test (do not forget to use the `Save` action to output some or all of the resulting component). Then go to the `File` menu and choose `Save History`. You will then have to manually edit the resulting XML file in order to fullfil the requirements (see below)
- rename all the input files needed to run your pipeline to the normalized form `input-#.xxx` and copy them to the `integration-test` subdirectory
- rename all the expected output files to compare with actual output to the normalized form `output-#.xxx` and copy them to the `integration-test` subdirectory
- run the CMake configuration again, et voilà, a new test called `extensiontype-integration-test` should be available

You can find an example in the `itkfilters` action extension of the `imaging` CEP (provided in the CamiTK Community Edition), and in the `image/reconstruction` action extension of the SDK.

211
!!! Note
212
213
214
215
216
217
218
    In order to transform a saved history to an functional `asm-input.xml`, you will need to edit the XML files with an XML aware editor. Just follow the step-by-step below.
    
Here are the steps required to transform a saved history SCXML to an functional `asm-input.xml`:

1. Move all your input files to the `integration-testdata` subdirectory and rename them using the normalized form `input-#.xxx` where `#` is the unique index
2. Move the `asm-input.xml` in the `integration-testdata` subdirectory
3. Change (if any) `Open` action into `Open File` instead (`Open File` does not require user interaction or the use of an open dialog):
Jean-Loup Haberbusch's avatar
Jean-Loup Haberbusch committed
219
```xml
220
    ...
Jean-Loup Haberbusch's avatar
Jean-Loup Haberbusch committed
221
222
223
224
	<camitk:action>
    	<camitk:name>Open</camitk:name>
    	<camitk:parameters/>
		...
Emmanuel Promayon's avatar
Emmanuel Promayon committed
225
226
227
```
Becomes
```xml
Jean-Loup Haberbusch's avatar
Jean-Loup Haberbusch committed
228
229
230
231
232
233
234
235
236
	<camitk:action>
    	<camitk:name>Open File</camitk:name>
    		<camitk:parameters>
      			<camitk:parameter name="File Name" value="input-#.mha" type="QString"/>
     		</camitk:parameters>
			...
```
4. For each action that produce a new component that will be later saved and compared to the expected result, you need to **rename** the component output name throughout the SCXML document to the normalized form `output-#.xxx`, where `#` is a unique index and `xxx` is the proper extension to save. For instance:
```xml
237
238
239
240
241
242
<camitk:action>
    <camitk:name>An Action That Creates a New Component</camitk:name>
    <camitk:parameters>
     		...
    </camitk:parameters>
    <camitk:inputs>
Jean-Loup Haberbusch's avatar
Jean-Loup Haberbusch committed
243
      		...
244
245
246
247
248
    </camitk:inputs>
	<camitk:outputs>
		<camitk:component name="named created automatically by the action (sometimes without extension)" type="ComponentClass"/>
	</camitk:outputs>
	...
Jean-Loup Haberbusch's avatar
Jean-Loup Haberbusch committed
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
```
Becomes:
```xml
<camitk:action>
	<camitk:name>An Action That Creates a New Component</camitk:name>
	<camitk:parameters>
		...
	</camitk:parameters>
	<camitk:inputs>
		...
	</camitk:inputs>
	<camitk:outputs>
		<camitk:component name="output-#.xxx" type="ComponentClass"/>
	</camitk:outputs>
	...
```
5. Change (if any) `Save As` action into `Save` instead. Also make sure that the input parameter of the `Save` action is using a component name corresponding to one of the `output-#.xxx` that you renamed in the previous step.
6. You may also rename the state's `id` attributes in order to use more explicit names (do not forget to modify the corresponding transition's `target` attributes)
7. test the `asm-input.xml` using the action state machine:
```bash
# On the command line, go to the directory where you stored all input files and asm-input.xml documents
camitk-actionstatemachine --file asm-input.xml --output-dir .
# you should only have to press "Next"
# The execution ends up with all the expected output in a new YYYY-MM-DDTHH-MM-SS directory
# There should be no diff between the expected output and the actual output (sometimes you need to run it once to produce the proper expected output)
# if that works well try the autonext option (it should not just run without any user interaction)
camitk-actionstatemachine --file asm-input.xml --output-dir . --autonext
```



### Step 5: Run your tests


#### Run all the tests

CMake automatically creates a target to run **all** the tests you have configured, that include the automatic test generated by the CamiTK testing framework.

- On Linux and MacOS the target's name is **test** so simply run:
```bash
make test
```
- On Windows system, running a Visual Studio instance, simply build the project `RUN_TESTS`

#### Run specific tests

It is possible to have a finer control on running the tests.

Your build directory must contain a file named `CTestTestfile.cmake` (generated during the configuration stage). This will allow you to type the following command, to execute _some_ of the tests defined:

```bash
ctest -C <configuration> -V # note that "-C <configuration>" is optional on Linux and MacOS
```

For example:
```bash
ctest -C Debug -V   # note that "-C Debug" is optional on Linux and MacOS
```

`-V` option is for verbose, `-VV` is for extra verbose

For a specific test identified by its index use the `-I` option:
```bash
ctest -C debug -I <start index>,<end index>
```

For example:
``` bash
ctest -C debug -I 5,8 # this will execute test #5, #6, #7 and #8
```

For a specific test identified by its name, use the `-R` option:
```bash
ctest -VV -R <test-name-regex>
```

For example:
```bash
ctest -VV -R action-myactionname # this will execute all the tests automatically created for myactionname
```

330
331
332
333
334
!!! Note
    The CamiTK test framework add specific prefix to the tests it automatically creates:
    - `action-xxx` → prefix of all the tests defined for the action extension named `xxx`
    - `component-xxx` → prefix of all the tests defined for the component extension named `xxx`
    - `application-xxx` → prefix of all the tests defined for the application extension named `xxx`
Jean-Loup Haberbusch's avatar
Jean-Loup Haberbusch committed
335
336
337
338
339


### Step 6: Check test coverage of your CEP

A report on test coverage can also be generated automatically for any CamiTK Extension Project. The requirements are that:
340

Jean-Loup Haberbusch's avatar
Jean-Loup Haberbusch committed
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
1. you need to use the GNU cxx compiler
2. you need to have **gcov** and **lcov** installed (**apt-get install** is your friend !)

Then, it is very easy: open the top level `CMakeLists.txt` file of your CamiTK
Extenstion Project and add the option `ENABLE_TEST_COVERAGE` to the `camitk_extension_project(...)` parameters:

```cmake
camitk_extension_project(...
                         ENABLE_TEST_COVERAGE
)
```

Once your project is built, you can then run the `camitk-test-coverage` target:

```bash
make camitk-test-coverage
```

It will run all the available tests of your CEP using `ctest` and then compute the test coverage and generat an html report. The reports will be available for visualisation in
`${CMAKE_BUILD_DIR}/camitk-test-coverage/index.html` (the `camitk-test-coverage` subdirectory of the build dir).
So for instance, just type:
```bash
firefox camitk-test-coverage/index.html
```


### Step 7: Add continuous integration

If your project is hosted on a gitlab server, you can quickly add continuous integration (CI) so that each modification triggers a configure/build/test pipeline.
CI is very useful for mature code to set high level of code quality in order to protect your code from any regression.

In order to do that you need:
373

Jean-Loup Haberbusch's avatar
Jean-Loup Haberbusch committed
374
375
376
377
- to install gitlab runners (they are in charge of executing the pipeline on the VM/OS of your choice)
- to add a specific `.gitlab-ci.yml` configuration file in the root directory of your CEP.

CamiTK Community Edition has continuous integration in place, so that:
378

Jean-Loup Haberbusch's avatar
Jean-Loup Haberbusch committed
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
- each push to the gitlab hosted code (in any branch) triggers the CI pipeline
- each day a nightly pipeline is executed on the `develop` branch.

CamiTK Community Edition uses 4 different runners: three on Linux (debian stable, old stable, and Ubuntu LTS, all using a docker executor) and one on Windows (Win7 using a bash shell executor).

In the end we get a summary of all the results (as well as report artefacts and emails to the developer in case of an error) send to the [CamiTK Community Edition dashboard](https://cdash-timc.imag.Fr/index.php).
For instance, this shows where a the nightly build:

![CamiTK console](assets/cdash.png)

Contact us if you would like more information about this and some support and help on how to set CI in place powerful for your project.






<!--
OLD Documentation: it is still in place, but is it worth describing?

Send your test results to a dashboard

The [CamiTK Community Edition dashboard](https://cdash-timc.imag.Fr/index.php) aims at
gathering compilation and testing reports. This server runs [CDash](http://http://www.cdash.org/) to gather this information, that are sent by each computer that compiles and run tests from its own “build” of CamiTK Community Edition source code. This tutorial will show you how to configure your computer to send build report and make test using CTest.

## How to send a build report to the server

Sending a build report to the [CamiTK Community Edition Dashboard](https://cdash-timc.imag.Fr/index.php) is pretty simple since all the configuration information are included in the `CMakeLists.txt` and `CTestConfig.txt`) files at the root of the source code.
When configuring a build for Windows or Linux and if you add the specific options to the `CMakeLists.txt` as described in the above steps, it will automatically create targets for continuous integration purpose.
Several
targets are created, such As “Nightly” etc. Building one of those
projects will automatically send a report to the server (considering you
have an active internet connection) tagged with your computer name and
the build configuration ‘Debug’ / ‘Release’.

To configure the computer name sending report, adjust the CMake variable
“SITE” In the global CamiTK CMake configuration (In ‘ungrouped
entries’).

## Manually submit test resport to the Server

On Linux and MacOS, this can be done easily from the command line.

To run the test and submit, just type:

``` bash
ctest -D Experimental
```

To check the test coverage, generate the report and submit, type:

``` bash
make camitk-test-coverage
ctest -T Coverage
ctest -T Submit
```

-->