Pipeline does not detect failed tests

About you

CamiTK developer

Overview

The action state machine tests failed, tests are sometimes reran, but the errors are not detected by the test script. The pipeline does not fail although it should clearly fail. We are lull to a false sense of security, while there is a bug! The two Linux configuration shows different logs but both do not detect the error

Steps to Reproduce

• For linux stable: check Job #177308
• For linux LTS: check Job #177309

Actual VS Expected Result

The pipeline should fail when there is a failed test, independently of the type of failure.

Relevant logs and/or screenshots

On Linux stable

In the Job #177308 log, the initial tests failed with:

4/5 Test #132: application-actionstatemachine-5 .........Child aborted***Exception:   2.81 sec
5/5 Test #131: application-actionstatemachine-4 .........***Timeout 1800.13 sec

But then at the end of the job log we have:

===== Fatal errors =====
Found fatal error in /builds/CamiTK/CamiTK/debian-stable-develop/log/ci-test.log
===== Failed tests =====
===== Re-running failed tests =====
Re-ran tests OK

The relevant lines in the test.log show:

132: X Error of failed request:  BadWindow (invalid Window parameter)
132:   Major opcode of failed request:  2 (X_ChangeWindowAttributes)
132:   Resource id in failed request:  0x2
132:   Serial number of failed request:  7
132:   Current serial number in output stream:  9
132: terminate called after throwing an instance of 'std::bad_array_new_length'
132:   what():  std::bad_array_new_length
4/5 Test #132: application-actionstatemachine-5 .........Child aborted***Exception:   2.81 sec
5/5 Test #131: application-actionstatemachine-4 .........***Timeout 1800.13 sec

The rerun-test.log contains only one line:

Checking /builds/CamiTK/CamiTK/debian-stable-develop/log/test.log for failed test

On Linux LTS

In the Job #177309 log, the initial tests failed with:

4/5 Test #131: application-actionstatemachine-4 .........Child aborted***Exception:   0.71 sec
5/5 Test #132: application-actionstatemachine-5 .........***Timeout 1800.13 sec

But then at the end of the job log we have:

===== Fatal errors =====
Found fatal error in /builds/CamiTK/CamiTK/ubuntu-lts-develop/log/ci-test.log
===== Failed tests =====
===== Re-running failed tests =====
Re-ran tests OK

The relevant lines in the test.log show:

131: ...
131: X Error of failed request:  BadWindow (invalid Window parameter)
131:   Major opcode of failed request:  2 (X_ChangeWindowAttributes)
131:   Resource id in failed request:  0x2
131:   Serial number of failed request:  7
131:   Current serial number in output stream:  9
131: terminate called after throwing an instance of 'std::bad_alloc'
131:   what():  std::bad_alloc
4/5 Test #131: application-actionstatemachine-4 .........Child aborted***Exception:   0.71 sec
5/5 Test #132: application-actionstatemachine-5 .........***Timeout 1800.13 sec

The rerun-test.log contains only one line:

Checking /builds/CamiTK/CamiTK/ubuntu-lts-develop/log/test.log for failed test

Interpretation & Possible fixes

The test script is faulty, see .gitlab/test.sh. It should be better at parsing the log to detect problems

CamiTK Version

CamiTK 4.2.dev.develop.92087062

Compiled using git Hash: 92087062, Date: Tue Oct 20 00:24:15 2020 +0200.