## About you CamiTK developer ## Overview The action state machine tests failed, tests are sometimes reran, but the errors are not detected by the test script. The pipeline does not fail although it should clearly fail. We are lull to a false sense of security, while there is a bug! The two Linux configuration shows different logs but both do not detect the error ## Steps to Reproduce - For linux stable: check [Job #177308](https://gricad-gitlab.univ-grenoble-alpes.fr/CamiTK/CamiTK/-/jobs/177308) - For linux LTS: check [Job #177309](https://gricad-gitlab.univ-grenoble-alpes.fr/CamiTK/CamiTK/-/jobs/177309) ## Actual VS Expected Result The pipeline should fail when there is a failed test, independently of the type of failure. ## Relevant logs and/or screenshots ### On Linux stable In the [Job #177308](https://gricad-gitlab.univ-grenoble-alpes.fr/CamiTK/CamiTK/-/jobs/177308) log, the initial tests failed with: ``` 4/5 Test #132: application-actionstatemachine-5 .........Child aborted***Exception: 2.81 sec 5/5 Test #131: application-actionstatemachine-4 .........***Timeout 1800.13 sec ``` But then at the end of the job log we have: ``` ===== Fatal errors ===== Found fatal error in /builds/CamiTK/CamiTK/debian-stable-develop/log/ci-test.log ===== Failed tests ===== ===== Re-running failed tests ===== Re-ran tests OK ``` The relevant lines in the test.log show: ``` 132: X Error of failed request: BadWindow (invalid Window parameter) 132: Major opcode of failed request: 2 (X_ChangeWindowAttributes) 132: Resource id in failed request: 0x2 132: Serial number of failed request: 7 132: Current serial number in output stream: 9 132: terminate called after throwing an instance of 'std::bad_array_new_length' 132: what(): std::bad_array_new_length 4/5 Test #132: application-actionstatemachine-5 .........Child aborted***Exception: 2.81 sec 5/5 Test #131: application-actionstatemachine-4 .........***Timeout 1800.13 sec ``` The rerun-test.log contains only one line: ``` Checking /builds/CamiTK/CamiTK/debian-stable-develop/log/test.log for failed test ``` ### On Linux LTS In the [Job #177309](https://gricad-gitlab.univ-grenoble-alpes.fr/CamiTK/CamiTK/-/jobs/177309) log, the initial tests failed with: ``` 4/5 Test #131: application-actionstatemachine-4 .........Child aborted***Exception: 0.71 sec 5/5 Test #132: application-actionstatemachine-5 .........***Timeout 1800.13 sec ``` But then at the end of the job log we have: ``` ===== Fatal errors ===== Found fatal error in /builds/CamiTK/CamiTK/ubuntu-lts-develop/log/ci-test.log ===== Failed tests ===== ===== Re-running failed tests ===== Re-ran tests OK ``` The relevant lines in the test.log show: ``` 131: ... 131: X Error of failed request: BadWindow (invalid Window parameter) 131: Major opcode of failed request: 2 (X_ChangeWindowAttributes) 131: Resource id in failed request: 0x2 131: Serial number of failed request: 7 131: Current serial number in output stream: 9 131: terminate called after throwing an instance of 'std::bad_alloc' 131: what(): std::bad_alloc 4/5 Test #131: application-actionstatemachine-4 .........Child aborted***Exception: 0.71 sec 5/5 Test #132: application-actionstatemachine-5 .........***Timeout 1800.13 sec ``` The rerun-test.log contains only one line: ``` Checking /builds/CamiTK/CamiTK/ubuntu-lts-develop/log/test.log for failed test ``` ## Interpretation & Possible fixes The test script is faulty, see `.gitlab/test.sh`. It should be better at parsing the log to detect problems ## CamiTK Version CamiTK 4.2.dev.develop.92087062 Compiled using git Hash: 92087062bc6533f58297c18826f5256436b37807, Date: Tue Oct 20 00:24:15 2020 +0200.