Failure in ctest: Required regular expression not found

[AdrianTianyi]

Hi, I am now building and installing TriBITS. I follow the README.rst. However, when running ctest, 11 tests fail with the same reason:
Required regular expression not found.
The failed tests are
172 - TriBITS_TribitsHelloWorld_CONFIGURE_OPTIONS_FILE (Failed)
237 - TriBITS_gitdist_UnitTests (Failed)
329 - TriBITS_CTestDriver_PBP_ST_ALL_PASS (Failed)
331 - TriBITS_CTestDriver_PBP_ST_BreakConfigureOptionalPkg (Failed)
332 - TriBITS_CTestDriver_PBP_ST_BreakBuildLibOptionalPkg (Failed)
333 - TriBITS_CTestDriver_PBP_ST_BreakBuildAllOptionalPkg (Failed)
334 - TriBITS_CTestDriver_PBP_ST_BreakTestPkg (Failed)
335 - TriBITS_CTestDriver_PBP_ST_BreakConfigureRequiredPkg (Failed)
340 - TriBITS_CTestDriver_PBP_PT_ALL_PASS_CALLS_2 (Failed)
341 - TriBITS_CTestDriver_PBP_PT_ALL_PASS_CALLS_4 (Failed)
361 - TriBITS_CTestDriver_AAOP_CTEST_DO_INSTALL_FAIL_bad_intall_filename (Failed)
What's the possible reason?
Thanks a lot!

[bartlettroscoe]

Hello @AdrianTianyi,

What version of CMake is this? Can you run make dashboard and post to CDash so we can see the failures on the TriBITS CDash Site? That should show me everything I need to see.

[AdrianTianyi]

Thanks very much for your reply! My CMake version is 3.21.0-rc2.

It is not convenient for me to upload the whole log file. The followings are some possible key points in the log file. If you need other specific contents, let me know.

In configuring:
Probing the environment ... -- USE_XSDK_DEFAULTS='FALSE' -- BUILD_SHARED_LIBS='ON' -- Setting CMAKE_BUILD_TYPE=RELEASE since it was not set ... -- CMAKE_BUILD_TYPE='RELEASE' -- The C compiler identification is GNU 7.5.0 ... -- Check for working C compiler: /usr/bin/gcc - skipped ... -- CMAKE_C_COMPILER_ID='GNU' -- CMAKE_C_COMPILER_VERSION='7.5.0' -- The CXX compiler identification is GNU 7.5.0 -- Check for working CXX compiler: /usr/bin/c++ - skipped ... -- CMAKE_CXX_COMPILER_ID='GNU' -- CMAKE_CXX_COMPILER_VERSION='7.5.0' -- TribitsExProj_SET_INSTALL_RPATH='OFF' -- CMAKE_INSTALL_RPATH_USE_LINK_PATH='TRUE' -- CMAKE_INSTALL_RPATH='' ... -- Found Perl: /usr/bin/perl (found version "5.26.1") ... -- Performing Test FINITE_VALUE_HAVE_GLOBAL_ISNAN -- Performing Test FINITE_VALUE_HAVE_GLOBAL_ISNAN - Failed ... -- Performing Test FINITE_VALUE_HAVE_GLOBAL_ISINF -- Performing Test FINITE_VALUE_HAVE_GLOBAL_ISINF - Failed ... -- Found Doxygen: /usr/bin/doxygen (found version "1.8.13") found components: doxygen missing components: dot
In testing:

`1: OVERALL FINAL RESULT: TEST FAILED (TAATDriver_TAAT_COPY_FILES_TO_TEST_DIR_bad_file_name)
1:
1: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
1:
1/1 Test #1: TAATDriver_TAAT_COPY_FILES_TO_TEST_DIR_bad_file_name ...***Failed Required regular expression not found. Regex=[OVERALL FINAL RESULT: TEST PASSED .TAATDriver_TAAT_COPY_FILES_TO_TEST_DIR_bad_file_name.
] 0.03 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) = 0.03 sec

The following tests FAILED:
1 - TAATDriver_TAAT_COPY_FILES_TO_TEST_DIR_bad_file_name (Failed)
Errors while running CTest
Output from these tests are in: /home/Adrian/gitspace/BUILD/test/core/CTestScriptsUnitTests/TriBITS_CTestScripts_TAAT_COPY_FILES_TO_TEST_DIR_bad_file_name_DRIVER/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.

TEST_2: Return code = 8
TEST_2: Pass criteria = Match REGEX {Error copying file .+/TAATDriver/bad_file_name/this_file_does_not_exist.txt. to .+/TriBITS_CTestScripts_TAAT_COPY_FILES_TO_TEST_DIR_bad_file_name_DRIVER/TAATDriver_TAAT_COPY_FILES_TO_TEST_DIR_bad_file_name/} [PASSED]
TEST_2: Pass criteria = Match REGEX {TEST_0: Result = FAILED} [PASSED]
TEST_2: Pass criteria = Match REGEX {TEST_1: Result = PASSED} [PASSED]
TEST_2: Pass criteria = Match REGEX {This is Test File A} [PASSED]
TEST_2: Pass criteria = Match REGEX {TEST_2: Return code = 0} [PASSED]
TEST_2: Pass criteria = Match REGEX {TEST_2: Pass criteria = Match REGEX .This is Test File A. .PASSED.} [PASSED]
TEST_2: Pass criteria = Match REGEX {TEST_2: Result = PASSED} [PASSED]
TEST_2: Pass criteria = Match REGEX {OVERALL FINAL RESULT: TEST FAILED [(]TAATDriver_TAAT_COPY_FILES_TO_TEST_DIR_bad_file_name[)]} [PASSED]
TEST_2: Result = PASSED`

In another part:

`
OVERALL FINAL RESULT: TEST PASSED (TriBITS_DepTestReduced_EnableEpetra_EnableRTOp_DisableTeuchos)

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Test time = 0.14 sec ---------------------------------------------------------- Test Pass Reason: Required regular expression found. Regex=[OVERALL FINAL RESULT: TEST PASSED .TriBITS_DepTestReduced_EnableEpetra_EnableRTOp_DisableTeuchos.] "TriBITS_DepTestReduced_EnableEpetra_EnableRTOp_DisableTeuchos" end time: Jul 23 14:36 CST "TriBITS_DepTestReduced_EnableEpetra_EnableRTOp_DisableTeuchos" time elapsed: 00:00:00 ----------------------------------------------------------
67/364 Testing: TriBITS_CTestScripts_TAAT_COPY_FILES_TO_TEST_DIR_bad_dest_dir_DRIVER
67/364 Test: TriBITS_CTestScripts_TAAT_COPY_FILES_TO_TEST_DIR_bad_dest_dir_DRIVER
Command: "/usr/local/bin/cmake" "-DTEST_CONFIG=${CTEST_CONFIGURATION_TYPE}" "-P" "/home/Adrian/gitspace/BUILD/test/core/CTestScriptsUnitTests/TriBITS_CTestScripts_TAAT_COPY_FILES_TO_TEST_DIR_bad_dest_dir_DRIVER.cmake"
Directory: /home/Adrian/gitspace/BUILD/test/core/CTestScriptsUnitTests
"TriBITS_CTestScripts_TAAT_COPY_FILES_TO_TEST_DIR_bad_dest_dir_DRIVER" start time: Jul 23 14:36 CST
Output:
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Advanced Test: TriBITS_CTestScripts_TAAT_COPY_FILES_TO_TEST_DIR_bad_dest_dir_DRIVER

Selected Test/CTest Propeties:
`

1: ================================================================================ 1: 1: TEST_2 1: 1: Copy files from: 1: /home/Adrian/gitspace/TriBITS/test/core/CTestScriptsUnitTests/test_data/ 1: to: 1: /home/Adrian/gitspace/BUILD/test/core/CTestScriptsUnitTests/TriBITS_CTestScripts_TAAT_COPY_FILES_TO_TEST_DIR_bad_dest_dir_DRIVER/TAATDriver_TAAT_COPY_FILES_TO_TEST_DIR_bad_dest_dir/read_only_dir/subdir/ 1: 1: -------------------------------------------------------------------------------- 1: 1: Creating dest directory /home/Adrian/gitspace/BUILD/test/core/CTestScriptsUnitTests/TriBITS_CTestScripts_TAAT_COPY_FILES_TO_TEST_DIR_bad_dest_dir_DRIVER/TAATDriver_TAAT_COPY_FILES_TO_TEST_DIR_bad_dest_dir/read_only_dir/subdir/ ... 1: Error creating directory "/home/Adrian/gitspace/BUILD/test/core/CTestScriptsUnitTests/TriBITS_CTestScripts_TAAT_COPY_FILES_TO_TEST_DIR_bad_dest_dir_DRIVER/TAATDriver_TAAT_COPY_FILES_TO_TEST_DIR_bad_dest_dir/read_only_dir/subdir". 1: Copy file test_file_b.txt ... 1: Error copying file "/home/Adrian/gitspace/TriBITS/test/core/CTestScriptsUnitTests/test_data/test_file_b.txt" to "/home/Adrian/gitspace/BUILD/test/core/CTestScriptsUnitTests/TriBITS_CTestScripts_TAAT_COPY_FILES_TO_TEST_DIR_bad_dest_dir_DRIVER/TAATDriver_TAAT_COPY_FILES_TO_TEST_DIR_bad_dest_dir/read_only_dir/subdir/". 1: 1: -------------------------------------------------------------------------------- 1: 1: 1: TEST_2: Result = FAILED 1: 1: ================================================================================

`
TEST_2: Return code = 8
TEST_2: Pass criteria = Match REGEX {Running: .mkdir. .read_only_dir.} [PASSED]
TEST_2: Pass criteria = Match REGEX {TEST_0: Result = PASSED} [PASSED]
TEST_2: Pass criteria = Match REGEX {Running: .chmod. .a-w. .read_only_dir.} [PASSED]
TEST_2: Pass criteria = Match REGEX {TEST_1: Result = PASSED} [PASSED]
TEST_2: Pass criteria = Match REGEX {Creating dest directory ./TriBITS_CTestScripts_TAAT_COPY_FILES_TO_TEST_DIR_bad_dest_dir_DRIVER/TAATDriver_TAAT_COPY_FILES_TO_TEST_DIR_bad_dest_dir/read_only_dir/subdir/} [PASSED]
TEST_2: Pass criteria = Match REGEX {Error .ing directory ./TriBITS_CTestScripts_TAAT_COPY_FILES_TO_TEST_DIR_bad_dest_dir_DRIVER/TAATDriver_TAAT_COPY_FILES_TO_TEST_DIR_bad_dest_dir/read_only_dir/subdir.[.]} [PASSED]
TEST_2: Pass criteria = Match REGEX {Error copying file ./CTestScriptsUnitTests/test_data/test_file_b.txt. to .*/TriBITS_CTestScripts_TAAT_COPY_FILES_TO_TEST_DIR_bad_dest_dir_DRIVER/TAATDriver_TAAT_COPY_FILES_TO_TEST_DIR_bad_dest_dir/read_only_dir/subdir/.} [PASSED]
TEST_2: Pass criteria = Match REGEX {TEST_2: Result = FAILED} [PASSED]
TEST_2: Pass criteria = Match REGEX {TEST_3: Result = PASSED} [PASSED]
TEST_2: Pass criteria = Match REGEX {This is Test File A} [PASSED]
TEST_2: Pass criteria = Match REGEX {TEST_4: Return code = 0} [PASSED]
TEST_2: Pass criteria = Match REGEX {TEST_4: Pass criteria = Match REGEX .This is Test File A. .PASSED.} [PASSED]
TEST_2: Pass criteria = Match REGEX {TEST_4: Result = PASSED} [PASSED]
TEST_2: Pass criteria = Match REGEX {OVERALL FINAL RESULT: TEST FAILED [(]TAATDriver_TAAT_COPY_FILES_TO_TEST_DIR_bad_dest_dir[)]} [PASSED]
TEST_2: Result = PASSED

================================================================================

OVERALL FINAL RESULT: TEST PASSED (TriBITS_CTestScripts_TAAT_COPY_FILES_TO_TEST_DIR_bad_dest_dir_DRIVER)

`

30898 DummyProj = DummyProj 30899 DummyProj_TRIBITS_DIR = /home/Adrian/gitspace/TriBITS/tribits 30900 -- TPL_NAME='HeaderOnlyTpl' 30901 -- TPL_HeaderOnlyTpl_ENABLING_PKG='SimpleCxx' 30902 -- HeaderOnlyTpl_FINDMOD='/home/Adrian/gitspace/TriBITS/tribits/examples/TribitsExampleProject/cmake/tpls/FindTPLHeaderOnlyTpl.cmake' 30903 30904 Processing enabled TPL: HeaderOnlyTpl (enabled by SimpleCxx, disable with -DTPL_ENABLE_HeaderOnlyTpl=OFF) 30905 -- Must find at least one header in each of the header sets "HeaderOnlyTpl_stuff.hpp" 30906 -- Searching for headers in HeaderOnlyTpl_INCLUDE_DIRS='/path_does_not_exist' 30907 -- Searching for a header file in the set "HeaderOnlyTpl_stuff.hpp": 30908 -- Searching for header 'HeaderOnlyTpl_stuff.hpp' ... 30909 -- ERROR: Could not find a header file in the set "HeaderOnlyTpl_stuff.hpp" 30910 -- ERROR: Could not find the include directories for TPL 'HeaderOnlyTpl'! 30911 -- TIP: If the TPL 'HeaderOnlyTpl' is on your system then you can set: 30912 -DHeaderOnlyTpl_INCLUDE_DIRS='<dir0>;<dir1>;...' 30913 to point to directories where these header files may be found. 30914 Or, just set: 30915 -DTPL_HeaderOnlyTpl_INCLUDE_DIRS='<dir0>;<dir1>;...' 30916 to point to the include directories which will bypass any search for 30917 header files and these include directories will be used without 30918 question in the build. (But this will result in a build-time error 30919 obviously if the necessary header files are not found in these 30920 include directories.) 30921 -- ERROR: Failed finding all of the parts of TPL 'HeaderOnlyTpl' (see above), Aborting! 30922 30923 -- NOTE: The find module file for this failed TPL 'HeaderOnlyTpl' is: 30924 /home/Adrian/gitspace/TriBITS/tribits/examples/TribitsExampleProject/cmake/tpls/FindTPLHeaderOnlyTpl.cmake 30925 which is pointed to in the file: 30926 30927 30928 TIP: One way to get past the configure failure for the 30929 TPL 'HeaderOnlyTpl' is to simply disable it with: 30930 -DTPL_ENABLE_HeaderOnlyTpl=OFF 30931 which will disable it and will recursively disable all of the 30932 downstream packages that have required dependencies on it, including 30933 the package 'SimpleCxx' which triggered its enable. 30934 When you reconfigure, just grep the cmake stdout for 'HeaderOnlyTpl' 30935 and then follow the disables that occur as a result to see what impact 30936 this TPL disable has on the configuration of DummyProj. 30937 30938 CMake Error at /home/Adrian/gitspace/TriBITS/tribits/core/package_arch/TribitsProcessEnabledTpl.cmake:144 (message): 30939 ERROR: TPL_HeaderOnlyTpl_NOT_FOUND=TRUE, aborting! 30940 Call Stack (most recent call first): 30941 CMakeLists.txt:75 (TRIBITS_PROCESS_ENABLED_TPL) 30942 30943 30944 -- Configuring incomplete, errors occurred!

[bartlettroscoe]

@AdrianTianyi,

My CMake version is 3.21.0-rc2.

This is what I suspected. These are known failures that were encountered as part of #363 in adding testing of TriBITS against CMake 3.21. The failures in:

172 - TriBITS_TribitsHelloWorld_CONFIGURE_OPTIONS_FILE (Failed)
361 - TriBITS_CTestDriver_AAOP_CTEST_DO_INSTALL_FAIL_bad_intall_filename (Failed)

were addressed in PR #392 and pushed to 'master' a few days ago. I have not seen any failures in the test TriBITS_gitdist_UnitTests so I am surprised by that.

The rest of the failing tests appear to be due to changes in behavior (i.e. likely regressions) in CMake 3.21 compared to prior versions of CMake as demonstrated and explained in the demonstration PR #394. I have a ticket open with Kitware to work through these issues (and I just met with them this morning about this actually). I don't think these are TriBITS defects if you look at the evidence in that PR.

From my analysis of these failures is that they seem to be isolated to ctest -S invocations and may not be impacting basic usage of CMake. Therefore, I would not worry too much about these (but it is important for Kitware to resolve this for some customers that use CMake and TriBITS).

NOTE: You might consider using CMake 3.21.0 now that it has been released instead of a release candidate 3.21.0-rc2

It is not convenient for me to upload the whole log file.

No, after configuring the TriBITS CMake project, you just literally run the command make dashboard and walk away and it takes care of everything in running the build and tests and submitting results to CDash. Can you run that command? It would be nice to see why the test TriBITS_gitdist_UnitTests is failing.

[AdrianTianyi]

Thanks very much for your kindly reply and valuable advice. I will try this approach to see whether it is this issue, and then return to you.
For the command make dashboard, I ran this command but the connection failured (I ran in a server, not my PC). I will show you the log file part for the failure of TriBITS_gitdist_UnitTests later.

[AdrianTianyi]

Hi. I just try to install TriBITS in my own PC with CMake 3.21.0-rc2 and CMake-3.21.0. However, both of them failed, though TriBITS_gitdist_UnitTests does not fail in both CMake versions. I have used make dashboard to upload the logs.
So, if it is all because of the version, I can downgrade cmake version. Is there a safe cmake version? I will try 3.20 first.

[AdrianTianyi]

By the way, since most of the tests succeed, if I ignore these failures, will the basic usage of TriBITS be influenced?

[AdrianTianyi]

Hi, I have tried several new combinations of the versions of cmake and g++,gcc, gfortran versions. However, all of them failed, which is very frustrated.
I have used make dashboard command to upload most of the tests. The versions I have tried are cmake 3.21.0, cmake 3.20.5, cmake 3.17.4, cmake 3.17.5, g++ 9.4, g++ 7.5, gcc 9.4, gcc 7.5, gfortran 7.5, gfortran 9.4.

[bartlettroscoe]

@AdrianTianyi, thanks for posting the configure, build, and test test results for different CMake versions to CDash with make dashboard. We can see these results at:

• https://testing.sandia.gov/cdash/index.php?project=TriBITS&filtercount=2&showfilters=1&filtercombine=and&field1=site&compare1=61&value1=tianyilu&field2=buildstarttime&compare2=84&value2=now

The tests all pass for CMake 3.17.5 except for two failing tests as shown here showing:

Name	Status	Time	Details	History	Summary
TriBITS_TribitsExampleProject_ALL_ST_NoFortran	Failed	4s 870ms	Completed (Failed)	Broken	Broken
TriBITS_TribitsHelloWorld_install_perms	Failed	2s 720ms	Completed (Failed)	Broken	Broken

It looks like those two tests fail due to non-standard output from commands like ls and diff (showing some non-English chars). So except for those non-standard ls and diff output those TriBITS tests look great.

I just try to install TriBITS in my own PC

When you say "PC", do you mean with native Windows or with the Windows Subsystem for Linux (WSL)?

As for "installing" TriBITS, it is just a set of CMake modules so all you do is just copy the TriBITS/tribits subdir somehwere or snapshot it into your CMake project (see TriBITS directory snapshotting).

[bartlettroscoe]

Hi, I have tried several new combinations of the versions of cmake and g++,gcc, gfortran versions. However, all of them failed, which is very frustrated.

@AdrianTianyi, sorry for any frustration this has caused you and I appreciate your patience here. The issue is that many of the TriBITS tests are very strong but in being strong, some of them can be a bit fragile by needing to depend on the behavior of the underlying system tools like the build tools make and ninja and tools like ls, diff, and others. (In the case of of your CMake 3.17.5 build, it was non-standard behavior of ls and diff on your system that broke those two tests.) Therefore, when porting to different platforms with different tools and upgraded compilers, etc. it is expected to see some failing tests and that does not necessarily mean these failures point to defects in TriBITS. For example, when testing against CMake 3.21.0 in #363, there were a few changes in CMake that changed its behavior in acceptable ways that required some tweaks to the TriBITS test suite to accommodate those changes in behavior (see PR #392). This is the double-edge sword of a stronger test suite: the stronger the tests, the more likely it will catch regressions but the more maintenance it will need as time goes on.

[AdrianTianyi]

Thanks a lot for your reply! I fully understand the reason and necessity for the strong tests.

After several times tries, I finally succeed in passing all ctests in my PC( a linux machine). The final setting is python3.8.8 , gcc 9.4, g++ 9.4 and cmake 3.18.6. Furthermore, I changed the language of the system to English. This makes all ctests succeed.

Meanwhile, I also test on the server. The gcc, g++ versions are both 7.4 and cmake 3.18.6. Then all the tests also pass. (The only change on the server is the version of cmake.)
This is a very good news for me.

[bartlettroscoe]

This is a very good news for me.

@AdrianTianyi, you getting all of the tests to pass, can we close this issue?

[AdrianTianyi]

Thanks. Let's close this issue.

[bartlettroscoe]

Thanks @AdrianTianyi!