cmake: Add new DISCOVERY_MODE option to catch_discover_tests

[hkaelber]

Description

Provide DISCOVERY_MODE option to catch_discover_tests to allow for PRE_TEST discovery mode.

Cf. details in commit msg-es.

This change follows closely the corresponding functionality / changes in the GoogleTest counterpart to maintain the convenient similarity to gtest_discover_tests() and friends.

GitHub Issues

Closes #2493

[codecov]

Codecov Report

Merging #2670 (3211fac) into devel (9a2a4ea) will increase coverage by 0.01%.
The diff coverage is n/a.

❗ Current head 3211fac differs from pull request most recent head d03fafe. Consider uploading reports for the commit d03fafe to get more accurate results

@@            Coverage Diff             @@
##            devel    #2670      +/-   ##
==========================================
+ Coverage   91.24%   91.25%   +0.01%     
==========================================
  Files         192      192              
  Lines        7808     7808              
==========================================
+ Hits         7124     7125       +1     
+ Misses        684      683       -1     

[hkaelber]

Sorry, forgot that you prefer fixup commits in your review process. But shoudl not be a big deal in this PR, pls cf. incremental diff here

[horenmar]

LGTM, thanks.

[riandrake]

Hi, sorry if this is the wrong place. When trying this, I get an error in CMake:

CMake Error in CMakeLists.txt:
  Evaluation file to be written multiple times with different content.  This
  is generally caused by the content evaluating the configuration type,
  language, or location of object files:

Attached is my usage:

I'm using CLion with Ninja Multi-Config.

Are there limitations I'm not aware of?

[hkaelber]

Hi, sorry if this is the wrong place. When trying this, I get an error in CMake:

CMake Error in CMakeLists.txt:
  Evaluation file to be written multiple times with different content.  This
  is generally caused by the content evaluating the configuration type,
  language, or location of object files:

Attached is my usage:

I'm using CLion with Ninja Multi-Config.

Are there limitations I'm not aware of?

Hi @riandrake, thx for spotting this.

Indeed currently multi-config generators are not supported with PRE_TEST discovery because I ignored them when porting this because -- of lazyness, as we don't have support for Ninja Multi-Config in our primay build-env and I could not easily test it. :-)

Could you maybe test #2739? Works for me so far, though I still miss testing all combinations. Thx

[riandrake]

@hkaelber this worked, thanks! I have a question though.

I noted that it didn't collect the tests until PRE_TEST time as expected, but this happened only once. My hope was that I could change the TEST_SPEC variable between runs and have it collect the tests again for targeted testing.

Am I misunderstanding how PRE_TEST works? If so, could I feature request this additional functionality somewhere?

[hkaelber]

@hkaelber this worked, thanks! I have a question though.

I noted that it didn't collect the tests until PRE_TEST time as expected, but this happened only once. My hope was that I could change the TEST_SPEC variable between runs and have it collect the tests again for targeted testing.

Am I misunderstanding how PRE_TEST works? If so, could I feature request this additional functionality somewhere?

Not sure I understood your point, @riandrake.

I've never used multi-config so far, but to my understanding

1. at configure/cmake time the config-specifc test includes are generated (containing the postponed discovery command) together with the main include file, that simply pulls in the config-specific test-include
2. at every ctest invocation (per concrete config, Debug, Release, etc) the actual config-specific test file should be generated and executed

Correct? (Though I'm 100% sure how ctest is supposed to be executed in a multi-config env.)
That means that the TEST_SPEC is already fixed in 1. for all configs.

[riandrake]

Understood, thank you.

…

On Mon, 11 Sep 2023 at 03:10, Holger Kaelberer ***@***.***> wrote: @hkaelber <https://github.com/hkaelber> this worked, thanks! I have a question though. I noted that it didn't collect the tests until PRE_TEST time as expected, but this happened only once. My hope was that I could change the TEST_SPEC variable between runs and have it collect the tests again for targeted testing. Am I misunderstanding how PRE_TEST works? If so, could I feature request this additional functionality somewhere? Not sure I understood your point, @riandrake <https://github.com/riandrake>. I've never used multi-config so far, but to my understanding 1. at configure/cmake time the config-specifc test includes are generated (containing the postponed discovery command) together with the main include file, that simply pulls in the config-specific test-include 2. at every ctest invocation (per concrete config, Debug, Release, etc) the actual config-specific test file should be generated and executed Correct? (Though I'm 100% sure how ctest is supposed to be executed in a multi-config env.) That means that the TEST_SPEC is already fixed in 1. for all configs. — Reply to this email directly, view it on GitHub <#2670 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABMYESQLTLRLWLKQ76H7AFTXZXJ6PANCNFSM6AAAAAAW2BI2K4> . You are receiving this because you were mentioned.Message ID: ***@***.***>