Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tribits_add[_advanced]_test(): Address handling of semi-colons in ARGS and ENVIRONMENT (#299) #464

Conversation

bartlettroscoe
Copy link
Member

@bartlettroscoe bartlettroscoe commented Mar 11, 2022

This PR makes the tribits_add_advanced_test() function handle arguments that need to include a semi-colon ';' without messing up the list handling in CMake. This introduces a new top-level argument LIST_SEPARATOR <sep> used exactly the same way as the LIST_SEPARATOR argument with ExternalProject_Add(). Apparently, this is an accepted workaround for the problem of lists elements containing semi-colons with CMake.

I need to be able to handle semi-colons with tribits_add_advanced_test() for the new tests I am developing for #299.

Note that this PR contains an earlier commit that allowed the user to use quoted args that contained ';' and silently replaced that with <semicolon> but I think it is better to follow the convention from ExternalProject_Add() and make this explicit (even if it is a bit of a hack). I spent many hours experimenting with ways to deal with these arguments with failure after failure but I wanted to document this one way I found that worked with a commit. (It was after I asked Brad King that he mentioned the ExternalProject_Add() way of using LIST_SEPARATOR <sep> to workaround this problem and I refactored to use that approach. The core CMake language is really just broken when it comes to dealing with general lists with spaces and semi-colons.)

NOTE: I also fixed checking for unparsed args to tribits_add_advanced_test() that were not being checked before and this technically breaks backward compatibility. (I had to fix one of the TriBITS tests that had an unparsed/ignored argument so I am sure this is going to break Trilinos once this gets merged to Trilinos 'develop'.)

See the individual commit log messages for more details.

…ning() (TriBITSPub#299)

With a newer version of CMake, we can use DEPRECATION instead of WARNING.
TriBITSPub#299)

This now allows arguments to 'ARGS' to contain semi-colons ';'.  To do this,
it adds a new function with unit tests tribits_parse_arguments_for_list() to
deal with these cases.

A number of other changes in this commit include:

* Fixed tribits_add_advanced_test() to actually check and report unused args
  (this did not work before) and added more tests to show this this.  (I had
  to fix one TriBITS test
  TriBITS_CTestScripts_cmnd_3_args_1_overall_working_directory defined with
  tribits_add_advanced_test() that was using MESSAGE incorrectly.)

* Fixed setting and testing with <Project>_CHECK_FOR_UNPARSED_ARGUMENTS
  CUSTOM_FAILURE_MODE and added more automated tests to show this.

* Renamed the test
  TriBITS_TribitsExampleProject_PkgWithUserErrors_UNPARSED_ARGUMENTS and the
  the name _UNPARSED_ARGS_ to _UNPARSED_ARGUMENTS_ for consistancy.

* Added tribits_parse_arguments_in_list()

* Added tribits_copy_list()

* Added unittest_compare_list_ele_const()
…c args with semi-colons (TriBITSPub#299)

This is a better solution that is consistent with how ExternalProject_Add()
gets around this problem.

What I did:

* Added explicit LIST_SEPARATOR to tribits_add_advanced_test()

* Removed usage of tribits_parse_arguments_in_list() (it is no longer needed)

* Moved top-level arguments to cmake_parse_arguments() into their own
  self-documenting variables that will be easier to maintain.
…#299)

There is no need for these anymore now that an explicit LIST_SEPARATOR is used
from the top down.  Parsing and everything happens just fine.
@bartlettroscoe bartlettroscoe force-pushed the 299-tribits_add_advanced_test_semicolons branch 2 times, most recently from f02f4a1 to 47bc34f Compare March 11, 2022 01:04
TriBITSPub#299)

Now I will be able to set env vars that have semi-colons, like
CMAKE_PREFIX_PATH.

NOTE: I had to use the:

  cmake_parse_arguments(PARSE_ARGV 0 FWD "" "" "")

tick to get the arugments forwarded correctly that I learned from
"Professional CMake".
@bartlettroscoe
Copy link
Member Author

Hello @MikolajZuzek and @mperrinel. Can one of you review this PR in detail? I don't think both of you need to review the entire PR. But you both might look at changes to these three files:

@bartlettroscoe
Copy link
Member Author

FYI: More discussion about how to handle semi-colons ; in list elements in:

@bartlettroscoe
Copy link
Member Author

@MikolajZuzek and @mperrinel,

BTW, please let me know (in a comment in this PR) when you have started the review. I am seeing a few things that need addressed so I will be adding a new commit or two to address the issues.

…NT (TriBITSPub#299)

This is to complement the same changes made to tribits_add_advanced_test().

I also added some missing tests for tribits_add_advanced_test() and updated
CHANGELOG.txt for more info.
…EPARATOR (TriBITSPub#299)

NOTE: I reversed the arguments to tribits_add_executable_and_test() in the
unit test since when I first added LIST_SEPARATOR <sep> after TIMEOUT 1.5, it
passed LIST_SEPARATOR <sep> as elements in TIMEOUT and passed my check at the
end.  By reversing the argments, this did not happen and the test failed
again.
@bartlettroscoe bartlettroscoe changed the title tribits_add_advanced_test(): Address handling of semi-colons in ARGS and ENVIRONMENT (#299) tribits_add[_advanced]_test(): Address handling of semi-colons in ARGS and ENVIRONMENT (#299) Mar 12, 2022
@bartlettroscoe
Copy link
Member Author

@MikolajZuzek and @mperrinel,

FYI, I added some new commits to round this out. I don't think I will be adding any more new commits (unless typos or other problems are found).

@mperrinel
Copy link
Collaborator

@bartlettroscoe I am starting the review on it

@mzuzek
Copy link
Collaborator

mzuzek commented Mar 16, 2022

Reviewing this as well

Copy link
Collaborator

@mperrinel mperrinel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was a technical one. The code seems good for me. I found a lot of TYPO mistakes. Will create a PR for that.

@bartlettroscoe
Copy link
Member Author

@mperrinel, thanks for the review. Yes, please put in a PR with the fixes. (Looks like I forgot to 'flyspell' when I edited these files.)

@mzuzek
Copy link
Collaborator

mzuzek commented Mar 17, 2022

@bartlettroscoe @mperrinel

Is LIST_SEPARATOR necessary to solve the underlying problem ?

I was able to get rid of it and make all [new] *semicolon* test pass by fixing some quotes here and there, so that embedded lists are properly carried out to PARENT_SCOPE, etc. Please see b2f0a41 and cd84710 on my 464_fix-test-args-with-semicolons test branch, which builds on top of current head of this PR.

It's more of a PoC than complete fix - there is still handful of tests failing.
Let me know if that makes any sense or am I missing something crucial ?

…po_fixes

Fixed typo mistake seen in PR 464
@bartlettroscoe
Copy link
Member Author

Thanks for the review @mperrinel and fixing the typos. No problems with any of the source code or tests?

@MikolajZuzek, note the final conclusions on the handling of semicolon ; char data in lists and function and macro arguments in:

@bradking did not refute these conclusions and added some CMake documentation MRs to officially document the workaround LIST_SEPARATOR as the only robust way to deal with data containing semi-colon ; chars.

However, it just occurred to me that we could consider using the generator expression $<SEMICOLON> with tribits_add_test() and tribits_add_advanced_test() since the final interpretation of this data is not needed until running the actual tests and therefore could call all commands that run at generation time and accept generator expressions (like add_test(), set_test_properties() and file(GENERATE)). See:

@MikolajZuzek, please don't just jump in and try to implement this. Let's wait for the response back from @bradking in that CMake GitLab Issue. Then, trying this should be fairly easy to try out given the tip of this existing branch (but adapting for the unit tests will be a little tricky since those need to generate the <testName>.cmake file in a cmake -P script).

@mzuzek
Copy link
Collaborator

mzuzek commented Mar 21, 2022

As you can see, the changes you made in this branch have broken backward compatibility. The fact that you had to quote a bunch of arguments all over the place is evidence of that. That is unacceptable.

@bartlettroscoe Thank you for the feedback. Here's the fixed version that greens out all relevant tests without changing their content (except for LIST_SEPARATOR retraction): 0001-remove-LIST_SEPARATOR-and-fix-list-passing.patch.zip

If you see any break in backward compatibility or any other issue, please expose it via unit test.

@mzuzek
Copy link
Collaborator

mzuzek commented Mar 21, 2022

@bartlettroscoe
DISCLAIMER: Please use any code presented by me to support the review discussions and focus it on relevant technical details with extreme care: it's barely a side experiment built on top of the implementation brought in by this PR, which helps me understand the problem being solved and the underlying CMake limitations causing it.

Copy link
Collaborator

@mzuzek mzuzek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bartlettroscoe @keitat @mperrinel

Looks good. The fix is consistent with semicolon handling in CMake and the test coverage is great!

It could be worthy to also explore handling problematic lists as strings to avoid complicating TriBITS API with LIST_SEPARATOR - see the discussion above starting at #464 (comment).

@mzuzek
Copy link
Collaborator

mzuzek commented Mar 21, 2022

@MikolajZuzek, please don't just jump in and try to implement this. Let's wait for the response back from @bradking in that CMake GitLab Issue. Then, trying this should be fairly easy to try out given the tip of this existing branch (but adapting for the unit tests will be a little tricky since those need to generate the .cmake file in a cmake -P script).

Of course. Let me know when the PR is ready for re-review.

@bartlettroscoe
Copy link
Member Author

Here's the fixed version that greens out all relevant tests without changing their content (except for LIST_SEPARATOR retraction): 0001-remove-LIST_SEPARATOR-and-fix-list-passing.patch.zip

@MikolajZuzek, can you please post that as a WIP PR so it is easier to see the changes?

However, Kitware's feedback up to this point (from multiple CMake maintainers) is that trying to keep raw semi-colons ; inside of CMake lists is not going to work in general. In fact, CMake does not even handle $<SEMICOLON> correctly with CTest properties as per:

so my idea to try out $<SEMICOLON> above is a no go.

Also, see recent comments:

The only robust way to deal with semi-colon ; chars as data to pass around in list elements is to not do that. Hence the LIST_SEPRATOR workaround with specific translations at final point of usage is the only robust way to deal with data in lists that contains semi-colon ; chars . I don't know why we would be arguing with Kitware at this point. They know more about this topic than all of us on the issue put together.

And now the LIST_SEPRATOR workaround is now the officially recommended CMake policy as per:

As noted by @mathstuf in:

All in all, I don't know how feasible it is to shuttle semicolons to any arbitrary corner of CMake's workings without fundamentally rethinking CMake's type representations (which touch compatibility points all along the way, but maybe could be smuggled around somehow).

Getting even the internals of CMake to deal with raw semi-colon ; data in lists is still an open question. So it is time give up and just accept the LIST_SEPRATOR workaround in the PR and move on.

@bradking
Copy link
Contributor

The only robust way to deal with semi-colon ; chars as data to pass around in list elements is to not do that. Hence the LIST_SEPRATOR workaround with specific translations at final point of usage is the only robust way to deal with data in lists that contains semi-colon ; chars .

Yes. That's why CMake's own ExternalProject module uses LIST_SEPARATOR.

I was able to get rid of it and make all [new] semicolon test pass by fixing some quotes here and there

CMake interfaces that don't document support for semicolons likely don't support them. We've now updated the documentation to clarify this point. It's unfortunate that the language is like this, but it wasn't designed to be as general purpose as today's use cases would like it to be.

In some cases one can add quoting or backslashes near semicolons until things seem to work locally, but the resulting code may be relying on assumptions about implementation details that might change.

@bartlettroscoe
Copy link
Member Author

In some cases one can add quoting or backslashes near semicolons until things seem to work locally, but the resulting code may be relying on assumptions about implementation details that might change.

@bradking, right, and CMake needs to write tests and documentation that pin down existing behaviors so at least it is possible to pass data containing semi-colon ; chars where it currently works (by adding the correct number of escape backslashes \ for the given usage) so it will keep working.

@bartlettroscoe
Copy link
Member Author

Without further objection, I am going to merge this to 'master'. There is just no other robust alternative (short of creating a new CMake scripting language that treats lists as first-class entities and has the notation to support that).

@mzuzek
Copy link
Collaborator

mzuzek commented Mar 24, 2022

@bartlettroscoe @bradking

This PR relies on add_test() and execute_process() properly handling "\\\;".
Should that be considered an undocumented implementation detail that may change anytime ?

Please correct me if I get your discussion wrong: LIST_SEPARATOR shall/will be introduced to various CMake commands so that users won't have to play with backslashes like we do here in this PR ?

@mzuzek
Copy link
Collaborator

mzuzek commented Mar 24, 2022

@bartlettroscoe

Here's the fixed version that greens out all relevant tests without changing their content (except for LIST_SEPARATOR retraction): 0001-remove-LIST_SEPARATOR-and-fix-list-passing.patch.zip

@MikolajZuzek, can you please post that as a WIP PR so it is easier to see the changes?

Of course - see bartlettroscoe#2.

Please note that there is no difference in how we feed CMake commands with "\\\;" for test arguments and ";" for test environment on both PRs - the fixes are just about passing test arguments as strings instead of lists from TriBITS interface to bottom CMake commands (to preserve contained semicolons).

@bartlettroscoe
Copy link
Member Author

This PR relies on add_test() and execute_process() properly handling "\\\;".
Should that be considered an undocumented implementation detail that may change anytime ?

@MikolajZuzek, yes Kitware knows about that and has promised to address this. See:

CC: @bradking

@mzuzek
Copy link
Collaborator

mzuzek commented Mar 24, 2022

@bartlettroscoe Thank you for the confirmation.

PS. Year ago I was happily using CMake 3.19+ to pass around JSON structures with recursively embedded lists and dictionaries. Now I'm a bit scarred of passing plain strings because they may eventually contain semicolons, which can lead to undefined behavior on client platforms...

@bartlettroscoe
Copy link
Member Author

Now I'm a bit scarred of passing plain strings because they may eventually contain semicolons

Right. The only solution is to develop a new CMake language and parser. I am working with Kitware to start to advertise the idea for a proposal for a new CMake language parser (with both a new declarative language and a new Turing-complete imperative scripting language).

@bradking
Copy link
Contributor

@MikolajZuzek passing around plain strings containing JSON is fine so long as nothing tries to interpret it as a list or anything else besides a single string. The problem with semicolons is only when one wants to store that plain string as an element of another list. Cleanly handling lists with arbitrary content in their elements, such as nested lists, will require the new language Roscoe mentioned.

@mzuzek
Copy link
Collaborator

mzuzek commented Mar 25, 2022

@MikolajZuzek passing around plain strings containing JSON is fine so long as nothing tries to interpret it as a list or anything else besides a single string. The problem with semicolons is only when one wants to store that plain string as an element of another list. Cleanly handling lists with arbitrary content in their elements, such as nested lists, will require the new language Roscoe mentioned.

Thank you @bradking: that's an excellent summary of my key takeaway and lesson learned from this PR, bartlettroscoe#2 and related discussions on CMake issues.

bartlettroscoe added a commit to bartlettroscoe/Trilinos that referenced this pull request May 16, 2022
…/TriBITS#464)

Before, this was passing in:

   FAIL_REGULAR_EXPRESSION "FAIL;BUMMER"

which just happened to be added as a list to the test property with the old
TriBITS.  But with the new TriBITS in TriBITSPub/TriBITS#464 which more
correctly deals with semi-colons and list arguments, this was being set as a
single property value "FAIL\\;BUMMER" which did not match the output.

Passing in the list with:

   FAIL_REGULAR_EXPRESSION FAIL BUMMER

works both with old and new TriBITS and is more logical given how CMake/CTest
is supposed to be used.  (The old TriBITS documentation was incorrect.)
bartlettroscoe added a commit to bartlettroscoe/Trilinos that referenced this pull request May 16, 2022
…BITS#464)

Tt turns out that these tests really did not need to pass a list of values to
FAIL_REGULAR_EXPRESSION.  Each test was really only looking for a single value
of 'FAILED' or 'BUMMER'.  And you can argue that the tests are stronger and
better by taking out the other value that is not supposed to be printed in
that test case.
bartlettroscoe added a commit to bartlettroscoe/TriBITS that referenced this pull request May 16, 2022
…ESSION (TriBITSPub#464)

Turns out that the tribits_add_advanced_test() test-block arguments
PASS_REGULAR_EXPRESSION and FAIL_REGULAR_EXPRESSION did not support taking in
a list of regexes like the built-in CMake CTest properties of the same names.
This commit updates TAAT() to allow a list of arguments and they behave the
exact same way as the built-in properties (i.e. matching any).

I also updated the documentation to show how to pass in a list of regexes
correctly for PASS_REGULAR_EXPRESSION, FAIL_REGULAR_EXPRESSION,
FINAL_PASS_REGULAR_EXPRESSION and FINAL_FAIL_REGULAR_EXPRESSION.

I also fixed the documentation for tribits_add_test() as well for
PASS_REGULAR_EXPRESSION and FAIL_REGULAR_EXPRESSION.

Basically, a well constructed CMake function should almost never take in a
list of arguments expecting an explicit semi-colon.  This was part of the
learning in PR TriBITSPub#464 with how to treat semi-colons in CMake.

This commit was driven by a usage of FAIL_REGULAR_EXPRESSION in Trilinos while
testing updated TriBITS as part of TriBITSPub#299 when I realized the documentation was
wrong for how to deal with these lists of regexes.
bartlettroscoe added a commit that referenced this pull request May 16, 2022
…expressions

Support list for TAAT() PASS_REGULAR_EXPRESSION and FAIL_REGULAR_EXPRESSION (#464)
bartlettroscoe added a commit to bartlettroscoe/Trilinos that referenced this pull request May 17, 2022
…/TriBITS#464)

Before, this was passing in:

   FAIL_REGULAR_EXPRESSION "FAIL;BUMMER"

which just happened to be added as a list to the test property with the old
TriBITS.  But with the new TriBITS in TriBITSPub/TriBITS#464 which more
correctly deals with semi-colons and list arguments, this was being set as a
single property value "FAIL\\;BUMMER" which did not match the output.

Passing in the list with:

   FAIL_REGULAR_EXPRESSION FAIL BUMMER

works both with old and new TriBITS and is more logical given how CMake/CTest
is supposed to be used.  (The old TriBITS documentation was incorrect.)
bartlettroscoe added a commit to bartlettroscoe/Trilinos that referenced this pull request May 17, 2022
…BITS#464)

Tt turns out that these tests really did not need to pass a list of values to
FAIL_REGULAR_EXPRESSION.  Each test was really only looking for a single value
of 'FAILED' or 'BUMMER'.  And you can argue that the tests are stronger and
better by taking out the other value that is not supposed to be printed in
that test case.
jmgate pushed a commit to tcad-charon/Trilinos that referenced this pull request May 27, 2022
…s:develop' (f7c3706).

* trilinos-develop:
  FindTPLNetcdf.cmake: Lower-case function names (TriBITSPub/TriBITS#274)
  FindTPLLAPACK.cmake: Lower-case function names (TriBITSPub/TriBITS#274)
  FindTPLBLAS.cmake: Lower-case function names (TriBITSPub/TriBITS#274)
  Isorropria: Unconditionally disable tests that depend on EpetraExt (trilinos#10534)
  Zoltan2: Use single value for FAIL_REGULAR_EXPRESSION (TriBITSPub/TriBITS#464)
  Zoltan2: Properly pass in list to FAIL_REGULAR_EXPRESSION (TriBITSPub/TriBITS#464)
  Sacado: Call target_include_directories() for new TriBITS (TriBITSPub/TriBITS#299)
  ATDM: Strip leading and trailing whitespace from flags for newer CMakes
jmgate pushed a commit to tcad-charon/Trilinos that referenced this pull request May 27, 2022
…s:develop' (f7c3706).

* trilinos-develop:
  FindTPLNetcdf.cmake: Lower-case function names (TriBITSPub/TriBITS#274)
  FindTPLLAPACK.cmake: Lower-case function names (TriBITSPub/TriBITS#274)
  FindTPLBLAS.cmake: Lower-case function names (TriBITSPub/TriBITS#274)
  Isorropria: Unconditionally disable tests that depend on EpetraExt (trilinos#10534)
  Zoltan2: Use single value for FAIL_REGULAR_EXPRESSION (TriBITSPub/TriBITS#464)
  Zoltan2: Properly pass in list to FAIL_REGULAR_EXPRESSION (TriBITSPub/TriBITS#464)
  Sacado: Call target_include_directories() for new TriBITS (TriBITSPub/TriBITS#299)
  ATDM: Strip leading and trailing whitespace from flags for newer CMakes
cgcgcg pushed a commit to cgcgcg/Trilinos that referenced this pull request Jun 7, 2022
…Trilinos:develop' (7d907aa).

* trilinos/develop: (31 commits)
  MiniEM: Fix lambda capture issues in evaluators
  Phalanx: fix for non-cuda spaces
  Cmake: Tpetra deprecation removal nightlies
  Phalanx: cleanup
  Phalanx: add new ctor to v_of_v
  Phalanx: another view-of-view utility
  FindTPLNetcdf.cmake: Lower-case function names (TriBITSPub/TriBITS#274)
  FindTPLLAPACK.cmake: Lower-case function names (TriBITSPub/TriBITS#274)
  FindTPLBLAS.cmake: Lower-case function names (TriBITSPub/TriBITS#274)
  Isorropria: Unconditionally disable tests that depend on EpetraExt (trilinos#10534)
  Panzer MiniEM: Disable tests for 2nd order elements
  MueLu Maxwell1: Small fix
  MiniEM: Add "p coarsen schedule"
  Zoltan2: Use single value for FAIL_REGULAR_EXPRESSION (TriBITSPub/TriBITS#464)
  Zoltan2: Properly pass in list to FAIL_REGULAR_EXPRESSION (TriBITSPub/TriBITS#464)
  Sacado: Call target_include_directories() for new TriBITS (TriBITSPub/TriBITS#299)
  Phalanx: Array of Views acceptance test.
  ATDM: Strip leading and trailing whitespace from flags for newer CMakes
  Ifpack2 Hiptmair: Add parameter "hiptmair: implicit transpose"
  Ifpack2 Hiptmair: timers
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

4 participants