test_generate_policy fails on Windows

[jacobperron]

Bug report

Required Info:

• Operating System:
  • Windows
• Installation type:
  • source
• Version or commit hash:
  • master

The test was introduced in #122.

It has been consistently failing the nightly Windows debug and release builds, e.g.

• https://ci.ros2.org/view/nightly/job/nightly_win_rel/1242/testReport/junit/sros2.test.sros2.commands.security.verbs/test_generate_policy/test_generate_policy_no_nodes/
• https://ci.ros2.org/view/nightly/job/nightly_win_deb/1305/testReport/junit/sros2.test.sros2.commands.security.verbs/test_generate_policy/test_generate_policy_no_nodes/

Output:

Error Message
AssertionError: assert 0 != 0  +  where 0 = <function main at 0x000002670D3A3C80>(argv=['security', 'generate_policy', 'C:\\WINDOWS\\TEMP\\tmpwsfvjedi\\test-policy.xml'])  +    where <function main at 0x000002670D3A3C80> = cli.main
Stacktrace
test\sros2\commands\security\verbs\test_generate_policy.py:67: in test_generate_policy_no_nodes
    assert cli.main(argv=[
E   AssertionError: assert 0 != 0
E    +  where 0 = <function main at 0x000002670D3A3C80>(argv=['security', 'generate_policy', 'C:\\WINDOWS\\TEMP\\tmpwsfvjedi\\test-policy.xml'])
E    +    where <function main at 0x000002670D3A3C80> = cli.main

[kyrofa]

The relevant line for the failure is here. I'm at a loss for why such a simple test-case would be failing on Windows and not Linux, and why only in the nightly and not otherwise. Any chance it runs tests in parallel in such a way that nodes ARE running when we don't expect any to be running?

[jacobperron]

I've also seen the test failing on regular CI invocations (not just nightlies), though I'm not sure why the build passed on the original PR, maybe the test is flaky.

When I have a moment, I can try to repro locally.

[kyrofa]

Yeah, the only reason I can think for it to be flaky is if it happens to be picking up other nodes. Maybe we should be randomizing the DDS domain or something.

[jacobperron]

I looked into this and it appears that nodes are being left over from a previous test before they ultimately time-out and are removed from the ROS graph. As a consequence, the sros2 generate policy tests sees one or more nodes that shouldn't be there.

I can reproduce by running tests for demo nodes prior to running the sros2 tests:

colcon test --packages-select demo_nodes_cpp && colcon test --packages-select sros2

@wjwwood speculated that it might be due to the way Windows is handling the SIGINT (or SIGTERM) sent by the launch testing framework. It's possible that the demo nodes do not exit cleanly and so they hang around for a while before timing out.

I tried waiting for the ROS graph to be empty before each test by checking that there are no nodes (except the one we're using for testing), but for some reason it always reported the graph as empty even though there were still topics/services lingering around. Next, I can try to update my implementation to wait for no topics, services, and actions before doing the test.

As an alternative, we could make the test simpler and assert that the generated policy contains the elements we expect as a subset, but I guess we'd have to disable the test for an empty ROS graph.

[ruffsl]

As an alternative, we could make the test simpler and assert that the generated policy contains the elements we expect as a subset, but I guess we'd have to disable the test for an empty ROS graph.

This sounds a bit simpler, and would be less likely to slow down the CI.