Bionic test_register_custom_callbacks Timeout

[mjcarroll]

Bug report

Required Info:

• Operating System:
  • Ubuntu 18.04 Bionic
• Installation type:
  • Source
• Version or commit hash:
  • master
• DDS implementation:
• Client library (if applicable):
  • rclcpp

Steps to reproduce issue

Run rclcpp_lifecycle test_register_custom_callbacks. After the test has passed, the program fails to terminate. In CI, this causes the test to timeout and be rerun 10 times.

Expected behavior

The test doesn't hang.

Actual behavior

The test hangs.

Additional information

This seems to only happen on bionic, regardless if this patch is applied or not: ros2/rmw_fastrtps#199

CI Failure: https://ci.ros2.org/job/ci_linux/4324/testReport/junit/(root)/projectroot/test_register_custom_callbacks/

[dhood]

I've seen the following output on a nightly once:

*** Error in `/home/rosbuild/ci_scripts/ws/build/rclcpp_lifecycle/test_register_custom_callbacks': corrupted double-linked list: 0x0000000038094840 ***

It was this ARM build from April 21st. That's just a flaky test though, not a consistent failure like you see on bionic, and tbh I only saw it that one time during a weekend where we had a lot of additional test failures/reported failures from the colcon switch, but it might be the same cause underneath.

[mikaelarguedas]

Interestingly this hang shows up only on jobs using FastRTPS by default but the same test passed on all jobs using Connext by default:
https://ci.ros2.org/job/ci_linux/4261/testReport/rclcpp_lifecycle/TestRegisterCustomCallbacks/custom_callbacks/
https://ci.ros2.org/job/ci_linux/4264/testReport/rclcpp_lifecycle/TestRegisterCustomCallbacks/custom_callbacks/

I confirmed locally that the test never hangs when using Opensplice as the default rmw_implementation

I also noticed that these tests don't shutdown their nodes on TearDown, but this doesnt fix the hanging problem, so I'll file a separate PR for that

[mjcarroll]

Asan and Tsan return no issues.

[mjcarroll]

If all of the test_node->register_on_XXX methods are commented out, the program doesn't hang. Enabling even a single registration causes the program to hang.