TestNodeConstructionDestruction__rmw_fastrtps_cpp.create_with_internal_errors fails due to invalid mutex on macOS

[caguero]

Bug report

Required Info:

• Operating System:
  OS X

Steps to reproduce issue

See https://ci.ros2.org/view/nightly/job/nightly_osx_release/1870/#showFailuresLink

Expected behavior

Actual behavior

Additional information

---

Feature request

Feature description

Implementation considerations

[jacobperron]

The test has been flaky at least since October 25th: https://ci.ros2.org/view/nightly/job/nightly_osx_release/1851/testReport/(root)/test_rmw_implementation/test_create_destroy_node__rmw_fastrtps_cpp_gtest_missing_result/

Most recent (December 1st): https://ci.ros2.org/view/nightly/job/nightly_osx_release/1887/testReport/junit/(root)/test_rmw_implementation/test_create_destroy_node__rmw_fastrtps_cpp_gtest_missing_result/

[hidmic]

@jacobperron I'll take a look.

[hidmic]

This one's tough. It's coming from within Fast-DDS. And it doesn't happen consistently.

[EduPonz]

Hi @hidmic

Do you know how/where is it coming from Fast DDS? Do you have some traces or dumps that we can take a look at to see if there's something to be done in our side?

[hidmic]

Turns out I spoke too soon. It was aborting within a Fast-DDS object destructor:

Backtrace on SIGABRT

``` thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT * frame #0: 0x00007fff6931b2c2 libsystem_kernel.dylib`__pthread_kill + 10 frame #1: 0x00007fff693d6bf1 libsystem_pthread.dylib`pthread_kill + 284 frame #2: 0x00007fff692856a6 libsystem_c.dylib`abort + 127 frame #3: 0x00007fff66460641 libc++abi.dylib`abort_message + 231 frame #4: 0x00007fff664607c7 libc++abi.dylib`default_terminate_handler() + 243 frame #5: 0x00007fff67a13eeb libobjc.A.dylib`_objc_terminate() + 105 frame #6: 0x00007fff6646c19e libc++abi.dylib`std::__terminate(void (*)()) + 8 frame #7: 0x00007fff6646c213 libc++abi.dylib`std::terminate() + 51 frame #8: 0x00000001013a9caf libfastrtps.2.dylib`__clang_call_terminate + 15 frame #9: 0x0000000101537e7d libfastrtps.2.dylib`eprosima::fastrtps::SubscriberImpl::~SubscriberImpl(this=0x000000010700fe00) at SubscriberImpl.cpp:93:1 frame #10: 0x0000000101537e95 libfastrtps.2.dylib`eprosima::fastrtps::SubscriberImpl::~SubscriberImpl(this=0x000000010700fe00) at SubscriberImpl.cpp:82:1 frame #11: 0x0000000101537eb9 libfastrtps.2.dylib`eprosima::fastrtps::SubscriberImpl::~SubscriberImpl(this=0x000000010700fe00) at SubscriberImpl.cpp:82:1 frame #12: 0x000000010150345b libfastrtps.2.dylib`eprosima::fastrtps::ParticipantImpl::removeSubscriber(this=0x00000001006085b0, sub=0x0000000100601cc0) at ParticipantImpl.cpp:108:13 frame #13: 0x00000001014edcc1 libfastrtps.2.dylib`eprosima::fastrtps::Domain::removeSubscriber(sub=0x0000000100601cc0) at Domain.cpp:139:36 frame #14: 0x0000000101226485 librmw_fastrtps_shared_cpp.dylib`rmw_fastrtps_shared_cpp::destroy_subscription(identifier="rmw_fastrtps_cpp", participant_info=0x00000001006003a0, subscription=0x0000000104600d40) at subscription.cpp:52:8 frame #15: 0x000000010101776d librmw_fastrtps_cpp.dylib`init_context_impl(this=0x00007ffeefbfac58, sub=0x0000000104600d40)::$_2::operator()(rmw_subscription_t*) const at init_rmw_context_impl.cpp:125:25 frame #16: 0x0000000101017712 librmw_fastrtps_cpp.dylib`decltype(__f=0x00007ffeefbfac58, __args=0x00007ffeefbfa8e0)::$_2&>(fp)(std::__1::forward(fp0))) std::__1::__invoke(init_context_impl(rmw_context_t*)::$_2&, rmw_subscription_t*&&) at type_traits:4339:1 frame #17: 0x00000001010176a2 librmw_fastrtps_cpp.dylib`void std::__1::__invoke_void_return_wrapper::__call(init_context_impl(rmw_context_t*)::$_2&, rmw_subscription_t*&&) at __functional_base:349:9 frame #18: 0x00000001010164d9 librmw_fastrtps_cpp.dylib`std::__1::__function::__func, void (rmw_subscription_t*)>::operator(this=0x00007ffeefbfac50, __arg=0x00007ffeefbfa8e0)(rmw_subscription_t*&&) at functional:1562:12 frame #19: 0x000000010101b665 librmw_fastrtps_cpp.dylib`std::__1::function::operator(this= Lambda in File init_rmw_context_impl.cpp at Line 124, __arg=0x0000000104600d40)(rmw_subscription_t*) const at functional:1913:12 frame #20: 0x000000010101b5af librmw_fastrtps_cpp.dylib`std::__1::unique_ptr >::reset(this=0x00007ffeefbfac40, __p=0x0000000000000000) at memory:2648:7 frame #21: 0x000000010101b53d librmw_fastrtps_cpp.dylib`std::__1::unique_ptr >::~unique_ptr(this=0x00007ffeefbfac40) at memory:2602:19 frame #22: 0x0000000101011f75 librmw_fastrtps_cpp.dylib`std::__1::unique_ptr >::~unique_ptr(this=0x00007ffeefbfac40) at memory:2602:17 frame #23: 0x0000000101011887 librmw_fastrtps_cpp.dylib`init_context_impl(context=0x0000000100503d78) at init_rmw_context_impl.cpp:184:1 frame #24: 0x0000000101010fa5 librmw_fastrtps_cpp.dylib`rmw_fastrtps_cpp::increment_context_impl_ref_count(context=0x0000000100503d78) at init_rmw_context_impl.cpp:195:21 frame #25: 0x000000010103d622 librmw_fastrtps_cpp.dylib`::rmw_create_node(context=0x0000000100503d78, name="my_node", namespace_="/my_ns") at rmw_node.cpp:60:21 frame #26: 0x000000010025c97d librmw_implementation.dylib`::rmw_create_node(v3=0x0000000100503d78, v2="my_node", v1="/my_ns") at functions.cpp:222:1 frame #27: 0x0000000100005f3f test_create_destroy_node__rmw_fastrtps_cpp`TestNodeConstructionDestruction__rmw_fastrtps_cpp_create_with_internal_errors_Test::TestBody(this=0x0000000100503d00) at test_create_destroy_node.cpp:184:3 frame #28: 0x0000000100050dbe test_create_destroy_node__rmw_fastrtps_cpp`void testing::internal::HandleSehExceptionsInMethodIfSupported(object=0x0000000100503d00, method=21 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00, location="the test body")(), char const*) at gtest.cc:2433:10 frame #29: 0x000000010001adcb test_create_destroy_node__rmw_fastrtps_cpp`void testing::internal::HandleExceptionsInMethodIfSupported(object=0x0000000100503d00, method=21 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00, location="the test body")(), char const*) at gtest.cc:2469:14 frame #30: 0x000000010001ad03 test_create_destroy_node__rmw_fastrtps_cpp`testing::Test::Run(this=0x0000000100503d00) at gtest.cc:2508:5 frame #31: 0x000000010001bf42 test_create_destroy_node__rmw_fastrtps_cpp`testing::TestInfo::Run(this=0x0000000100503610) at gtest.cc:2684:11 frame #32: 0x000000010001ced7 test_create_destroy_node__rmw_fastrtps_cpp`testing::TestSuite::Run(this=0x00000001005029b0) at gtest.cc:2816:28 frame #33: 0x0000000100029cb6 test_create_destroy_node__rmw_fastrtps_cpp`testing::internal::UnitTestImpl::RunAllTests(this=0x0000000100502780) at gtest.cc:5338:44 frame #34: 0x000000010005513e test_create_destroy_node__rmw_fastrtps_cpp`bool testing::internal::HandleSehExceptionsInMethodIfSupported(object=0x0000000100502780, method=a0 98 02 00 01 00 00 00 00 00 00 00 00 00 00 00, location="auxiliary test code (environments or event listeners)")(), char const*) at gtest.cc:2433:10 frame #35: 0x000000010002962b test_create_destroy_node__rmw_fastrtps_cpp`bool testing::internal::HandleExceptionsInMethodIfSupported(object=0x0000000100502780, method=a0 98 02 00 01 00 00 00 00 00 00 00 00 00 00 00, location="auxiliary test code (environments or event listeners)")(), char const*) at gtest.cc:2469:14 frame #36: 0x0000000100029500 test_create_destroy_node__rmw_fastrtps_cpp`testing::UnitTest::Run(this=0x000000010006bfc8) at gtest.cc:4925:10 frame #37: 0x000000010000cf71 test_create_destroy_node__rmw_fastrtps_cpp`RUN_ALL_TESTS() at gtest.h:2473:46 frame #38: 0x000000010000cf50 test_create_destroy_node__rmw_fastrtps_cpp`main(argc=1, argv=0x00007ffeefbfb908) at gtest_main.cc:45:10 frame #39: 0x00007fff691e03d5 libdyld.dylib`start + 1 frame #40: 0x00007fff691e03d5 libdyld.dylib`start + 1 ```

but because an exception was being thrown from an rmw_fastrtps listener implementation:

Backtrace on throw

* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 10.1
  * frame #0: 0x00007fff6645ef28 libc++abi.dylib`__cxa_throw
    frame #1: 0x00007fff664520e7 libc++.1.dylib`std::__1::__throw_system_error(int, char const*) + 77
    frame #2: 0x00007fff6640c96d libc++.1.dylib`std::__1::mutex::lock() + 29
    frame #3: 0x0000000101a21c73 librmw_fastrtps_shared_cpp.dylib`std::__1::lock_guard<std::__1::mutex>::lock_guard(this=0x00007ffeefbf9d10, __m=0x000000010061a120) at __mutex_base:104:27
    frame #4: 0x0000000101a21b3d librmw_fastrtps_shared_cpp.dylib`std::__1::lock_guard<std::__1::mutex>::lock_guard(this=0x00007ffeefbf9d10, __m=0x000000010061a120) at __mutex_base:104:21
    frame #5: 0x0000000101a21a54 librmw_fastrtps_shared_cpp.dylib`GuardCondition::trigger(this=0x000000010061a120) at guard_condition.hpp:37:33
    frame #6: 0x0000000101a21a1c librmw_fastrtps_shared_cpp.dylib`rmw_fastrtps_shared_cpp::__rmw_trigger_guard_condition(identifier="rmw_fastrtps_cpp", guard_condition_handle=0x0000000100618620) at rmw_trigger_guard_condition.cpp:38:20
    frame #7: 0x000000010181b045 librmw_fastrtps_cpp.dylib`init_context_impl(this=0x000000010051e7e8)::$_4::operator()() const at init_rmw_context_impl.cpp:156:7
    frame #8: 0x000000010181b00d librmw_fastrtps_cpp.dylib`decltype(__f=0x000000010051e7e8)::$_4&>(fp)()) std::__1::__invoke<init_context_impl(rmw_context_t*)::$_4&>(init_context_impl(rmw_context_t*)::$_4&) at type_traits:4339:1
    frame #9: 0x000000010181afbd librmw_fastrtps_cpp.dylib`void std::__1::__invoke_void_return_wrapper<void>::__call<init_context_impl(__args=0x000000010051e7e8)::$_4&>(init_context_impl(rmw_context_t*)::$_4&) at __functional_base:349:9
    frame #10: 0x0000000101819e01 librmw_fastrtps_cpp.dylib`std::__1::__function::__func<init_context_impl(rmw_context_t*)::$_4, std::__1::allocator<init_context_impl(rmw_context_t*)::$_4>, void ()>::operator(this=0x000000010051e7e0)() at functional:1562:12
    frame #11: 0x0000000100298765 libmemory_tools.dylib`std::__1::function<void ()>::operator(this= Lambda in File init_rmw_context_impl.cpp at Line 155)() const at functional:1913:12
    frame #12: 0x0000000104ae248b librmw_dds_common.dylib`rmw_dds_common::GraphCache::remove_reader(this=0x000000010051e790, gid=0x00007ffeefbf9fa8) at graph_cache.cpp:138:3
    frame #13: 0x0000000104ae2516 librmw_dds_common.dylib`rmw_dds_common::GraphCache::remove_entity(this=0x000000010051e790, gid=0x00007ffeefbf9fa8, is_reader=true) at graph_cache.cpp:146:18
    frame #14: 0x00000001019e39b5 librmw_fastrtps_shared_cpp.dylib`void ParticipantListener::process_discovery_info<eprosima::fastrtps::rtps::ReaderProxyData const>(this=0x0000000100513790, proxyData=0x0000000140009200, is_alive=false, is_reader=true) at custom_participant_info.hpp:154:30
    frame #15: 0x00000001019dea7f librmw_fastrtps_shared_cpp.dylib`ParticipantListener::onSubscriberDiscovery(this=0x0000000100513790, (null)=0x00000001005179a0, info=0x00007ffeefbfa158) at custom_participant_info.hpp:117:7
    frame #16: 0x0000000101d07405 libfastrtps.2.dylib`eprosima::fastrtps::ParticipantImpl::MyRTPSParticipantListener::onReaderDiscovery(this=0x0000000100513b78, (null)=0x0000000100515220, info=0x00007ffeefbfa158) at ParticipantImpl.cpp:495:48
    frame #17: 0x00000001020ceb5a libfastrtps.2.dylib`eprosima::fastrtps::rtps::PDP::removeReaderProxyData(this=0x0000000101004a00, reader_guid=0x0000000105a06120) at PDP.cpp:593:31
    frame #18: 0x00000001021140f7 libfastrtps.2.dylib`eprosima::fastrtps::rtps::EDPSimple::removeLocalReader(this=0x0000000101005c00, R=0x0000000105a06110) at EDPSimple.cpp:832:20
    frame #19: 0x00000001020c15c9 libfastrtps.2.dylib`eprosima::fastrtps::rtps::BuiltinProtocols::removeLocalReader(this=0x0000000100607c40, R=0x0000000105a06110) at BuiltinProtocols.cpp:271:33
    frame #20: 0x0000000101cc3f05 libfastrtps.2.dylib`eprosima::fastrtps::rtps::RTPSParticipantImpl::deleteUserEndpoint(this=0x0000000100801000, p_endpoint=0x0000000105a06110) at RTPSParticipantImpl.cpp:1150:38
    frame #21: 0x0000000101ce1830 libfastrtps.2.dylib`eprosima::fastrtps::rtps::RTPSDomain::removeRTPSReader(reader=0x0000000105a06110) at RTPSDomain.cpp:346:44
    frame #22: 0x0000000101d37d52 libfastrtps.2.dylib`eprosima::fastrtps::SubscriberImpl::~SubscriberImpl(this=0x0000000140002c00) at SubscriberImpl.cpp:95:5
    frame #23: 0x0000000101d37e85 libfastrtps.2.dylib`eprosima::fastrtps::SubscriberImpl::~SubscriberImpl(this=0x0000000140002c00) at SubscriberImpl.cpp:84:1
    frame #24: 0x0000000101d37ea9 libfastrtps.2.dylib`eprosima::fastrtps::SubscriberImpl::~SubscriberImpl(this=0x0000000140002c00) at SubscriberImpl.cpp:84:1
    frame #25: 0x0000000101d033fb libfastrtps.2.dylib`eprosima::fastrtps::ParticipantImpl::removeSubscriber(this=0x00000001005137b0, sub=0x0000000105a00b70) at ParticipantImpl.cpp:108:13
    frame #26: 0x0000000101cedc61 libfastrtps.2.dylib`eprosima::fastrtps::Domain::removeSubscriber(sub=0x0000000105a00b70) at Domain.cpp:139:36
    frame #27: 0x0000000101a26415 librmw_fastrtps_shared_cpp.dylib`rmw_fastrtps_shared_cpp::destroy_subscription(identifier="rmw_fastrtps_cpp", participant_info=0x0000000100618da0, subscription=0x0000000100619ea0) at subscription.cpp:53:8
    frame #28: 0x000000010181776d librmw_fastrtps_cpp.dylib`init_context_impl(this=0x00007ffeefbfac58, sub=0x0000000100619ea0)::$_2::operator()(rmw_subscription_t*) const at init_rmw_context_impl.cpp:125:25
    frame #29: 0x0000000101817712 librmw_fastrtps_cpp.dylib`decltype(__f=0x00007ffeefbfac58, __args=0x00007ffeefbfa8e0)::$_2&>(fp)(std::__1::forward<rmw_subscription_t*>(fp0))) std::__1::__invoke<init_context_impl(rmw_context_t*)::$_2&, rmw_subscription_t*>(init_context_impl(rmw_context_t*)::$_2&, rmw_subscription_t*&&) at type_traits:4339:1
    frame #30: 0x00000001018176a2 librmw_fastrtps_cpp.dylib`void std::__1::__invoke_void_return_wrapper<void>::__call<init_context_impl(__args=0x00007ffeefbfac58, __args=0x00007ffeefbfa8e0)::$_2&, rmw_subscription_t*>(init_context_impl(rmw_context_t*)::$_2&, rmw_subscription_t*&&) at __functional_base:349:9
    frame #31: 0x00000001018164d9 librmw_fastrtps_cpp.dylib`std::__1::__function::__func<init_context_impl(rmw_context_t*)::$_2, std::__1::allocator<init_context_impl(rmw_context_t*)::$_2>, void (rmw_subscription_t*)>::operator(this=0x00007ffeefbfac50, __arg=0x00007ffeefbfa8e0)(rmw_subscription_t*&&) at functional:1562:12
    frame #32: 0x000000010181b665 librmw_fastrtps_cpp.dylib`std::__1::function<void (rmw_subscription_t*)>::operator(this= Lambda in File init_rmw_context_impl.cpp at Line 124, __arg=0x0000000100619ea0)(rmw_subscription_t*) const at functional:1913:12
    frame #33: 0x000000010181b5af librmw_fastrtps_cpp.dylib`std::__1::unique_ptr<rmw_subscription_t, std::__1::function<void (rmw_subscription_t*)> >::reset(this=0x00007ffeefbfac40, __p=0x0000000000000000) at memory:2648:7
    frame #34: 0x000000010181b53d librmw_fastrtps_cpp.dylib`std::__1::unique_ptr<rmw_subscription_t, std::__1::function<void (rmw_subscription_t*)> >::~unique_ptr(this=0x00007ffeefbfac40) at memory:2602:19
    frame #35: 0x0000000101811f75 librmw_fastrtps_cpp.dylib`std::__1::unique_ptr<rmw_subscription_t, std::__1::function<void (rmw_subscription_t*)> >::~unique_ptr(this=0x00007ffeefbfac40) at memory:2602:17
    frame #36: 0x0000000101811887 librmw_fastrtps_cpp.dylib`init_context_impl(context=0x0000000100502ca8) at init_rmw_context_impl.cpp:184:1
    frame #37: 0x0000000101810fa5 librmw_fastrtps_cpp.dylib`rmw_fastrtps_cpp::increment_context_impl_ref_count(context=0x0000000100502ca8) at init_rmw_context_impl.cpp:195:21
    frame #38: 0x000000010183d622 librmw_fastrtps_cpp.dylib`::rmw_create_node(context=0x0000000100502ca8, name="my_node", namespace_="/my_ns") at rmw_node.cpp:60:21
    frame #39: 0x000000010025c97d librmw_implementation.dylib`::rmw_create_node(v3=0x0000000100502ca8, v2="my_node", v1="/my_ns") at functions.cpp:222:1
    frame #40: 0x0000000100005f3f test_create_destroy_node__rmw_fastrtps_cpp`TestNodeConstructionDestruction__rmw_fastrtps_cpp_create_with_internal_errors_Test::TestBody(this=0x0000000100502c30) at test_create_destroy_node.cpp:184:3
    frame #41: 0x0000000100050dbe test_create_destroy_node__rmw_fastrtps_cpp`void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(object=0x0000000100502c30, method=21 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00, location="the test body")(), char const*) at gtest.cc:2433:10
    frame #42: 0x000000010001adcb test_create_destroy_node__rmw_fastrtps_cpp`void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(object=0x0000000100502c30, method=21 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00, location="the test body")(), char const*) at gtest.cc:2469:14
    frame #43: 0x000000010001ad03 test_create_destroy_node__rmw_fastrtps_cpp`testing::Test::Run(this=0x0000000100502c30) at gtest.cc:2508:5
    frame #44: 0x000000010001bf42 test_create_destroy_node__rmw_fastrtps_cpp`testing::TestInfo::Run(this=0x0000000100601040) at gtest.cc:2684:11
    frame #45: 0x000000010001ced7 test_create_destroy_node__rmw_fastrtps_cpp`testing::TestSuite::Run(this=0x00000001006007f0) at gtest.cc:2816:28
    frame #46: 0x0000000100029cb6 test_create_destroy_node__rmw_fastrtps_cpp`testing::internal::UnitTestImpl::RunAllTests(this=0x00000001006005c0) at gtest.cc:5338:44
    frame #47: 0x000000010005513e test_create_destroy_node__rmw_fastrtps_cpp`bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(object=0x00000001006005c0, method=a0 98 02 00 01 00 00 00 00 00 00 00 00 00 00 00, location="auxiliary test code (environments or event listeners)")(), char const*) at gtest.cc:2433:10
    frame #48: 0x000000010002962b test_create_destroy_node__rmw_fastrtps_cpp`bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(object=0x00000001006005c0, method=a0 98 02 00 01 00 00 00 00 00 00 00 00 00 00 00, location="auxiliary test code (environments or event listeners)")(), char const*) at gtest.cc:2469:14
    frame #49: 0x0000000100029500 test_create_destroy_node__rmw_fastrtps_cpp`testing::UnitTest::Run(this=0x000000010006bfc8) at gtest.cc:4925:10
    frame #50: 0x000000010000cf71 test_create_destroy_node__rmw_fastrtps_cpp`RUN_ALL_TESTS() at gtest.h:2473:46
    frame #51: 0x000000010000cf50 test_create_destroy_node__rmw_fastrtps_cpp`main(argc=1, argv=0x00007ffeefbfb908) at gtest_main.cc:45:10
    frame #52: 0x00007fff691e03d5 libdyld.dylib`start + 1
    frame #53: 0x00007fff691e03d5 libdyld.dylib`start + 1

I think #496 should fix the issue.

[hidmic]

Hmm, https://ci.ros2.org/view/nightly/job/nightly_osx_extra_rmw_release/lastCompletedBuild/testReport/(root)/projectroot/test_create_destroy_node__rmw_fastrtps_dynamic_cpp/ I still see it. New error this time.

[hidmic]

Issue above is a completely different beast. Closing this and opening another ticket.