Ros2 Service Send Request Does Not Send

[borgmanJeremy]

Bug report

Required Info:

• Operating System:
  • Ubuntu 16.04
  • Docker instance based on 18.04
• Installation type:
  • source
• Version or commit hash:
  ros2.repos:

repositories:
  ament/ament_cmake:
    type: git
    url: https://github.com/ament/ament_cmake.git
    version: 0.6.0
  ament/ament_index:
    type: git
    url: https://github.com/ament/ament_index.git
    version: 0.5.1
  ament/ament_lint:
    type: git
    url: https://github.com/ament/ament_lint.git
    version: 0.6.2
  ament/ament_package:
    type: git
    url: https://github.com/ament/ament_package.git
    version: 0.6.0
  ament/googletest:
    type: git
    url: https://github.com/ament/googletest.git
    version: c8b6f2be853002b9a0abfb30ea537402383bf1f4
  ament/osrf_pycommon:
    type: git
    url: https://github.com/osrf/osrf_pycommon.git
    version: 0.1.6
  ament/uncrustify_vendor:
    type: git
    url: https://github.com/ament/uncrustify_vendor.git
    version: 1.0.0
  eProsima/Fast-CDR:
    type: git
    url: https://github.com/eProsima/Fast-CDR.git
    version: v1.0.8
  eProsima/Fast-RTPS:
    type: git
    url: https://github.com/eProsima/Fast-RTPS.git
    version: b48ce9d2fba6fc94e756da01c58b72f2ad848238
  osrf/osrf_testing_tools_cpp:
    type: git
    url: https://github.com/osrf/osrf_testing_tools_cpp.git
    version: 1.1.0
  ros/class_loader:
    type: git
    url: https://github.com/ros/class_loader.git
    version: 1.2.0
  ros/pluginlib:
    type: git
    url: https://github.com/ros/pluginlib.git
    version: 2.2.1
  ros/resource_retriever:
    type: git
    url: https://github.com/ros/resource_retriever.git
    version: 2.1.0
  ros/ros_environment:
    type: git
    url: https://github.com/ros/ros_environment.git
    version: 2.2.0
  ros/urdfdom_headers:
    type: git
    url: https://github.com/ros/urdfdom_headers.git
    version: 1.0.2
  ros-perception/laser_geometry:
    type: git
    url: https://github.com/ros-perception/laser_geometry.git
    version: 2.0.0
  ros-planning/navigation_msgs:
    type: git
    url: https://github.com/ros-planning/navigation_msgs.git
    version: 2.0.0
  ros-visualization/python_qt_binding:
    type: git
    url: https://github.com/ros-visualization/python_qt_binding.git
    version: 1.0.1
  ros-visualization/qt_gui_core:
    type: git
    url: https://github.com/ros-visualization/qt_gui_core.git
    version: 1.0.3
  ros-visualization/rqt:
    type: git
    url: https://github.com/ros-visualization/rqt.git
    version: 1.0.1
  ros-visualization/rqt_console:
    type: git
    url: https://github.com/ros-visualization/rqt_console.git
    version: 1.0.1
  ros-visualization/rqt_msg:
    type: git
    url: https://github.com/ros-visualization/rqt_msg.git
    version: 1.0.1
  ros-visualization/rqt_plot:
    type: git
    url: https://github.com/ros-visualization/rqt_plot.git
    version: 1.0.2
  ros-visualization/rqt_publisher:
    type: git
    url: https://github.com/ros-visualization/rqt_publisher.git
    version: 1.0.3
  ros-visualization/rqt_py_console:
    type: git
    url: https://github.com/ros-visualization/rqt_py_console.git
    version: 1.0.0
  ros-visualization/rqt_service_caller:
    type: git
    url: https://github.com/ros-visualization/rqt_service_caller.git
    version: 1.0.1
  ros-visualization/rqt_shell:
    type: git
    url: https://github.com/ros-visualization/rqt_shell.git
    version: 1.0.0
  ros-visualization/rqt_srv:
    type: git
    url: https://github.com/ros-visualization/rqt_srv.git
    version: 1.0.1
  ros-visualization/rqt_top:
    type: git
    url: https://github.com/ros-visualization/rqt_top.git
    version: 1.0.0
  ros2/ament_cmake_ros:
    type: git
    url: https://github.com/ros2/ament_cmake_ros.git
    version: 0.5.0
  ros2/common_interfaces:
    type: git
    url: https://github.com/ros2/common_interfaces.git
    version: 0.6.1
  ros2/console_bridge_vendor:
    type: git
    url: https://github.com/ros2/console_bridge_vendor.git
    version: 1.1.0
  ros2/demos:
    type: git
    url: https://github.com/ros2/demos.git
    version: 0.6.1
  ros2/examples:
    type: git
    url: https://github.com/ros2/examples.git
    version: 0.6.1
  ros2/example_interfaces:
    type: git
    url: https://github.com/ros2/example_interfaces.git
    version: 0.6.1
  ros2/geometry2:
    type: git
    url: https://github.com/ros2/geometry2.git
    version: 0.10.1
  ros2/kdl_parser:
    type: git
    url: https://github.com/ros2/kdl_parser.git
    version: 2.2.0
  ros2/launch:
    type: git
    url: https://github.com/ros2/launch.git
    version: 0.7.3
  ros2/libyaml_vendor:
    type: git
    url: https://github.com/ros2/libyaml_vendor.git
    version: 1.0.0
  ros2/message_filters:
    type: git
    url: https://github.com/ros2/message_filters.git
    version: 3.0.0
  ros2/orocos_kinematics_dynamics:
    type: git
    url: https://github.com/ros2/orocos_kinematics_dynamics.git
    version: 3.1.0
  ros2/poco_vendor:
    type: git
    url: https://github.com/ros2/poco_vendor.git
    version: 1.1.1
  ros2/rcl_logging:
    type: git
    url: https://github.com/ros2/rcl_logging.git
    version: 0.1.0
  ros2/rcl:
    type: git
    url: https://github.com/ros2/rcl.git
    version: 0.6.3
  ros2/rcl_interfaces:
    type: git
    url: https://github.com/ros2/rcl_interfaces.git
    version: 0.6.1
#  ros2/rclc:
#    type: git
#    url: https://github.com/ros2/rclc.git
#    version: master
  ros2/rclcpp:
    type: git
    url: https://github.com/ros2/rclcpp.git
    version: 0.6.2
  ros2/rclpy:
    type: git
    url: https://github.com/ros2/rclpy.git
    version: 0.6.1
  ros2/rcutils:
    type: git
    url: https://github.com/ros2/rcutils.git
    version: 0.6.1
  ros2/realtime_support:
    type: git
    url: https://github.com/ros2/realtime_support.git
    version: 0.6.0
  ros2/rmw:
    type: git
    url: https://github.com/ros2/rmw.git
    version: 0.6.1
  ros2/rmw_connext:
    type: git
    url: https://github.com/ros2/rmw_connext.git
    version: 0.6.1
  ros2/rmw_fastrtps:
    type: git
    url: https://github.com/ros2/rmw_fastrtps.git
    version: 0.6.1
  ros2/rmw_implementation:
    type: git
    url: https://github.com/ros2/rmw_implementation.git
    version: 0.6.1
  ros2/rmw_opensplice:
    type: git
    url: https://github.com/ros2/rmw_opensplice.git
    version: 0.6.2
  ros2/robot_state_publisher:
    type: git
    url: https://github.com/ros2/robot_state_publisher.git
    version: 2.1.0
  ros2/ros1_bridge:
    type: git
    url: https://github.com/ros2/ros1_bridge.git
    version: 0.6.1
  ros2/ros2cli:
    type: git
    url: https://github.com/ros2/ros2cli.git
    version: 0.6.2
  ros2/rosidl:
    type: git
    url: https://github.com/ros2/rosidl.git
    version: 0.6.2
  ros2/rosidl_dds:
    type: git
    url: https://github.com/ros2/rosidl_dds.git
    version: 0.6.0
  ros2/rosidl_defaults:
    type: git
    url: https://github.com/ros2/rosidl_defaults.git
    version: 0.6.0
  ros2/rosidl_python:
    type: git
    url: https://github.com/ros2/rosidl_python.git
    version: 0.6.1
  ros2/rosidl_typesupport:
    type: git
    url: https://github.com/ros2/rosidl_typesupport.git
    version: 0.6.1
  ros2/rosidl_typesupport_connext:
    type: git
    url: https://github.com/ros2/rosidl_typesupport_connext.git
    version: 0.6.3
  ros2/rosidl_typesupport_fastrtps:
    type: git
    url: https://github.com/ros2/rosidl_typesupport_fastrtps.git
    version: 0.6.0
  ros2/rosidl_typesupport_opensplice:
    type: git
    url: https://github.com/ros2/rosidl_typesupport_opensplice.git
    version: 0.6.1
  ros2/rviz:
    type: git
    url: https://github.com/ros2/rviz.git
    version: 5.0.0
  ros2/sros2:
    type: git
    url: https://github.com/ros2/sros2.git
    version: 0.6.0
  ros2/system_tests:
    type: git
    url: https://github.com/ros2/system_tests.git
    version: 0.6.0
  ros2/tinyxml_vendor:
    type: git
    url: https://github.com/ros2/tinyxml_vendor.git
    version: 0.5.0
  ros2/tinyxml2_vendor:
    type: git
    url: https://github.com/ros2/tinyxml2_vendor.git
    version: 0.6.0
  ros2/tlsf:
    type: git
    url: https://github.com/ros2/tlsf.git
    version: 0.5.0
#  ros2/tutorials:
#    type: git
#    url: https://github.com/ros2/tutorials.git
#    version: master
  ros2/unique_identifier_msgs:
    type: git
    url: https://github.com/ros2/unique_identifier_msgs.git
    version: 2.0.0
  ros2/urdf:
    type: git
    url: https://github.com/ros2/urdf.git
    version: 2.2.0
  ros2/urdfdom:
    type: git
    url: https://github.com/ros2/urdfdom.git
    version: 2.1.0
  ros2/yaml_cpp_vendor:
    type: git
    url: https://github.com/ros2/yaml_cpp_vendor.git
    version: 5.0.0

• DDS implementation:
  -Fast-RTPS
• Client library (if applicable):
  -rclcpp

Steps to reproduce issue

I initially posted this on ROS answers: https://answers.ros.org/question/313378/ros2-service-send-request-does-not-send/

But after digging into it some more I have become fairly convinced this is a bug.

This example uses ros2 services. The server manages a vector of ints. Clients can "register" new values, and the server will append these values to the vector. Clients can also query a position in the vector, and if the position exists, the server will return the value in that position.

The issue I have is that when starting with my launch file (ros2 launch ros_service_mwe mwe_service.launch.py) sometimes the client that is waiting for a value at particular position gets stuck in spin_until_future_complete(). I did some debugging with WireShark and it looks like the service request never makes it to the wire. This is tricky to troubleshoot as it fails ~1 in 10 times on my particular machine, but the failure rate changes depending on the machine.

The easiest way to detect this condition is to look at the terminal output and wait for an instance where the [getNode] does not print: [INFO] [getNode]: Value of Pos: 7

I can solve this issue by adding a timeout / retry strategy to the spin_until_future_complete() call, but I really don't understand the root cause of the issue and am afraid I will mask an issue that will manifest in the future.

I started this minimal example with the add_two_ints example, so the print_usage() function still matches that example.

Here is a link to the MWE: https://github.com/borgmanJeremy/ros_service_mwe

Expected behavior

The expected behavior is that spin_until_future_complete() will return once the server is available to handle the callback.

Actual behavior

The actual behavior is spin_until_future_complete() never returns

Additional Info:

This might be related to this question? #455

[DensoADAS]

We encountered the same problem. We'd be glad about a solution.

[dschugk]

We encountered a similar problem. A solution would be great.

[dirk-thomas]

The problem comes down to a race condition between (A) when it is detected that the server is running and (B) when the topics between the server and client have actually been matched. In the Crystal release the client->wait_for_service() call returns after (A). Hence your first service call races with the event (B). If the call happens before (B) it won't arrive at the server and therefore the getter node won't be able to retrieve the value (ever).

Since the Crystal release this patch has been merged which makes the wait_for_service call only return true after (B) has happened. That should avoid the race you are seeing.

Please consider trying to build the latest state from the master branch to check if it resolves your problem.

[borgmanJeremy]

Thanks - I can confirm that this patch on master fixes my issue.