This article analyzes the performance implications of enforcing a one-to-one mapping between ROS Nodes and DDS Participants, and proposes alternative implementation approaches.
Authors: Ivan Paunovic
Date Written: 2020-06
Last Modified: 2020-07
In ROS, a Node is an entity used to group other entities. For example: Publishers, Subscriptions, Servers, Clients. Nodes ease organization and code reuse, as they can be composed in different ways.
A Participant is a type of DDS entity. Participant also group other entities, like Publishers, Subscribers, Data Writters, Data Readers, etc. Creating more Participants adds overhead to an application:
For those reasons, a Participant is a heavyweight entity.
Note: This might actually depend on the DDS vendor, some of them share these resources between Participants (e.g. OpenSplice).
Many DDS vendors, however, do not perform this kind of optimization (e.g.: RTI Connext and Fast-RTPS), and actually recommend creating just one Participant per process.
In ROS, a Context is the non-global state of an init-shutdown cycle. It also encapsulates shared state between Nodes and other entities. In most applications, there is only one ROS Context in a process.
There is a one-to-one mapping between Nodes and DDS Participants. This simplified the original implementation, as DDS Participants provide many features equivalent to the ones of ROS Nodes. The drawback of this approach is the overhead that comes with creating many Participants. Furthermore, the maximum number of Domain Participants is rather small. For example, in RTI Connext it is limited to 120 Participants per Domain.
The goal of this proposal is to improve overall performance by avoiding the creation of one Participant per Node. API changes will be avoided, if possible.
There are two alternatives, besides the one-to-one Node to Participant mapping used pre-Foxy:
The second approach is much more flexible, allowing more than one Participant in a single application for those that need it e.g. domain bridge applications. Thus, a one-to-one Participant to Context mapping was chosen.
When multiple Nodes are running in a single process, there are different options for grouping them by - ranging from a separate context for each Node, over grouping a few Nodes in the same context, to using a single context for all Nodes. For most applications, only one Context is created.
If a one-to-one Node to Participant mapping is not used, extra discovery information is needed to be able to match other entities to a Node e.g. Publishers, Subscriptions, etc. Several approaches can be used to share this information. The proposed approach uses a topic for it. Each Participant publishes a message with all the information needed to match an entity to a Node. The message structure is the following:
When one entity is updated (e.g.: a Publisher is created or destroyed), a new message is sent.
Identification of Clients and Servers happens according to the ROS conventions for their topic names (see Topic and Service name mapping to DDS).
This topic is considered an implementation detail, and not all rmw implementations have to use it.
Thus, all the necessary logic has to be in the rmw implementation itself or in an upstream package.
Implementing this logic in rcl would make it part of the API, and not an implementation detail.
To avoid code repetition, a common implementation of this logic is provided by the rmw_dds_common package.
ros_discovery_infoPreviously, each Node could have different security artifacts. That was possible because each Node was mapped to one Participant. The new approach allows to specify different security artifacts for each process. For more details, see ROS 2 Security Enclaves.
There is an ignore_local_publications option that can be set when creating a Subscription.
That option avoids receiving messages from Publishers within the same Node.
This wasn’t implemented in all the rmw implementations (e.g.: FastRTPS).
After this change, implementing this feature will be less direct. Some extra logic needs to be added in order to identify from which Node a Publisher was created.
Keyed topics could be used, with the Participant gid as the key. That would allow the Reader side of the topic to use a keep last, depth 1 history.
Instead of using a custom topic to share the extra ROS specific discovery information, a combination of Participant/Reader/Writer userData could be used:
userData: list of Nodes created by the Participant (updated when Node created destroyed).userData: Node name/namespace.userData: Node name/namespace.That information would be enough to satisfy ROS required graph API. This mechanism would also preclude having a topic that has to be read and written by all Nodes, which is better from a security perspective.
This alternative wasn’t implemented because of lack of support in some of the currently supported DDS vendors.
This optimization has been applied to rmw_cyclonedds, rmw_fastrtps_cpp and rmw_fastrtps_dynamic_cpp.
Other DDS based rmw implementations, like rmw_connext_cpp could use the same approach.