How to use ROS 2 shared memory in snaps

If you already tried to package ROS 2 Foxy applications into snaps, you might have encountered the following error regarding shared memory:

[RTPS_MSG_OUT Error] Domain name: fastrtps -> Function compute_per_allocation_extra_size
[RTPS_TRANSPORT_SHM Error] Failed to create segment bed1660b134d4b19: Permission denied -> Function compute_per_allocation_extra_size
[RTPS_MSG_OUT Error] Permission denied -> Function init

This log is stating that FastDDS (formerly known as FastRTPS) couldn’t create a file for the shared memory mechanism due to denied permission. Fortunately, FastDDS is smart enough to fallback to a non-shared memory-based option, allowing your program to run just fine.

In this post, we will review the shared memory mechanism in ROS 2, explain why this error is happening in snaps and propose different solutions to tackle it in strictly confined snaps. We therefore assume that you are familiar with both ROS 2 and snaps. If not, have a look at either (or both) the ROS 2 documentation and the snap landing page.

What is ROS 2 shared memory?

Shared memory is a fast interprocess communication mechanism, allowing multiple processes to access the shared memory area like regular working memory. By creating a memory segment accessible to several processes, these processes can read/write to this memory with the help of a synchronization mechanism. In FastDDS, shared memory usually has a  lower latency and higher throughput than socket communication due to using a smaller-packet based mechanism and efficient copy strategies.

On Linux, shared memory segments are created under the /dev/shm path. In this directory, one usually finds the memory segment files as well as synchronization related files (locks, mutexes, etc.).  Here is an example of what ROS 2 Foxy is creating in /dev/shm.

-rw-r--r--  1 guillaume guillaume 545K mai   11 09:40 fastrtps_1ab64bba8500211c
-rw-rw-r--  1 guillaume guillaume    0 mai   11 09:40 fastrtps_1ab64bba8500211c_el
-rw-r--r--  1 guillaume guillaume  33K mai   11 09:40 fastrtps_port7424
-rw-rw-r--  1 guillaume guillaume    0 mai   11 09:40 fastrtps_port7424_el
-rw-r--r--  1 guillaume guillaume  33K mai   11 09:40 fastrtps_port7425
-rw-rw-r--  1 guillaume guillaume    0 mai   11 09:40 fastrtps_port7425_el
-rw-r--r--  1 guillaume guillaume   32 mai   11 09:40 sem.fastrtps_port7424_mutex
-rw-r--r--  1 guillaume guillaume   32 mai   11 09:40 sem.fastrtps_port7425_mutex

The files with ‘_el’ suffix are locks, files with the ‘sem.’ prefix are named mutexes, fastrtps_port prefixed files are ports and the rest are actual shared memory files. Named mutexes are from the glibc. Their creation will generate some temporary files with the format /dev/shm/XXXXXX.

Note that FastDDS is using boost::interprocess as its backend for the shared memory mechanism.

ROS 2 communication can benefit from the shared memory mechanism when nodes are on the same machine. Let us explore how this works.

How is ROS 2 Foxy using FastDDS shared memory

FastDDS implements different communication mechanisms UDP, TCP etc, but also the shared memory, enabled by default. Shared memory is used when available and from the same host. Meaning that using shared memory won’t prevent you from monitoring the data from another machine. It will fall back to the regular messaging mechanism. 

The most important files in the shared memory transport under /dev/shm are fastrtps_port* and fastrtps_* files.

Fastrtps_* contains the block of shared memory (segment) accessible by multiple processes. Every DomainParticipant configured with shared memory creates such a file on publisher/subscriber creation. The data are written to this file and possibly read by any other domain participants.

To be aware of these files and what data they contain, FastDDS is using ports (fastrtps_port* files). The first port created is used for participant’s discovery. Every participant is declaring in this port its existence and what port it is listening to. Any participant might write a data descriptor to any other participant port declaring what data is available on what shared memory segment.

You can find in the FastDDS documentation an example scenario of domain participants exchanging data through shared memory.

The remaining files *_el and sem.* are respectively lock files (to avoid segment and ports zombie files) and named mutexes for interprocess port access synchronization.

ROS 2 shared memory in a strictly confined snap

Now that we have reviewed the issue and the technical bits behind it, let us discuss solutions. By default, when strictly confined, snaps don’t have access to the /dev/shm path on the host machine but only to a predefined subdirectory specific to each snap following the pattern: /dev/shm/snap.SNAP_NAME. The FastDDS shared memory directory of ROS 2 Foxy is not configurable as of the time of writing. We cannot force it to use the shared memory subdirectory created for the snap. 

Note that everything discussed hereafter is exemplified on GitHub.

Public shared memory interface for ROS 2

Snap is providing an interface called shared-memory.  This interface allows different snaps to have access (read and/or write) to a specified path. This is meant to share resources in /dev/shm across snaps. To do so, we have to declare both a slot as well as plug. Before running the snap, we will have to manually connect these slot & plug, possibly connecting several other snaps to the same slot.

The slot is the one defining the shared memory paths to be accessed. With ROS 2 Foxy, we actually need access to everything within the /dev/shm directory (due to the semaphore temporary files). An example of the slot and plug is as follows:

slots:
  shmem-slot:
    interface: shared-memory
    write: ['*'] # paths are relative to /dev/shm
    private: false 
plugs:
  shmem-plug:
    interface: shared-memory
    shared-memory: shmem-slot
    private: false

Then both, the plug and the slot are added to the apps:

apps:
  my-ros-2-app:
    [...]
    plugs: [network, network-bind, shmem-plug]
    slots: [shmem-slot] 

Once the snap is built and installed, we can connect the shared memory with the command

sudo snap connect my_snap_name:shmem-plug my_snap_name:shmem-slot

Additional snaps will simply have to create their own plug and connect it to the very same slot.

This solution has an important overhead, since one must define slots and plugs, possibly across multiple snaps. But it is the de-facto way to enable the use of the shared memory feature in strictly confined snaps. Note that snapd 2.56.2 (or above) is necessary.

Private shared memory interface

Snap is providing another interface called private shared memory that vastly simplifies shared memory support with snap. The private shared memory is a subset of the shared memory interface. Without modifying your software, all calls to /dev/shm are going to be bound to /dev/shm/snap.SNAP_NAME automatically.

In a ROS 2 Foxy context, adding the private shared memory allows the shared memory to work within a given snap, hence benefiting from better performance.

As mentioned earlier, the shared memory is used when applications are running on the same host. When using the private shared memory, all the files we detailed earlier are automatically placed in a path specific to a given snap. However, FastDDS doesn’t know about such a corner case and therefore proceeds with establishing a local communication using the shared memory mechanism. Little does it know that this will not work. Everything is fine within the snap, but the application is mute to the outside. ROS 2 applications running on the host, but outside the snap, will not be able to see the topics from the snap; let alone to subscribe to them.

Similarly, two local users on the same host won’t have write access to each other’s shared memory files due to permissions. Hence, one user cannot subscribe to another user’s topic through shared memory. Similar causes, same result.

This being said, it is important to note that if you access these topics published from the snap from another computer, it will work seamlessly as the data are shared via UDP.

Disabling shared memory for ROS 2

Last but not least, the big hammer. FastDDS offers two options to totally disable the shared memory feature; either at compile time or at run time. We are detailing both options hereafter.

At compile-time

FastDDS offers an option to compile without the shared memory feature by simply specifying a CMake variable: -DSHM_TRANSPORT_DEFAULT=OFF. With this, no shared memory nor any associated files – ciao the error message. Of course, the main drawback of this approach is that we have to recompile FastDDS with every snap.

Disabling shared memory at run-time

FastDDS also allows for providing a configuration XML file at runtime in order to customize several aspects of the middleware. Such as, forcing the transport to use UDPv4. The XML profile is passed through an environment variable: 

FASTRTPS_DEFAULT_PROFILES_FILE: myProfileLocation/fastdds_no_shared_memory.xml

This is much easier to set up and to change in subsequent releases of a snap.

Summary

I hope that reading this article helps you understand a bit more about the ROS 2 Foxy shared memory feature and how to handle it with snaps. Although the shared memory mechanism has clear benefits, it also comes with constraints. We have seen that file permissions or even containerisation can cause the shared memory not to work as expected. Understanding the shared memory implementation from FastDDS in ROS 2 Foxy helped us to figure out solutions to use it in snaps together with their limitations. Often times, the simplest solution is to package the entire stack into a single snap using the private shared memory, or to simply disable it. 

If you have any other feedback or ideas regarding ROS 2 Foxy and shared memory, please join our forum and let us know what you think. Furthermore, have a look at the snap documentation if you want to learn more about snaps for robotics applications.