STATERA Library · Series 08 · Volume 05 · Chapter 05 · EN
Source: https://github.com/Statera-Library/08_Vol_05_Robot_Middleware_and_ROS2/blob/main/Chapter_05_ROS2_Real_Time/Chapter_05_ROS2_Real_Time_EN.md

Volume 05 Robot Middleware and ROS2

05. ROS2 Real-Time

05.01 ROS2 Real-Time Limitations and Realistic Goals

ROS 2 is frequently described as suitable for real-time robotics, but this statement requires careful interpretation. ROS 2 provides architectural mechanisms that can support deterministic execution, yet a standard ROS 2 application running on a general-purpose Linux system is not automatically a hard real-time system. The practical objective is therefore to identify which parts of a robot require bounded timing and engineer those paths separately from functions that tolerate variable latency.

Real-time behavior is fundamentally different from simply achieving high computational speed. A system may execute a control operation in hundreds of microseconds on average and still be unsuitable for a critical control loop if occasional delays extend into several milliseconds. Real-time engineering focuses on predictability, bounded worst-case latency, deadline satisfaction, and controlled jitter rather than average throughput alone. This distinction establishes the correct performance goal for ROS 2 systems.

Several software layers influence ROS 2 timing simultaneously. An application callback passes through rclcpp or rclpy, the ROS client library, the rmw abstraction, a DDS implementation, operating-system scheduling, memory management, device drivers, and potentially a network stack. Each layer can introduce scheduling delays or resource contention. Consequently, changing only the DDS QoS configuration or increasing processor performance cannot guarantee deterministic end-to-end behavior.

The ROS 2 executor is one important source of timing variability. Executors determine when subscriptions, timers, services, and other callbacks become executable and when they actually receive CPU time. If multiple callbacks share an executor, a long-running callback can delay a time-sensitive one. Multi-threading can improve concurrency but may also introduce contention, synchronization overhead, cache interference, and less predictable scheduling behavior unless callback groups and priorities are carefully designed.

Dynamic memory allocation is another obstacle to deterministic execution. Operations such as heap allocation, message construction, container resizing, serialization buffers, and object destruction may exhibit variable execution times. Page faults can produce much larger disturbances when memory pages are accessed unexpectedly. Real-time-oriented ROS 2 software therefore attempts to preallocate frequently used objects, reuse buffers, lock required memory, and eliminate unnecessary allocation from critical execution paths.

The operating system establishes another boundary on achievable determinism. Standard Linux is optimized for throughput, fairness, and broad application compatibility rather than strict latency bounds. Kernel activity, interrupts, background services, scheduler decisions, power-management transitions, and device drivers can interrupt robot workloads. PREEMPT_RT significantly improves Linux scheduling behavior, but it should be regarded as an enabling technology whose performance must still be measured under representative system load.

DDS also provides real-time-oriented communication capabilities, but DDS QoS policies should not be confused with hard timing guarantees. Reliability, history depth, durability, deadline, lifespan, and transport configuration influence communication behavior, resource usage, and failure handling. A poorly selected reliable configuration can increase blocking or retransmission effects, while excessive queue depth can preserve stale data. QoS must therefore reflect the semantic and temporal requirements of each data flow.

Network communication makes strict determinism substantially more difficult. Ethernet congestion, switch buffering, retransmissions, packet fragmentation, competing traffic, and clock differences can all affect end-to-end latency. A ROS 2 topic that appears deterministic between processes on one computer may behave differently across machines. Realistic validation must therefore measure complete paths from data production through middleware and networking to the consuming callback rather than benchmarking isolated API operations.

Robot architecture should distinguish hard real-time, firm or soft real-time, and non-real-time workloads. Motor current regulation, torque control, and some safety functions may require tightly bounded execution and are commonly placed on dedicated controllers, MCUs, FPGAs, or specialized real-time environments. Higher-level trajectory generation, navigation, perception, diagnostics, visualization, and fleet coordination generally tolerate progressively larger timing variation and are better suited to conventional ROS 2 execution.

This separation leads to a practical architectural principle: ROS 2 should often coordinate real-time subsystems rather than replace every low-level real-time mechanism. A servo controller can execute a deterministic inner loop locally while ROS 2 supplies targets, trajectories, configuration, state reporting, and supervisory commands. ros2_control and micro-ROS support variations of this layered approach, allowing strict device-level timing requirements to coexist with flexible distributed robot software.

A realistic goal for ROS 2 is therefore bounded and measurable latency for selected critical paths rather than universal hard real-time behavior across the entire software graph. Engineers should define deadlines for individual functions, acceptable jitter, maximum callback execution time, message age limits, and overload behavior. These requirements should be derived from robot dynamics and safety needs instead of adopting an arbitrary target such as "1 ms real-time" for every subsystem.

Average latency is insufficient for validating such requirements. Measurements should include worst observed latency, high-percentile latency, jitter distributions, deadline misses, scheduling delays, and behavior during CPU, memory, disk, GPU, and network stress. Long-duration tests are particularly important because rare latency spikes may remain invisible in short benchmarks. Tracing facilities can then correlate abnormal timing with executor activity, callback execution, DDS communication, or operating-system scheduling.

CPU affinity, thread priority, interrupt placement, memory locking, preallocation, executor configuration, and isolation of critical workloads can progressively improve predictability. However, these techniques should be applied as part of a measured optimization process rather than as independent tuning tricks. A configuration that improves one benchmark may degrade another because CPU isolation, networking, device interrupts, and application concurrency interact at the complete-system level.

Modern Physical AI robots make this distinction even more important. GPU inference for vision, foundation models, world models, or learned policies may require tens or hundreds of milliseconds and exhibit workload-dependent latency, while stabilization or actuator loops may operate at hundreds or thousands of hertz. Attempting to place these functions under one timing model is unrealistic. The architecture should instead connect asynchronous intelligence with deterministic control through explicitly defined interfaces.

Safety requirements must also remain independent of assumptions about ROS 2 timing performance. Emergency stopping, torque limitation, protective monitoring, and other safety-related mechanisms may require certified hardware or safety controllers whose guarantees cannot be replaced merely by assigning high-priority ROS 2 threads. ROS 2 can exchange status and supervisory information with such systems, but architectural boundaries must preserve the required independence and failure response.

The practical objective of ROS 2 real-time engineering is therefore not zero jitter or identical execution time on every cycle. It is controlled temporal behavior: critical deadlines are identified, latency sources are understood, execution paths are isolated where necessary, and deadline violations can be detected and handled safely. This makes determinism an engineering property demonstrated through architecture, configuration, measurement, and validation rather than a feature obtained simply by selecting ROS 2.

Within the broader robot software architecture, ROS 2 real-time capability should be treated as a continuum between conventional distributed middleware and dedicated deterministic control. This positioning complements separate real-time operating-system and control-software layers while allowing ROS 2 to provide standardized communication and integration across them. The most successful systems therefore assign each workload to the timing domain appropriate to its physical consequence.

A mature ROS 2 real-time design ultimately begins with realistic expectations. ROS 2 can support demanding robotic control when combined with appropriate operating systems, middleware configuration, executor design, memory discipline, hardware partitioning, and systematic benchmarking. It should not be expected to transform arbitrary Linux applications into hard real-time software. Defining this boundary clearly provides the foundation for the subsequent techniques used to optimize ROS 2 executors, memory behavior, PREEMPT_RT integration, and control-loop performance.

05.02 SingleThreadedExecutor Real-Time Tuning [w/Code]

The SingleThreadedExecutor in ROS 2 provides one of the simplest execution models for understanding and controlling callback scheduling. It processes executable entities through a single execution thread, preventing callbacks assigned to that executor from running concurrently. For real-time-oriented systems, this simplicity can be valuable because it eliminates many synchronization and thread-contention effects introduced by concurrent execution, although it does not by itself guarantee deterministic timing.

During execution, the executor monitors subscriptions, timers, services, clients, guard conditions, and other entities associated with its nodes. When work becomes available, the executor selects an executable callback and runs it before proceeding to another callback. Because only one callback executes at a time, callback execution becomes serialized. This makes timing interactions easier to analyze, but a long-running callback can directly delay every other callback waiting for the same executor.

The most important tuning principle is therefore to keep callbacks short, predictable, and non-blocking. A callback that performs lengthy computation, waits for network communication, accesses slow storage, or invokes an unpredictable external library can occupy the executor thread for an excessive period. Time-critical callbacks should perform only bounded operations and move non-critical processing to separate execution contexts when necessary, preserving the responsiveness of the primary executor.

Timer-driven control loops require particular attention because timer readiness does not mean immediate execution. A timer may become ready while another callback is already running, causing scheduling latency before the timer callback begins. The effective control-loop period therefore consists not only of the configured timer interval but also of executor waiting time, callback execution time, operating-system scheduling delay, and interference from other entities handled by the same executor.

Callback execution time should be treated as part of a timing budget. If a 1 kHz control loop has a nominal period of 1 ms, every operation sharing its execution path must fit within a substantially constrained budget while leaving margin for scheduling and middleware overhead. When callback execution approaches the available period, even small disturbances can produce deadline misses. Real-time tuning therefore requires worst-case-oriented measurements rather than optimization based solely on average execution time.

Node composition also affects SingleThreadedExecutor behavior. Placing many nodes in one executor simplifies deployment but increases the number of callbacks competing for the same execution thread. A diagnostic subscription, parameter callback, logging operation, or high-bandwidth sensor callback can unexpectedly interfere with a control timer. Critical and non-critical functionality should therefore be partitioned according to timing requirements instead of placing every ROS 2 node into one execution context for convenience.

A useful architecture is to dedicate a SingleThreadedExecutor to a small set of closely related real-time-oriented callbacks while moving perception, logging, visualization, diagnostics, and other variable-latency functions elsewhere. This does not require the entire robot software stack to use the same executor model. Different processes or execution contexts can be assigned according to workload characteristics, allowing the critical path to remain simple while higher-level software retains greater concurrency.

Memory behavior inside callbacks must also be controlled. Dynamic allocation, container resizing, message copying, string manipulation, and creation of temporary objects can introduce variable execution times. Frequently used buffers and messages should be allocated before entering the critical execution phase whenever possible. Reusing objects and maintaining fixed-capacity data structures reduces allocator activity and makes callback execution time more stable under sustained operation.

Blocking synchronization primitives should be minimized on the critical executor thread. Waiting for a mutex, condition variable, service response, file operation, or another thread can destroy the timing advantages of serialized execution. Even when a mutex normally becomes available immediately, contention under unusual load may create substantial latency. Real-time-oriented callback paths should therefore minimize shared mutable state and avoid dependencies whose completion time cannot be bounded.

ROS 2 communication settings also influence executor behavior indirectly. A subscription receiving data faster than its callback can process it may accumulate pending work according to its QoS history configuration. Reliable communication can introduce additional transport behavior, while unnecessarily deep queues may allow stale messages to remain available for processing. QoS depth, reliability, deadline, and message rates should therefore be selected together with the executor\'s processing capacity.

For sensor and control applications, processing the newest meaningful state can be more valuable than processing every historical sample. A high-rate sensor stream that overwhelms a SingleThreadedExecutor may cause increasing data age even if no individual callback is particularly slow. System designers should distinguish information that must never be lost from state data for which freshness is more important than completeness, and configure communication and processing policies accordingly.

Operating-system scheduling becomes increasingly important after application-level sources of variability have been reduced. The executor thread can be assigned an appropriate real-time scheduling policy and priority on systems configured for such operation. CPU affinity can restrict the thread to selected processor cores, while isolation strategies can reduce interference from unrelated workloads. PREEMPT_RT can further reduce kernel scheduling latency when Linux is used for demanding control applications.

CPU affinity alone does not provide isolation because interrupts, kernel threads, device activity, and other processes may still execute on the same processor. Effective tuning considers the complete CPU environment, including interrupt affinity, background services, power-management behavior, frequency scaling, and competing real-time threads. Assigning the highest possible priority to a ROS 2 executor without understanding these interactions can create starvation or system-level instability rather than improved determinism.

Memory locking is another important technique when the operating system supports it. Locking process memory can reduce the risk of page faults during critical execution, while prefaulting stack and heap regions can ensure that required pages are resident before real-time operation begins. These techniques are most effective when combined with preallocation because locking memory does not eliminate unpredictable behavior caused by dynamic allocation or application-level resource contention.

Instrumentation is essential because executor tuning should be based on observed timing behavior. Useful measurements include callback start latency, callback execution duration, timer period error, message age, deadline misses, and end-to-end latency between data production and control output. ros2_tracing and operating-system tracing tools can help distinguish whether a delay originated in callback execution, executor scheduling, middleware processing, kernel scheduling, or another subsystem.

Testing should reproduce realistic worst-case conditions rather than an idle development computer. CPU-intensive perception, GPU inference, network traffic, sensor streams, logging, storage activity, and background services can expose interference that remains invisible during simple tests. Long-duration measurements are especially valuable because rare scheduling disturbances may occur only after millions of callback cycles. Percentile distributions and maximum observed latency should accompany average values.

The SingleThreadedExecutor is particularly useful when execution order simplicity and absence of callback concurrency are more important than maximum throughput. It is less suitable when several computationally expensive callbacks must execute simultaneously or when one blocking operation cannot be removed. In those cases, separating workloads across processes or executors may provide better timing isolation than simply replacing the design with unrestricted multithreaded execution.

For robot control, a practical design often combines a dedicated ROS 2 executor with a lower-level deterministic controller. The ROS 2 executor may handle command exchange, state updates, trajectory references, and supervisory logic, while an MCU, dedicated controller, or tightly controlled real-time thread executes the fastest servo loop. This separation prevents middleware scheduling from becoming the sole timing mechanism for control functions requiring stronger guarantees.

Real-time tuning should therefore proceed from architecture to measurement rather than from isolated parameter changes. First minimize the critical callback set, then eliminate blocking operations and unnecessary allocation, establish suitable QoS and message rates, configure operating-system scheduling and CPU resources, and finally validate the complete path under stress. Each optimization should be justified by measured improvement in latency, jitter, or deadline behavior.

The realistic objective of SingleThreadedExecutor tuning is not to make every ROS 2 callback execute at an identical instant. Its strength is that a deliberately small and controlled callback set can produce an execution model that is easier to reason about, profile, and validate. Combined with disciplined callback design, memory control, PREEMPT_RT, CPU isolation, and systematic benchmarking, it can form a predictable ROS 2 execution layer for many real-time-oriented robotic workloads.

05.03 rclcpp Real-Time Node Design Patterns [w/Code]

The design of a real-time-oriented ROS 2 node begins with recognizing that rclcpp provides communication and execution abstractions, not automatic timing guarantees. A node becomes predictable only when its callbacks, memory behavior, communication paths, synchronization mechanisms, and execution environment are deliberately constrained. The objective is to create a small critical path whose execution time and interference sources can be measured and controlled.

A useful rclcpp design pattern separates initialization from real-time execution. Publishers, subscriptions, timers, parameters, buffers, and internal data structures should be created before the critical operating phase begins. Initialization may perform dynamic allocation and configuration because it occurs outside the timing-critical path. Once real-time execution starts, the node should transition toward stable resource usage and avoid operations whose duration cannot be bounded.

Callback design is the center of real-time node architecture. A critical callback should perform only the operations required to transform its input into the next control or state output. Expensive perception, logging, file access, network requests, parameter reconfiguration, and other variable-duration work should not be inserted into the same callback path. Short callbacks reduce blocking of other entities and make worst-case execution time easier to characterize.

Dynamic memory allocation should be minimized after initialization. Creating temporary vectors, resizing containers, repeatedly constructing messages, manipulating large strings, or allocating new objects inside high-frequency callbacks can introduce allocator-dependent latency. A preferable pattern is to allocate fixed or bounded buffers beforehand, reserve container capacity, reuse message objects, and maintain stable data structures throughout the real-time operating period.

Message ownership and copying also influence timing. Large sensor or state messages may generate significant overhead when repeatedly copied between callbacks and publishers. rclcpp applications should therefore consider message reuse, move semantics, loaned-message capabilities where supported, and intra-process communication when components operate in the same process. The appropriate mechanism depends on middleware and message characteristics, so its timing benefit should be validated rather than assumed.

Synchronization between callbacks requires particular discipline. A mutex protecting shared state may appear inexpensive under normal operation, but its acquisition time becomes unpredictable when another thread holds the lock. Critical data exchange should minimize shared mutable state and keep protected sections extremely short. Where appropriate, bounded queues, atomic variables, double-buffering, or carefully designed lock-free communication can reduce blocking between real-time and non-real-time execution domains.

A common pattern is to divide the node or system into real-time and non-real-time paths. The real-time side performs control calculations, state updates, and time-critical publication, while the non-real-time side handles diagnostics, logging, configuration, visualization, and computationally variable tasks. Data crossing this boundary should use explicitly designed interfaces so that slow auxiliary functions cannot block or acquire resources required by the critical path.

Timers are frequently used to implement periodic behavior in rclcpp, but a timer interval should not be interpreted as a guaranteed execution period. The timer indicates that work is ready; the executor and operating system still determine when the callback actually starts. Periodic nodes should therefore measure actual wake-up time, callback start latency, execution duration, and cycle-to-cycle jitter instead of assuming that a configured period directly represents physical control-loop timing.

Callback groups provide an architectural mechanism for controlling execution relationships. Mutually exclusive groups can prevent selected callbacks from executing simultaneously, while reentrant groups permit concurrency when the executor provides multiple threads. For real-time-oriented nodes, callback-group assignment should express intentional timing and synchronization relationships rather than simply follow software functionality. Critical callbacks should not accidentally share execution resources with unpredictable work.

The executor should be selected together with the node design. A carefully constructed rclcpp node cannot achieve predictable behavior if its callbacks are placed in an executor containing many unrelated, long-running entities. A dedicated SingleThreadedExecutor can provide a simple serialized model for a small critical callback set, while separate executors or processes can handle variable workloads. Execution topology is therefore part of node architecture rather than merely a deployment detail.

Publisher behavior should also be designed around bounded resource use. A control callback that publishes at high frequency can encounter serialization, middleware, transport, or queue-related overhead. Message size and publication rate should match the physical requirements of the subsystem. Large diagnostic information should not share the same critical communication path as compact control-state data when their timing characteristics and priorities are fundamentally different.

Subscription design must consider data freshness as well as reliability. If messages arrive faster than the callback can consume them, queued data may become stale even when no messages are technically lost. For state estimation and control, processing the newest available state may be more valuable than preserving a long history. QoS history depth, reliability, deadline, and publication frequency should therefore reflect the temporal meaning of the data.

Services and synchronous interactions require caution in real-time nodes because waiting for a response introduces a dependency on another execution context. A critical callback should generally avoid synchronous service calls, blocking futures, filesystem operations, and external process communication. Configuration and supervisory commands can often be handled outside the periodic control path, allowing asynchronous events to modify prepared state without suspending time-critical execution.

Logging is another easily overlooked source of timing variation. Formatting strings, acquiring internal locks, writing to consoles, or forwarding logs to storage can introduce unpredictable delays. Detailed logging is valuable during development, but high-frequency critical callbacks should minimize synchronous logging. Timing information can instead be collected through lightweight tracing, counters, or preallocated telemetry mechanisms and processed outside the real-time execution domain.

Parameter updates should similarly be separated from deterministic computation. Runtime parameters are useful for robot configuration, but arbitrary parameter changes can require validation, memory updates, or reconstruction of internal resources. A robust node can receive configuration changes in a non-critical callback, validate them, construct the required state, and then expose the prepared configuration to the real-time path through a bounded handoff mechanism.

Error handling must be designed without introducing uncontrolled recovery work into the critical callback. Exceptions, resource reconstruction, repeated allocation, network reconnection, and extensive diagnostic generation can produce large latency variations. The real-time path should preferably detect abnormal conditions quickly, record compact fault information, transition to a defined safe or degraded state, and delegate complex recovery operations to a supervisory non-real-time component.

Lifecycle concepts complement this pattern because initialization, activation, deactivation, cleanup, and recovery can be separated from normal periodic execution. Resources needed by critical callbacks can be prepared before activation, while cleanup and reconfiguration can occur after execution has been stopped. This separation reduces the amount of unpredictable resource management that must occur while the node is performing time-sensitive robot functions.

At the operating-system boundary, the thread executing critical rclcpp callbacks may require suitable scheduling priority, CPU affinity, memory locking, and protection from unrelated workloads. PREEMPT_RT can reduce Linux scheduling latency, but application design remains essential. A poorly structured node containing blocking operations and dynamic allocation will not become deterministic merely because it executes on a real-time-enabled kernel.

Real-time node validation should measure the complete behavior rather than only callback computation time. Important observations include callback release-to-start latency, execution duration, publication latency, message age, timer jitter, deadline misses, and end-to-end response time. Measurements should be collected under realistic CPU, network, sensor, storage, and accelerator workloads because interference frequently appears only when the integrated robot system is stressed.

For Physical AI systems, rclcpp nodes often connect deterministic control with computationally variable AI pipelines. GPU inference, world models, perception networks, and learned policies should generally be treated as asynchronous producers whose outputs carry timestamps and validity information. The control path can consume the latest acceptable result while maintaining its own deterministic cycle, rather than blocking until every AI inference operation completes.

The resulting design pattern is a layered rclcpp node architecture in which critical execution is deliberately small, bounded, and isolated while flexible ROS 2 functionality surrounds it. Preallocated memory, short callbacks, controlled synchronization, suitable QoS, intentional executor placement, and measured operating-system behavior collectively determine predictability. Real-time performance is therefore an emergent property of the complete node design rather than a single rclcpp API feature.

A mature rclcpp real-time node should remain understandable enough that engineers can identify every operation occurring on its critical path and explain the timing consequence of each dependency. When execution boundaries, resource ownership, communication semantics, and failure behavior are explicit, the node becomes easier to profile and validate. This disciplined design provides the foundation for integrating ROS 2 communication with demanding robot control without confusing middleware flexibility with hard real-time guarantees.

05.04 Memory Locking / Stack Pre-allocation in ROS2 [w/Code]

Memory behavior is one of the most important sources of latency variation in real-time-oriented ROS 2 systems. Even when callback computation is fast, page faults, heap allocation, memory reclamation, and unexpected stack growth can interrupt execution for unpredictable periods. Memory locking and pre-allocation therefore aim to ensure that the memory required by critical ROS 2 callbacks is already resident, allocated, and accessible before deterministic operation begins.

Virtual memory allows Linux processes to operate with an address space larger and more flexible than directly mapped physical memory. However, a virtual page may not be resident in RAM when a real-time thread accesses it. The operating system must then resolve a page fault and potentially perform memory-management work before execution continues. Although this behavior is acceptable for general applications, an unexpected page fault can create unacceptable latency in a high-frequency robot control loop.

Memory locking prevents selected process memory from being paged out or otherwise becoming unavailable during critical execution. On Linux, mechanisms such as mlock() and mlockall() can request that memory remain resident in physical RAM. A real-time ROS 2 process commonly locks its current and future mappings before entering its critical operating phase, reducing the probability that callback execution will be interrupted by major or minor page-fault handling.

Calling memory-locking functions alone is not sufficient because memory pages may still be allocated lazily. Operating systems often map virtual address ranges without immediately establishing all required physical pages. A real-time process should therefore touch or prefault memory regions before critical execution. By deliberately accessing stack, heap, and preallocated buffers during initialization, the application forces required pages to become resident before timing-sensitive callbacks begin.

Stack prefaulting is particularly important because thread stacks can grow as functions are called and local variables are created. If a real-time callback reaches a stack page that has never previously been touched, the operating system may need to map that page during execution. A common strategy is to reserve an adequate stack size and access representative stack memory before entering the real-time loop, ensuring that expected stack pages are committed in advance.

Stack size should be selected with sufficient safety margin but should not be treated as unlimited. Deep call chains, recursion, large local arrays, and complex library functions can increase stack consumption unexpectedly. Real-time-oriented ROS 2 code should prefer bounded call depth and predictable local storage. Large buffers are generally better allocated explicitly during initialization rather than created as large automatic variables inside high-frequency callbacks.

Heap allocation introduces a different class of timing variability. Calls to new, malloc(), container growth, shared-pointer construction, message allocation, and internal library operations may interact with memory allocators whose execution time depends on previous allocation history and synchronization. The real-time path should therefore avoid repeated allocation and deallocation whenever possible, particularly when the callback deadline is short or execution frequency is high.

Pre-allocation moves this work into the initialization phase. Vectors can reserve their maximum expected capacity, message buffers can be created beforehand, queues can use fixed or bounded storage, and reusable working memory can be assigned to algorithms before activation. Once the node enters real-time operation, callbacks operate on these existing resources rather than requesting new memory. This transforms memory consumption from an event-driven behavior into a planned resource model.

ROS 2 messages deserve special attention because apparently simple publish and subscribe operations can involve message construction, serialization, copying, middleware buffers, and transport resources. Reusing message objects can reduce application-level allocation, while intra-process communication and loaned-message mechanisms may reduce copies in appropriate configurations. However, actual behavior depends on the rclcpp, rmw, DDS, and transport implementation, so allocation behavior should be measured on the deployed stack.

Standard C++ containers are useful in ROS 2 software but require disciplined use in critical paths. std::vector may reallocate when capacity is exceeded, std::string can allocate as content grows, and associative containers can allocate nodes dynamically. Reserving capacity or replacing unbounded structures with fixed-capacity alternatives can improve predictability. The critical requirement is that runtime data cannot unexpectedly exceed the storage assumptions established during initialization.

Custom allocators can provide additional control when standard allocation behavior is insufficient. rclcpp and C++ interfaces can support allocator-aware designs in selected areas, allowing applications to use preallocated pools or deterministic allocation strategies. Nevertheless, introducing a custom allocator does not automatically create real-time behavior. Middleware, logging, synchronization, and application libraries may still allocate memory, so the complete execution path must be examined.

Memory pools are useful when objects must be acquired and released during operation but their maximum number can be bounded. Instead of requesting arbitrary heap memory, the application reserves a fixed set of objects during initialization and recycles them at runtime. Pool exhaustion must have explicitly defined behavior, because silently falling back to dynamic allocation would defeat the deterministic design. Failure should instead be detected and handled according to system requirements.

Real-time and non-real-time memory domains can also be separated architecturally. A control callback may operate only on fixed-size state structures and bounded buffers, while perception, diagnostics, configuration, and logging use conventional dynamic memory outside the critical path. Data exchanged between the domains can pass through preallocated queues, double buffers, or bounded shared structures, preventing flexible high-level software from disturbing deterministic control memory behavior.

Memory locking interacts with operating-system resource limits. Linux commonly restricts the amount of memory a process may lock, and insufficient privileges or configuration can cause locking operations to fail. A production ROS 2 application should check return values rather than assuming memory locking succeeded. System configuration, process limits, capabilities, and deployment settings must therefore be treated as part of the real-time software architecture.

Locking excessive amounts of memory can also harm the overall system. If a process pins more physical memory than necessary, other applications and kernel activities may experience memory pressure. Real-time engineering should therefore identify the working set required by critical execution and provision adequate physical RAM. Memory locking is a resource-management technique, not a reason to indiscriminately lock every available memory region on the robot computer.

The correct sequence of operations is important. A node typically creates required ROS 2 entities, allocates buffers, reserves container capacity, establishes communication resources, configures memory locking, and prefaults required pages before beginning critical periodic execution. After this transition, the real-time path should avoid operations that create new mappings or unpredictable allocations. Lifecycle-based initialization can provide a convenient architectural boundary for this preparation.

Thread creation also deserves consideration because every real-time thread has its own stack and scheduling context. Creating threads after memory preparation can introduce new stacks and resources that were not previously prefaulted. Critical worker threads should therefore be created before the deterministic phase, configured with suitable stack sizes, and exercised sufficiently to establish required memory pages before high-priority operation begins.

Monitoring page faults provides direct evidence about whether memory preparation is effective. Real-time validation can record minor and major page-fault counts before and during critical execution while simultaneously measuring callback latency and jitter. Ideally, the steady-state critical phase should show no unexpected faults associated with the real-time thread. Tracing and operating-system statistics can help correlate latency spikes with memory-management activity.

Stress testing remains necessary because allocation behavior may change under integrated workloads. High-rate sensor traffic, DDS communication, GPU pipelines, logging, and diagnostic activity can increase memory pressure even when the control callback itself performs no allocation. Tests should therefore reproduce realistic robot workloads and run long enough to expose rare events. Memory stability must be validated as a system property rather than only through inspection of one callback.

Physical AI systems make memory-domain separation especially valuable. AI inference pipelines may allocate large CPU and GPU buffers, change tensor shapes, load models, or perform asynchronous transfers, while low-level control requires stable timing. These workloads should not be assumed to share identical memory requirements. Deterministic control memory can remain fixed and resident while AI components use larger flexible memory regions through explicitly controlled interfaces.

Memory locking and pre-allocation should ultimately be viewed as complementary techniques. Locking keeps required pages resident, prefaulting ensures those pages are actually established before use, and pre-allocation prevents the critical path from repeatedly requesting new storage. None of these techniques alone guarantees real-time performance, but together they remove major sources of memory-related latency variation from a properly structured ROS 2 execution path.

A mature ROS 2 real-time memory design therefore establishes a clear transition from flexible initialization to controlled execution. Required memory is allocated, sized, locked, touched, and validated before critical callbacks begin, while runtime data structures remain bounded and reusable. Combined with disciplined rclcpp node design, executor isolation, PREEMPT_RT, and latency measurement, this approach creates a substantially more predictable memory environment for demanding robotic control workloads.

05.05 Real-Time Publisher Latency Measurement Automation [w/Code]

Real-time publisher latency measurement in ROS 2 is intended to determine how long information takes to travel from a defined publication event to a corresponding observation point and how consistently that delay remains bounded. A useful measurement system must capture more than average latency. It should expose jitter, percentile distributions, maximum observed delay, deadline violations, and rare latency spikes that can affect control behavior even when typical performance appears excellent.

The first requirement is to define the measurement boundary precisely. Publisher latency may refer to time spent inside publish(), middleware processing time, process-to-process delivery, host-to-host transport, or complete application-level latency from message generation to subscriber callback execution. These measurements describe different phenomena. Automated benchmarks should therefore record explicit measurement points so that results from different configurations remain meaningful and comparable.

A common application-level method places a timestamp and sequence identifier in each test message immediately before publication. The subscriber records another timestamp when the corresponding message reaches the selected observation point. The difference represents latency across the defined path, while the sequence number enables detection of lost, duplicated, reordered, or delayed samples. Additional timestamps can separate publisher-side processing, middleware transport, and callback scheduling effects when deeper analysis is required.

Clock selection is critical because measurement accuracy cannot exceed the quality of the time source. For communication within one computer, a monotonic clock is generally preferable because it is not affected by wall-clock corrections. For latency measurements across separate computers, clocks must be synchronized sufficiently well for the required resolution. PTP, hardware timestamping, or other synchronization mechanisms may be necessary when microsecond-level network measurements are expected.

Round-trip measurement provides an alternative when synchronized clocks are unavailable. A publisher sends a timestamped message to a responder, which returns it to the originating process, allowing round-trip time to be calculated using one local clock. This avoids direct clock-offset errors but measures two communication paths plus responder processing. Dividing round-trip time by two is only an approximation because forward and reverse delays are not necessarily symmetric.

Measurement instrumentation should introduce as little disturbance as possible. Printing every sample to the console, dynamically allocating result objects, writing each measurement immediately to disk, or performing complex statistics inside the subscriber callback can alter the latency being measured. A better pattern stores compact timestamps and sequence data in preallocated memory during the test and performs aggregation, formatting, visualization, and file export after the measurement interval.

Automating the benchmark requires reproducible control of message size, publication frequency, test duration, warm-up period, QoS configuration, executor model, and transport environment. A test controller can generate a matrix of configurations and execute each case repeatedly. Metadata should record ROS 2 distribution, rmw implementation, DDS configuration, operating-system kernel, CPU configuration, network topology, and relevant real-time settings so that results can be interpreted later.

A warm-up phase is important because startup behavior may differ substantially from steady-state operation. DDS discovery, endpoint matching, memory allocation, cache population, thread initialization, and network setup can produce temporary latency that does not represent sustained execution. Automated tests should therefore distinguish initialization latency from steady-state latency, while still preserving startup measurements when system activation time is itself an engineering requirement.

Publication rate should be varied systematically because latency often changes when the system approaches saturation. A publisher operating at 10 Hz may appear perfectly deterministic while the same path at hundreds or thousands of messages per second develops queue buildup and increasing message age. Rate-sweep automation can identify the region where throughput remains stable and the point where latency, loss, or deadline misses begin to increase rapidly.

Message size is another essential test dimension. Small control-state messages exercise scheduling and middleware overhead differently from camera frames, point clouds, tensors, or other large payloads. Automated tests should sweep representative payload sizes rather than extrapolating from a single message type. Large messages can introduce serialization cost, fragmentation, memory copying, transport buffering, and network congestion that are nea...