STATERA Library · Series 08 · Volume 05 · Chapter 10 · EN
Source: https://github.com/Statera-Library/08_Vol_05_Robot_Middleware_and_ROS2/blob/main/Chapter_10_ROS2_for_Physical_AI/Chapter_10_ROS2_for_Physical_AI_EN.md

Volume 05 Robot Middleware and ROS2

10. ROS2 for Physical AI

10.01 Physical AI Inference and ROS2 Integration Architecture

Physical AI extends conventional robotic software by introducing learned models that interpret multimodal observations, reason about the physical environment, and generate actions that affect the real world. Within the ROS2 structure, this intelligence should not replace the middleware or deterministic control layers. Instead, AI inference becomes a modular computational layer connected to sensing, state estimation, planning, motion generation, and control through explicit ROS2 interfaces.

A practical architecture separates the system into sensing, data preparation, AI inference, decision integration, and physical execution. Cameras, LiDAR, radar, IMU, force sensors, joint encoders, and other devices publish observations through ROS2 topics. Preprocessing nodes synchronize timestamps, transform coordinate frames, normalize measurements, generate tensors, and construct the multimodal context required by perception networks, policies, world models, or Vision-Language-Action models.

The inference layer should be treated as a collection of specialized ROS2 components rather than a single monolithic AI node. Separate inference nodes may perform object detection, semantic understanding, 3D perception, terrain analysis, policy inference, trajectory prediction, or language-conditioned action generation. This decomposition follows the broader ROS2 node architecture while allowing individual models to use different accelerators, execution frequencies, lifecycle policies, and computational resources.

ROS2 provides several communication patterns that map naturally to Physical AI workloads. Topics are appropriate for continuous streams such as images, point clouds, embeddings, detections, and predicted states. Services can expose request-response inference operations when immediate streaming is unnecessary, while actions are useful for longer AI-driven operations that require feedback, cancellation, and completion status. The interface should therefore be selected according to inference semantics rather than model implementation convenience.

Large AI payloads require particular architectural attention. Raw images, point clouds, tensor representations, and high-dimensional embeddings can create substantial serialization, memory-copy, and network overhead. Physical AI systems should minimize unnecessary conversions between ROS2 messages, CPU memory, pinned memory, and accelerator memory. Intra-process communication, composition, shared-memory mechanisms, preallocated buffers, and accelerator-aware pipelines can reduce data movement when several processing stages execute on the same edge computer.

Time consistency is equally important because multimodal inference depends on observations representing approximately the same physical state. Sensor timestamps should retain their provenance through preprocessing and inference instead of being replaced by the time at which an AI node finishes processing. ROS2 time synchronization, tf2 transformations, hardware timestamps, and synchronized sensor acquisition allow inference outputs to remain associated with the physical observations that generated them, which is essential for moving robots.

AI inference and real-time control should remain architecturally separated. Neural models can produce semantic states, target poses, navigation objectives, policy outputs, or motion references, but safety-critical actuator loops normally require predictable timing beyond what general-purpose inference pipelines can guarantee. ROS2 can connect these domains through bounded interfaces in which AI proposes an action and deterministic control software validates, constrains, interpolates, or rejects it before commands reach motors and actuators.

This separation becomes particularly important when inference latency varies because of GPU scheduling, model complexity, thermal throttling, memory pressure, or concurrent workloads. Every AI output should therefore carry enough metadata to determine whether it is still valid, including observation timestamps, inference timestamps, model identity, confidence information, and potentially sequence identifiers. Downstream nodes can then detect stale results and apply timeout, fallback, hold, stop, or conventional-control behavior instead of blindly executing delayed predictions.

Lifecycle management provides another important connection between ROS2 and Physical AI. An inference node may need to load model weights, allocate accelerator memory, initialize TensorRT or another runtime, validate device availability, and perform warm-up inference before becoming operational. ROS2 Lifecycle Nodes provide a systematic mechanism for separating configuration from activation. Failure during model initialization can therefore prevent the inference component from entering the active system state.

Model deployment should also be independent from the logical ROS2 interface wherever possible. A perception node may internally use PyTorch, ONNX Runtime, TensorRT, CUDA, or another accelerator runtime while publishing the same stable ROS2 message contract. This abstraction allows model optimization and hardware migration without forcing changes across navigation, manipulation, fleet, or control software. The architecture therefore separates the model implementation lifecycle from the robot interface lifecycle.

Physical AI frequently combines several inference rates within one robot. Camera perception may execute at tens of frames per second, a large multimodal model at a lower frequency, a world model at another update rate, and low-level control at hundreds or thousands of cycles per second. ROS2 nodes should not artificially force these workloads into one timing domain. Instead, asynchronous communication, timestamped state caches, QoS configuration, and explicit freshness policies coordinate information across heterogeneous execution rates.

QoS design becomes a system-level decision in this environment. High-rate sensor streams may favor best-effort delivery and shallow histories to prevent old data from accumulating, while critical state transitions or compact inference results may require reliable delivery. Deadline, lifespan, history, durability, and reliability settings should reflect the physical meaning of each data flow. The objective is not maximum communication reliability everywhere, but delivery behavior consistent with the temporal value of robotic information.

Physical AI integration must also include observability. Inference latency, preprocessing time, queue delay, GPU utilization, memory consumption, input frequency, output frequency, dropped samples, confidence distributions, and timeout events should be measurable alongside conventional ROS2 diagnostics. This enables engineers to distinguish whether degraded robot behavior originates in sensing, middleware communication, inference execution, decision logic, or control, rather than treating the AI model as an opaque component.

Safety supervision should remain outside the authority of a learned inference model. AI-generated motion requests can be checked against velocity limits, collision constraints, operational zones, actuator limits, robot state, and independent safety signals before execution. When inference becomes unavailable or uncertain, the robot should transition according to predefined degradation behavior. Such architecture preserves the flexibility of learned intelligence while maintaining a controlled boundary between probabilistic reasoning and physical actuation.

The resulting ROS2 architecture forms an integration fabric between conventional robotics and learned intelligence. Sensors create timestamped physical observations; preprocessing converts them into model-ready representations; inference nodes produce semantic or action-level information; decision and motion nodes interpret those outputs; and deterministic control executes validated commands. Diagnostics, lifecycle management, QoS, security, logging, and deployment mechanisms surround the complete path rather than being added only after AI functionality is implemented.

This positioning is consistent with the broader software structure in which ROS2 middleware connects control software, edge computing, perception, Physical AI foundation models, robot learning, VLA systems, data architecture, and fleet intelligence as separate but interoperable domains. Physical AI inference therefore becomes one participant in a larger software-defined robot architecture, allowing intelligence to evolve without collapsing perception, reasoning, middleware, safety, and control into an inseparable stack.

10.02 ROS2 Message Type Extension: Tensor / Embedding [w/Code]

In Physical AI systems, conventional ROS2 message definitions are often insufficient for representing neural-network data such as tensors, embeddings, feature maps, latent states, and model outputs. Extending the ROS2 interface layer for these representations allows AI components to participate in the same distributed communication architecture as sensors, planners, controllers, and diagnostics while preserving explicit data contracts between independently developed nodes.

A tensor message must describe not only numerical values but also the structure required to interpret them correctly. Important metadata includes element data type, dimensionality, shape, memory layout, and optionally stride information. A sequence of floating-point values without this context is ambiguous because the receiver cannot determine whether it represents an image feature map, action vector, point feature tensor, language embedding, or another multidimensional representation.

Tensor dimensions should therefore be represented explicitly rather than being embedded in node-specific assumptions. For example, an image tensor may follow batch, channel, height, and width dimensions, whereas another model may use batch, height, width, and channel ordering. Publishing shape and layout metadata enables subscribers to validate compatibility before inference and prevents silent interpretation errors when AI models or preprocessing pipelines are replaced.

The element type is another essential part of the interface contract. Physical AI inference pipelines may exchange FP32, FP16, BF16, INT8, signed or unsigned integer values depending on model architecture and accelerator optimization. The ROS2 message definition should identify the data representation independently of the byte payload so that receiving components can reject unsupported formats or perform deliberate conversion rather than interpreting raw memory incorrectly.

Embeddings require a related but semantically different interface design. An embedding is normally a learned vector representation encoding visual, linguistic, spatial, object-level, or multimodal information. Although it can technically be transmitted as a one-dimensional tensor, a practical ROS2 message should associate the vector with semantic metadata such as embedding dimension, model identifier, source modality, timestamp, reference frame when applicable, and the entity or observation from which it was generated.

Model identity is especially important because equal-sized embeddings from different models are not necessarily compatible. A 512-dimensional visual embedding generated by one encoder cannot automatically be compared with a 512-dimensional vector generated by another model. Message metadata should therefore provide sufficient model and representation information for downstream nodes to determine whether similarity calculation, retrieval, fusion, or policy inference is semantically valid.

Timestamp information should represent the physical observation associated with the tensor rather than merely the publication time of the ROS2 message. A feature tensor generated from a camera frame should preserve the acquisition timestamp of that frame, while derived multimodal embeddings should maintain traceability to their contributing observations. This allows synchronization, latency analysis, state reconstruction, and stale-data detection across asynchronous Physical AI pipelines.

Coordinate-frame information remains relevant when learned representations correspond to physical geometry. Point-cloud features, occupancy tensors, voxel embeddings, object features, and spatial latent representations may be associated with sensor, robot, odometry, or map frames. Integrating standard ROS2 header conventions and tf2-compatible frame identifiers helps maintain the relationship between learned representations and the physical coordinate system in which robotic decisions are executed.

A general tensor message can be designed around a compact metadata section followed by a contiguous payload. However, transmitting large numerical arrays through ordinary serialized ROS2 messages may introduce memory copies and serialization overhead. This becomes significant for high-resolution feature maps or high-frequency perception pipelines. Message-type design must therefore be considered together with intra-process communication, shared-memory transport, node composition, and accelerator-oriented data movement.

For nodes executing inside the same process, ROS2 composition can reduce unnecessary communication overhead, but application architecture should avoid assuming that every AI component will always remain colocated. A well-defined tensor interface preserves logical interoperability even when deployment changes from one process to several processes or computers. Transport optimization should consequently be separated from the semantic definition of the tensor whenever practical.

Direct GPU-to-GPU communication presents a more complex problem because a conventional ROS2 byte array normally represents host-accessible data rather than accelerator memory. High-performance implementations may introduce handles, shared buffers, loaned data, or middleware-specific zero-copy mechanisms. Such optimization should remain behind a controlled interface boundary so that the system can distinguish portable ROS2 data contracts from deployment-specific accelerator transport mechanisms.

Tensor messages also require validation at subscriber boundaries. Before consuming incoming data, a node can verify rank, shape, element type, expected model identity, payload size, timestamp freshness, and representation version. These checks are particularly valuable in Physical AI because an apparently valid numerical array may produce a plausible but incorrect neural-network result when dimensions or preprocessing conventions do not match the receiving model.

Version management becomes necessary as tensor and embedding interfaces evolve. Adding quantization parameters, model provenance, normalization metadata, semantic labels, or new element types can affect compatibility across deployed robots. Stable message definitions and explicit representation versions help prevent model upgrades from silently breaking downstream components. Interface evolution should therefore be managed independently from individual model releases whenever the underlying semantic contract remains unchanged.

Quantized inference introduces additional metadata requirements. INT8 or other compressed representations may require scale factors, zero points, quantization axes, or calibration conventions to reconstruct their intended numerical meaning. If these parameters are omitted, the raw tensor values cannot be reliably interpreted outside the originating runtime. A reusable ROS2 tensor representation should either carry the required quantization description or explicitly define it through a referenced model contract.

Embedding communication also enables higher-level Physical AI functions beyond direct perception. Visual and language embeddings can support semantic retrieval, object memory, scene association, multimodal grounding, task planning, and robot knowledge sharing. ROS2 provides the communication fabric through which these representations can move between perception, memory, reasoning, VLA, and fleet-level components without requiring every subsystem to consume raw sensor streams.

Care is required when embeddings are distributed across multiple robots because their compact size can make them attractive as shared semantic representations. Compatibility still depends on common encoder versions, preprocessing rules, normalization conventions, and semantic definitions. A fleet architecture should therefore treat embedding schema and model provenance as part of the distributed interface contract rather than assuming that vector dimensionality alone establishes interoperability.

Tensor and embedding topics also require appropriate ROS2 QoS configuration. High-frequency intermediate tensors may have little value once newer data arrives and can therefore use shallow queues and freshness-oriented delivery behavior. Compact semantic embeddings or important inference states may require different reliability characteristics. QoS should reflect the lifetime and operational significance of the learned representation instead of applying identical settings to every AI message.

Observability should accompany these extended message types. Diagnostics can track tensor dimensions, payload sizes, publication rates, serialization time, transport latency, dropped samples, incompatible schema events, and stale embeddings. Combined with inference latency measurements, these metrics make it possible to determine whether Physical AI performance degradation originates from model execution or from the communication and representation pipeline surrounding the model.

ROS2 tensor and embedding extensions ultimately establish a typed boundary between neural computation and distributed robotic software. The objective is not simply to transport arrays of numbers, but to preserve shape, type, time, provenance, semantics, compatibility, and physical context as learned representations move through the robot. This enables Physical AI models to evolve as modular components while maintaining predictable integration with perception, reasoning, planning, control, data logging, and multi-robot systems.

10.03 Camera AI Inference Node Pipeline Design [w/Code]

A camera AI inference node in ROS2 forms the bridge between continuous visual sensing and higher-level robotic intelligence. Its primary responsibility is not simply to execute a neural network, but to manage the complete path from camera acquisition through preprocessing, inference, postprocessing, and publication of structured results. Designing this path as an explicit pipeline makes visual intelligence modular, observable, replaceable, and compatible with the broader Physical AI software architecture.

The pipeline normally begins with a ROS2 camera driver that publishes image data together with camera calibration information. Depending on the hardware, the source may be an RGB camera, stereo camera, depth camera, thermal camera, or synchronized multi-camera system. The inference node should consume standard ROS2 image interfaces whenever practical, preserving timestamps and frame identifiers so that visual results remain traceable to the physical observation from which they were generated.

Image acquisition and AI inference frequently operate at different rates. A camera may produce thirty or sixty frames per second while a large neural model processes considerably fewer frames. The node should therefore avoid allowing an unlimited callback queue to accumulate outdated images. Queue depth, callback scheduling, frame dropping, and freshness policies should be explicitly designed so that the model normally processes the most operationally relevant observations instead of increasingly stale frames.

Preprocessing converts ROS2 image data into the representation expected by the model. Typical operations include color-space conversion, resizing, cropping, normalization, tensor conversion, channel reordering, and batch construction. These operations must exactly match the model\'s training and deployment assumptions. Differences in RGB/BGR ordering, normalization constants, image resolution, aspect-ratio handling, or tensor layout can significantly degrade inference even when the neural-network execution itself is correct.

Geometric preprocessing requires special attention because inference results often need to be mapped back to the original camera image. If an image is resized, cropped, padded, or letterboxed before inference, the corresponding transformation parameters should remain available during postprocessing. Bounding boxes, segmentation masks, keypoints, depth estimates, or other predictions can then be transformed accurately into the coordinate system of the original sensor image and subsequently associated with robot geometry.

The inference stage should be isolated behind a model-runtime abstraction whenever possible. The same ROS2 node architecture may execute a model using PyTorch, ONNX Runtime, TensorRT, CUDA-based libraries, or hardware-specific accelerators without changing the external ROS2 interface. Separating model execution from communication allows the deployment platform and optimization method to evolve while downstream perception, planning, and control nodes continue to consume stable message contracts.

GPU execution introduces additional pipeline considerations. Host-to-device transfer, tensor allocation, kernel execution, synchronization, and device-to-host transfer can collectively consume substantial time beyond the neural-network computation itself. Reusable buffers, pinned memory, asynchronous execution, preallocated tensors, optimized CUDA streams, and accelerator-specific runtimes can reduce overhead. Performance analysis should therefore measure end-to-end inference-node latency rather than reporting only the model\'s kernel execution time.

Postprocessing converts raw neural-network outputs into information meaningful to the robot. Detection networks may require confidence filtering and non-maximum suppression, segmentation models require mask reconstruction, and pose models may generate keypoints or geometric estimates. The resulting ROS2 messages should represent operational semantics such as object class, confidence, image coordinates, physical position, segmentation region, tracking identity, or model provenance rather than exposing undocumented model-specific arrays.

Visual inference results should retain both temporal and spatial context. The output should preserve the original observation timestamp and appropriate frame identifier, while also recording the inference completion time when latency measurement is required. This distinction allows downstream nodes to determine when the physical scene was observed and how long processing required. When geometric information is available, tf2 can relate camera-frame detections to robot, odometry, or map coordinate frames.

ROS2 composition can improve performance when the camera driver, preprocessing stage, and inference component execute on the same computer. Intra-process communication may reduce serialization and memory-copy overhead, particularly for large image messages. However, the logical interfaces between stages should remain clearly defined so that the system can later separate components across processes or computers without redesigning the entire perception architecture.

QoS configuration should reflect the temporal characteristics of camera inference. High-rate images generally benefit from limited history because old frames rapidly lose value on a moving robot. Best-effort communication may be appropriate for some sensor streams, while compact inference outputs may use reliable delivery when their operational significance justifies it. QoS settings should be selected according to sensor behavior, network conditions, processing rate, and downstream requirements rather than using one universal configuration.

Concurrency must also be controlled deliberately. Image reception, preprocessing, GPU inference, result publication, diagnostics, and parameter updates can otherwise interfere with one another through callback scheduling or shared resources. ROS2 callback groups and executors can separate workloads, while bounded queues prevent overload. For GPU pipelines, allowing uncontrolled simultaneous inference requests may increase memory consumption and latency instead of increasing useful throughput.

Lifecycle management provides a structured method for initializing camera AI nodes. During configuration, the node can load model files, initialize the inference runtime, allocate GPU memory, validate parameters, and establish subscriptions and publishers. Activation can begin actual image processing only after these resources are ready. Deactivation can stop inference without destroying configuration, while cleanup can release accelerator resources, supporting controlled startup, restart, model replacement, and fault recovery.

Runtime parameters should expose operational settings that can safely change without rebuilding the node. Examples include confidence thresholds, inference frequency, input resolution selections, enabled output classes, visualization options, and diagnostic limits. Parameters affecting the model\'s fundamental tensor structure or safety behavior require stronger validation. Dynamic configurability is useful only when parameter changes cannot silently violate the assumptions of the inference pipeline.

Observability is essential because camera AI performance depends on more than model accuracy. The node should measure camera input rate, accepted and dropped frames, preprocessing latency, inference latency, postprocessing latency, total end-to-end latency, output rate, queue occupancy, GPU utilization, and memory consumption. Timestamped measurements make it possible to identify whether degradation originates in sensor delivery, ROS2 scheduling, data movement, model execution, or downstream communication.

Fault handling should define behavior for missing images, invalid encodings, camera disconnection, GPU allocation failure, model loading errors, inference exceptions, excessive latency, and invalid outputs. A camera AI node should report these conditions through diagnostics and lifecycle state transitions instead of continuing to publish apparently normal results. Downstream components can then distinguish valid perception from degraded or unavailable AI capability and select appropriate fallback behavior.

For multi-camera robots, the architecture should avoid creating ambiguous relationships between images and inference outputs. Namespaces, camera identifiers, frame identifiers, timestamps, and model instances should clearly identify each processing path. Some systems may use independent inference nodes for each camera, while others may batch synchronized images into a multimodal model. The interface design should preserve sensor provenance regardless of the internal inference strategy.

The complete camera AI pipeline therefore connects sensing, preprocessing, accelerated inference, postprocessing, ROS2 communication, diagnostics, and lifecycle management as one controlled data path. Its architectural objective is predictable integration rather than merely maximum neural-network throughput. When time, geometry, model provenance, resource use, and failure behavior remain explicit, camera-based Physical AI can operate as a reliable modular subsystem supporting perception, navigation, manipulation, VLA reasoning, and autonomous robot behavior.

10.04 LiDAR 3D Perception AI Node Integration [w/Code]

LiDAR-based 3D perception provides a robot with direct geometric observations of the surrounding physical environment and is therefore a fundamental component of autonomous navigation and Physical AI. In a ROS2 architecture, a 3D perception AI node transforms raw point-cloud streams into structured representations such as objects, surfaces, obstacles, semantic regions, occupancy information, or learned spatial features that higher-level robot software can consume.

The processing pipeline normally begins with a LiDAR driver publishing point clouds through standard ROS2 interfaces such as PointCloud2. Each measurement contains three-dimensional coordinates and may additionally include intensity, reflectivity, ring index, return information, or sensor-specific attributes. Preserving the original timestamp and frame identifier is essential because subsequent perception results must remain associated with the physical observation and coordinate system from which they were generated.

Raw LiDAR data usually requires preprocessing before neural-network inference. Typical operations include invalid-point removal, range filtering, region-of-interest selection, ground filtering, coordinate transformation, downsampling, and feature construction. Depending on the AI architecture, the point cloud may then be represented as raw points, pillars, voxels, range images, sparse tensors, or bird\'s-eye-view features before being transferred to the inference runtime.

Preprocessing design directly influences both accuracy and computational cost. Aggressive downsampling can reduce latency and GPU memory consumption but may remove small or distant objects, while dense representations preserve geometric detail at greater computational expense. The preprocessing node should therefore expose explicit spatial ranges, voxel resolution, sampling rules, and feature definitions so that runtime behavior remains consistent with the assumptions used during model training.

Coordinate-frame management is particularly important for 3D perception because every prediction has physical spatial meaning. Sensor measurements may originate in a LiDAR frame while navigation and planning operate in base, odometry, or map frames. ROS2 tf2 transformations should be applied with the correct timestamps so that detections are transformed using the robot pose corresponding to the original scan rather than a newer pose obtained after inference has completed.

The AI inference stage can support several classes of 3D perception models. A node may perform 3D object detection, semantic segmentation, point classification, traversability estimation, terrain understanding, occupancy prediction, or learned feature extraction. More advanced Physical AI systems can combine LiDAR features with camera, radar, or language-conditioned representations, allowing geometric measurements to contribute to multimodal scene understanding and downstream reasoning.

Inference execution should remain separated from the external ROS2 message contract. The internal implementation may use PyTorch, ONNX Runtime, TensorRT, CUDA, sparse convolution libraries, or dedicated accelerator frameworks while publishing consistent outputs to downstream nodes. This separation allows models, inference engines, precision modes, and computing hardware to change without forcing navigation, mapping, planning, or fleet software to be redesigned.

Large point clouds create substantial memory and communication overhead. Repeated serialization, deserialization, and copying of PointCloud2 data can become a significant part of end-to-end latency, particularly with high-channel-count LiDAR sensors. ROS2 composition, intra-process communication, shared-memory mechanisms, reusable buffers, and carefully designed data ownership can reduce unnecessary movement when preprocessing and inference execute on the same edge computer.

GPU-oriented pipelines should minimize conversions between point-cloud representations. A poorly designed path may deserialize ROS2 data into CPU structures, copy it into another preprocessing buffer, generate a tensor, transfer that tensor to GPU memory, and then repeat similar copies during postprocessing. Efficient implementations attempt to reuse memory, batch transformations where appropriate, preallocate tensors, and keep intermediate representations close to the accelerator whenever deployment constraints permit.

Postprocessing converts model-specific outputs into robot-level spatial information. A 3D detector may produce object classes, confidence values, centers, dimensions, orientations, and velocities, while segmentation networks may assign semantic labels to points or voxels. Occupancy-oriented models can generate free, occupied, unknown, or semantic spatial states. ROS2 output interfaces should expose these operational meanings rather than requiring every subscriber to understand internal neural-network tensor formats.

Temporal consistency becomes challenging when the robot or surrounding objects move during inference. The timestamp of a published detection should therefore identify the observation time, while an additional processing timestamp can indicate when inference finished. Downstream systems can use this information to compensate for ego-motion, reject stale predictions, associate detections across scans, or determine whether an AI result remains sufficiently current for navigation and control.

Multi-sensor fusion requires even stricter synchronization. Camera images, LiDAR scans, radar measurements, and IMU data may arrive at different frequencies and with different transport delays. Fusion nodes should associate measurements according to acquisition time and calibrated transformations rather than ROS2 callback arrival order. Hardware timestamps, synchronized clocks, tf2 history, and explicit temporal tolerances help maintain physically meaningful correspondence between modalities.

ROS2 QoS settings should reflect the high-volume and time-sensitive nature of LiDAR streams. Deep queues can be harmful because old point clouds may continue to consume processing resources after they have lost operational relevance. Sensor streams may favor shallow histories and freshness-oriented behavior, whereas compact detection or occupancy outputs can use different reliability policies according to downstream requirements. QoS should therefore be designed per interface.

Concurrency should prevent high-rate point-cloud reception from blocking inference, diagnostics, or control-related communication. Callback groups and appropriate ROS2 executors can isolate responsibilities, while bounded processing queues prevent uncontrolled backlog. When inference cannot maintain the LiDAR acquisition rate, explicit scan selection or frame-skipping policies are preferable to silently accumulating latency because predictable freshness is often more valuable than processing every scan.

Lifecycle management is useful for controlling expensive 3D perception resources. During configuration, the node can load model weights, initialize sparse-convolution or TensorRT engines, allocate GPU memory, validate LiDAR parameters, and establish publishers and subscribers. Activation begins actual perception only after initialization succeeds, while deactivation and cleanup provide controlled mechanisms for stopping inference, releasing resources, recovering from failures, or replacing models.

Observability should cover the complete LiDAR AI pipeline rather than only neural-network execution. Useful measurements include point-cloud input rate, point count, filtered point count, preprocessing latency, host-to-device transfer time, inference latency, postprocessing latency, output rate, queue occupancy, GPU utilization, memory consumption, dropped scans, and end-to-end age of published results. These measurements make bottlenecks visible across sensing, middleware, preprocessing, inference, and publication.

Fault handling must account for malformed point clouds, missing scans, incorrect frame transformations, timestamp discontinuities, model initialization failures, GPU errors, excessive inference latency, and invalid outputs. Diagnostic states should clearly indicate whether perception is healthy, degraded, or unavailable. Navigation and control components can then select conservative fallback behavior rather than treating missing or stale AI perception as valid information about free space.

For multi-LiDAR robots, namespaces and frame identifiers should clearly preserve sensor provenance. Separate sensors may cover front, rear, side, or elevated fields of view, and their streams may be processed independently or fused into a common representation. Calibration, synchronization, overlap handling, and duplicate-object management become part of the integration architecture when multiple point clouds contribute to a shared 3D environmental model.

The complete ROS2 LiDAR 3D perception architecture therefore connects physical sensing, spatial preprocessing, accelerator-based AI inference, geometric postprocessing, coordinate transformation, middleware communication, diagnostics, and lifecycle management into a controlled pipeline. By preserving time, geometry, model provenance, data freshness, and failure state, the perception node can provide reliable spatial intelligence to SLAM, navigation, manipulation, world models, VLA systems, and autonomous Physical AI behaviors.

10.05 VLA Model Inference: ROS2 Service Wrapper [w/Code]

A Vision-Language-Action model connects visual perception and natural-language instructions directly to robot-oriented actions, making it an important component of Physical AI systems. In ROS2, a VLA model should normally be isolated behind a well-defined software interface rather than tightly coupled to navigation or control nodes. A ROS2 service wrapper provides this boundary by converting robot requests into model inputs and translating inference results into structured responses.

The service wrapper acts as an adaptation layer between ROS2 semantics and the internal representation required by the VLA runtime. A requesting node should not need to understand tokenizers, vision encoders, tensor layouts, GPU memory, or model-specific APIs. Instead, it sends a clearly defined request containing information such as an instruction, observation reference, robot state, task context, or execution constraints and receives an explicitly typed result.

Service-based communication is appropriate when VLA inference behaves as a bounded request-response operation. A planner may request an interpretation of the current scene, a manipulation target, or a high-level action and wait for the result before continuing. However, ROS2 services are not intended to represent every VLA interaction. Long-running tasks requiring continuous feedback, cancellation, or intermediate progress are generally better represented through ROS2 actions or higher-level task orchestration.

The service request should preserve sufficient context for physically meaningful inference. Visual observations may include RGB or depth images, references to recently acquired sensor data, object information, or embeddings generated by perception nodes. Language input can represent operator commands, mission descriptions, or task constraints. Robot-state information may include pose, joint state, gripper state, available capabilities, and environmental context needed to prevent the model from reasoning about actions the robot cannot perform.

Large visual inputs require careful interface design. Embedding complete high-resolution images directly inside every service request can create unnecessary serialization and memory-copy overhead. Depending on the deployment architecture, the wrapper may instead use synchronized topic data, cached observations, shared-memory references, or compact learned embeddings. The chosen mechanism should preserve the relationship between the request and the exact observation used for inference.

Timestamp and provenance information are essential because VLA reasoning is grounded in a changing physical environment. The wrapper should know when the visual observation was acquired, which sensor produced it, and which robot state corresponds to that observation. If inference requires significant time, the response should distinguish observation time from inference completion time so that downstream components can determine whether the result remains valid for execution.

Inside the wrapper, preprocessing converts ROS2-level inputs into the model-specific multimodal representation. Images may require resizing, normalization, tensor conversion, or vision-encoder processing, while language instructions may require tokenization and prompt construction. Robot states may be normalized or encoded into structured tokens. These transformations should remain internal to the wrapper so that changes in model implementation do not propagate throughout the ROS2 software stack.

The model-runtime interface should similarly be abstracted from the ROS2 service definition. A VLA implementation may use PyTorch, ONNX Runtime, TensorRT, accelerator-specific libraries, or a remote inference backend. The service contract can remain stable while the internal model, runtime, precision mode, or computing platform changes. This separation supports model upgrades and hardware migration without forcing every requesting ROS2 node to change.

VLA outputs require explicit semantic interpretation before they are exposed to the robot. A model may internally generate action tokens, textual descriptions, discrete skill identifiers, target poses, waypoint sequences, or continuous action vectors. The wrapper should convert these model-specific outputs into well-defined ROS2 representations that communicate what the robot is being asked to do rather than exposing undocumented neural-network output formats.

Direct conversion from VLA inference to actuator commands should generally be avoided. The service response is better interpreted as an action proposal, task decision, target state, skill selection, or motion objective that downstream planning and control components can validate. Deterministic software can then apply kinematic constraints, collision checking, workspace limits, velocity restrictions, and safety policies before physical execution, preserving a controlled boundary between learned reasoning and actuation.

Confidence and validity information should accompany VLA results when the model or application can provide meaningful measures. The response can identify whether inference succeeded, which model produced the result, what observation was used, and whether parsing or validation detected an unsupported action. A structured status field is preferable to treating every completed service call as a valid command because successful computation does not necessarily imply a physically executable result.

Timeout handling is particularly important because VLA models can have substantially higher and more variable latency than conventional ROS2 functions. The service client should define how long a result remains useful, while the wrapper should detect overloaded inference queues or stalled runtimes. If the response arrives after the relevant scene has changed, downstream software should be able to reject it rather than executing an action derived from obsolete environmental information.

Concurrency policies should be explicit when multiple ROS2 nodes request VLA inference simultaneously. Uncontrolled parallel requests can exhaust GPU memory or create unpredictable latency, especially for large multimodal models. The wrapper can serialize requests, maintain a bounded queue, support controlled batching, or route requests among multiple inference workers. The selected strategy should reflect whether the primary requirement is response latency, throughput, fairness, or resource isolation.

Lifecycle management provides a useful mechanism for controlling the VLA service. During configuration, the wrapper can load model weights, initialize tokenizers and vision encoders, allocate accelerator memory, validate model files, and perform warm-up inference. Only after these operations succeed should the service become operational. Deactivation can stop accepting inference requests, while cleanup releases model and accelerator resources for controlled restart or replacement.

Runtime model replacement can be supported without changing the service-facing ROS2 architecture. A new VLA model can be loaded and validated while clients continue to use the same logical request and response definitions, provided the semantic contract remains compatible. Model identifiers and interface versions should be recorded so that results can be traced to the exact model configuration that produced them and incompatible representations can be detected.

Observability should cover the entire service path. Useful measurements include request rate, queue waiting time, preprocessing latency, model inference latency, postprocessing time, total response latency, timeout count, failure rate, GPU utilization, memory consumption, and model identity. These metrics allow engineers to distinguish whether slow VLA behavior originates from ROS2 communication, scheduling, preprocessing, accelerator execution, or output interpretation.

Fault handling should address malformed requests, missing observations, invalid language inputs, unavailable model files, GPU allocation failures, inference exceptions, unsupported action outputs, and excessive latency. The wrapper should return explicit error states and publish diagnostic information instead of producing partially valid action data. Higher-level orchestration can then retry, choose another model, invoke a conventional planner, request operator assistance, or enter a predefined safe behavior.

Security also becomes important when language instructions can influence physical behavior. ROS2 access control should restrict which nodes are permitted to invoke the VLA service, while requests and responses may require authentication and protected communication in distributed deployments. The wrapper should treat language input as untrusted task data and ensure that model-generated outputs still pass through robot capability, operational, and safety validation before execution.

The VLA ROS2 service wrapper therefore creates a controlled boundary between multimodal foundation-model inference and conventional robotic software. It encapsulates preprocessing, model runtime, resource management, validation, timing, diagno...