STATERA Library · Series 09 · Volume 06 · Chapter 08 · EN
Source: https://github.com/Statera-Library/09_Vol_06_Physical_AI_Hardware_Software_Co_Design/blob/main/Chapter_08_Reference_Architectures/Chapter_08_Reference_Architectures_EN.md
Volume 06. Physical AI Hardware Software Co Design
Chapter 08. Reference Architectures
08.01. Physical AI Reference Architecture
A Physical AI reference architecture defines how sensing, computation, intelligence, communication, and actuation are organized into a complete cyber-physical system. Unlike a conventional AI architecture that primarily transforms digital inputs into digital outputs, Physical AI must continuously connect perception and reasoning to physical actions while respecting timing, energy, safety, and hardware constraints.
The architecture begins with the physical embodiment and sensing layer, where cameras, LiDAR, radar, IMUs, encoders, force sensors, microphones, GNSS, and other modalities observe both the external environment and the robot itself. Sensor placement, resolution, update rate, synchronization, and redundancy must be designed together because they determine what information higher AI layers can reliably use.
Raw sensor streams are passed through acquisition and preprocessing functions that perform timestamping, calibration, filtering, synchronization, compression, and coordinate transformation. This layer converts heterogeneous hardware signals into consistent machine-readable representations. Deterministic data transport is particularly important because perception errors can originate from timing misalignment rather than from the AI models themselves.
The perception layer transforms synchronized sensor data into structured estimates of the surrounding world. Neural networks and geometric algorithms may perform detection, segmentation, depth estimation, tracking, localization, terrain analysis, and multimodal fusion. Instead of treating perception as an isolated model, the reference architecture connects its outputs directly to world representation, planning, safety, and control requirements.
Above perception, the world-model layer maintains a temporally coherent representation of the robot, environment, objects, agents, geometry, semantics, and uncertainty. Depending on the application, this representation may use maps, occupancy grids, BEV features, object-centric states, latent representations, or predictive dynamics. The world model provides the shared state upon which reasoning and future-oriented decision making operate.
Planning and reasoning transform the estimated world state into feasible future actions. This layer may contain behavior selection, task planning, trajectory generation, optimization, learned policies, reinforcement learning, predictive models, or vision-language-action reasoning. High-level intelligence should express goals and constraints without directly assuming that every desired command can immediately be executed by the physical hardware.
An action interface therefore separates high-level AI decisions from deterministic low-level control. Planned motion, velocity, pose, force, torque, or task commands are translated into actuator-compatible references by motion and control modules. Fast feedback loops remain close to motors and actuators, while slower perception and reasoning processes execute on more powerful processors with different timing characteristics.
A heterogeneous computing architecture supports these different workloads. MCUs and real-time processors can handle safety-critical I/O and high-frequency control, CPUs manage system logic and middleware, and GPUs or NPUs accelerate perception, world models, and learned policies. Separating control compute from AI compute prevents temporary AI overload from directly destabilizing essential physical control loops.
The reference architecture also defines hierarchical placement of computation across robot edge, on-premise infrastructure, and cloud resources. Immediate perception, collision avoidance, control, and safety functions remain locally available, while fleet learning, large-model training, global optimization, analytics, and long-term data processing can execute remotely. The robot must preserve essential autonomy when external connectivity is degraded or unavailable.
Communication and middleware connect these layers through explicitly defined data flows, interfaces, message semantics, timing requirements, and quality-of-service policies. High-bandwidth sensor streams, compact world states, control commands, diagnostics, and model updates have very different transport requirements. Architectural interfaces should therefore expose not only data types but also latency, frequency, reliability, and synchronization expectations.
Safety and runtime assurance operate across the entire architecture rather than appearing as a final software component. Independent monitoring can detect sensor failures, unreasonable AI outputs, actuator saturation, timing violations, thermal limits, communication loss, or compute overload. When nominal intelligence becomes unreliable, the system should transition toward degraded operation, fallback control, controlled stopping, or another predefined safe state.
Power and thermal management form another cross-cutting architectural function because available intelligence depends directly on physical energy and cooling capacity. Sensors, processors, communication devices, and actuators compete for a finite power budget. Runtime management may adjust model complexity, inference frequency, sensor activation, processor operating states, or workload placement to maintain required autonomy without violating thermal or battery constraints.
Observability is essential for operating and improving the architecture. Sensor quality, inference latency, control timing, processor utilization, memory bandwidth, power consumption, temperatures, network status, actuator feedback, and safety events should be measurable through a unified diagnostics framework. Recorded operational data can then support debugging, validation, fleet analytics, model improvement, and hardware-software redesign.
Modularity allows the same architectural principles to support AMRs, mobile manipulators, quadrupeds, humanoids, aerial robots, and heterogeneous fleets. Their sensors, actuators, compute capacities, dynamics, and control frequencies may differ substantially, but standardized boundaries between sensing, perception, world modeling, reasoning, planning, control, safety, and infrastructure enable components to evolve without redesigning the entire system.
The resulting Physical AI architecture is therefore not a simple linear pipeline from sensor input to actuator output. It is a hierarchy of interacting real-time loops operating at different spatial, temporal, computational, and safety scales. Effective hardware-software co-design aligns these loops with compute capacity, communication bandwidth, power, thermal limits, embodiment, and operational requirements so that AI capability remains physically executable and dependable.
08.02. Perception World Model Planning Control Stack
A Perception--World Model--Planning--Control stack organizes Physical AI as a continuous transformation from raw observations into physically executed behavior. Each stage solves a different abstraction problem: perception extracts meaningful information, the world model maintains state and predicts evolution, planning selects feasible future behavior, and control converts planned actions into stable physical motion. Together, they form a closed-loop intelligence architecture.
Perception begins with synchronized observations from cameras, LiDAR, radar, IMUs, encoders, GNSS, force sensors, and other platform-dependent sensors. Rather than forwarding raw measurements directly to decision-making modules, perception converts them into structured information such as objects, free space, terrain properties, depth, motion, robot pose, contact states, and semantic features. Sensor fusion combines complementary modalities to improve robustness and environmental coverage.
The perception layer must operate under strict latency and bandwidth constraints because its outputs become the evidence used by every downstream stage. High-resolution sensing may improve accuracy but increases memory traffic, computation, and inference latency. Physical AI therefore requires perception models to be selected together with sensor resolution, update frequency, accelerator capability, and communication architecture rather than optimized independently as isolated neural networks.
Perception outputs are integrated into a world model that represents the current physical situation across time. The world model may contain metric maps, occupancy representations, bird\'s-eye-view features, object-centric states, semantic information, robot state, dynamic agents, uncertainty, or learned latent variables. Its purpose is not merely to store perception results, but to construct a coherent state that remains useful when observations are incomplete, noisy, or temporarily unavailable.
Temporal modeling distinguishes the world model from instantaneous perception. By combining previous states with current observations, the system can estimate velocity, motion trends, object persistence, hidden states, and possible future transitions. Predictive world models can further simulate how the environment may evolve under different actions, enabling Physical AI to reason about consequences before committing actuators to potentially irreversible behavior.
Uncertainty should propagate through this stack instead of being discarded at the perception boundary. Ambiguous detections, uncertain localization, partially observed obstacles, or unpredictable agents can materially change what constitutes a safe plan. A world model can preserve confidence estimates, probability distributions, alternative hypotheses, or risk measures so that planning responds differently to uncertain situations than to equally shaped but confidently observed environments.
Planning transforms the maintained world state into candidate behaviors and trajectories. Depending on the embodiment, this may involve route planning, behavior selection, motion planning, manipulation sequencing, footstep generation, trajectory optimization, or task-level reasoning. Planning must consider geometric feasibility, dynamics, actuator limitations, collision constraints, energy, mission objectives, safety margins, and uncertainty rather than simply selecting an abstract action with the highest predicted reward.
A useful architecture separates planning into multiple time horizons. High-level planning may reason over tasks, destinations, goals, or long-horizon behavior, while local planning reacts to nearby geometry and dynamic agents. Motion planning then converts behavioral intent into executable trajectories. These layers may operate at different frequencies, allowing expensive reasoning to run slowly while short-horizon planning continuously adapts to newly perceived physical conditions.
Learned policies, reinforcement learning, optimization, classical planning, and model-predictive methods can coexist within the planning layer. A learned model may propose actions while a trajectory optimizer enforces physical constraints, or a world model may predict candidate futures that are evaluated by a safety-aware planner. The reference stack therefore does not require every component to be neural; it defines functional boundaries through which learned and deterministic methods can cooperate.
Control receives desired trajectories, velocities, poses, forces, torques, or other references and converts them into commands compatible with the physical actuators. Feedback from encoders, IMUs, force sensors, motor controllers, and other proprioceptive sources allows the controller to continuously correct deviations between desired and actual behavior. This closes the physical loop between computational intention and the dynamics of the real machine.
Control generally operates much faster than perception and high-level planning. Motor current or torque loops may execute at very high frequencies, while position, velocity, locomotion, or vehicle controllers operate at intermediate rates and AI reasoning runs more slowly. This multi-rate structure is fundamental to Physical AI because mechanical stability cannot depend on the completion time of a large perception network, world model, or reasoning model.
The boundary between planning and control is also a safety boundary. High-level AI should not be allowed to issue unrestricted hardware commands that bypass actuator limits, stability requirements, or collision constraints. Command validation, saturation handling, dynamic envelopes, safety interlocks, and fallback controllers can constrain AI-generated actions before they reach motors, steering systems, manipulators, legs, propellers, or other physical effectors.
Feedback does not flow only from control to actuators. Executed motion changes the physical world, producing new observations that return through sensing and perception. Differences between predicted and observed outcomes can update the world model, trigger replanning, reveal actuator degradation, or provide learning signals. The architecture is therefore cyclic rather than sequential: Sense, Perceive, Model, Predict, Plan, Act, Observe, and Correct continuously repeat.
Compute partitioning should follow the timing characteristics of these loops. Safety-critical control and hardware interfaces are commonly placed on deterministic embedded processors, while perception, multimodal fusion, world modeling, and learned planning use GPUs, NPUs, or other AI accelerators. Higher-level reasoning may additionally use on-premise or cloud resources, but essential local planning and control must remain operational when network connectivity becomes unavailable.
The complete stack should consequently be treated as one co-designed system rather than four independently optimized AI modules. Improving perception accuracy is valuable only if its latency still supports planning; improving prediction is useful only if planners can exploit it; sophisticated plans matter only when controllers and actuators can execute them. Physical AI emerges when perception, world modeling, planning, and control operate as coordinated loops constrained by the same physical reality.
08.03. Low Level Control and High Level AI Separation
Low-level control and high-level AI serve fundamentally different roles in a Physical AI system and should therefore be separated by explicit architectural boundaries. High-level AI interprets the environment, predicts future conditions, selects behaviors, and generates motion intentions, while low-level control ensures that these intentions are executed with deterministic timing, physical stability, and direct awareness of actuator dynamics.
High-level AI typically includes perception, world modeling, task reasoning, behavior planning, learned policies, trajectory generation, and increasingly foundation-model or VLA-based capabilities. These workloads may require GPUs or NPUs and can exhibit variable execution times because model complexity, input size, memory traffic, and reasoning depth change dynamically. Such variability is acceptable for many deliberative functions but unsuitable for safety-critical actuator loops.
Low-level control operates much closer to the physical hardware. Motor current, torque, velocity, position, steering, force, balance, and stabilization loops interact directly with drives, joints, wheels, legs, manipulators, and other actuators. These functions usually require predictable execution periods, bounded latency, low jitter, and reliable access to proprioceptive feedback, making deterministic processors, MCUs, or real-time CPUs natural execution platforms.
The separation can be understood as a difference between deciding what the machine should do and ensuring that the machine can physically do it. High-level AI may request a target pose, velocity, trajectory, grasp, footstep sequence, or navigation maneuver. Low-level controllers convert these abstract commands into actuator-level references while continuously compensating for disturbances, friction, payload variation, terrain interaction, and modeling errors.
A carefully defined command interface forms the boundary between these domains. Instead of allowing AI models to directly manipulate motor voltages or unrestricted torque commands, the architecture exposes constrained action representations such as bounded velocities, trajectories, poses, forces, or motion primitives. The interface can validate command ranges, timestamps, reference frames, rates, and feasibility before accepting a request for physical execution.
This boundary also provides an important safety barrier. AI-generated actions can be checked against velocity limits, acceleration limits, joint limits, stability envelopes, collision constraints, actuator saturation, workspace boundaries, and other platform-specific restrictions. Commands that violate these conditions can be clipped, rejected, modified, or replaced by a predefined safe response without requiring the high-level AI itself to guarantee every hardware-level safety property.
Timing separation is equally important. A large perception or reasoning model may operate at tens of hertz or even more slowly, whereas motion controllers can require hundreds of hertz and inner actuator loops may execute at kilohertz rates. Low-level stability must therefore continue independently between consecutive AI updates. The robot should never become dynamically unstable simply because an inference cycle takes longer than expected.
A multi-rate architecture naturally emerges from this principle. Slow task reasoning defines goals, intermediate planning produces trajectories, faster motion control tracks those trajectories, and still faster actuator controllers regulate physical variables. Each layer receives commands from the slower layer above while using faster local feedback below, creating nested control loops that preserve responsiveness without forcing every intelligence function to execute at the highest control frequency.
Compute separation reinforces timing separation. High-level perception, multimodal fusion, world models, and learned planning can execute on GPU- or NPU-based AI computers, while safety-critical control remains on dedicated embedded processors or real-time computing domains. Communication between them should use explicitly specified interfaces with bounded update rates, timestamps, watchdogs, validity indicators, and timeout behavior rather than assuming continuous availability of the AI computer.
This arrangement also contains failures. A GPU overload, memory exhaustion, crashed AI process, delayed inference, corrupted model output, or temporary restart should not automatically propagate into uncontrolled actuator behavior. The control domain can detect stale or missing commands and transition toward trajectory completion, reduced-speed operation, holding behavior, controlled stopping, or another defined fallback state according to the platform and operational context.
Conversely, low-level control must communicate physical limitations back toward high-level intelligence. Tracking error, actuator saturation, wheel slip, joint limits, motor temperature, available torque, contact state, battery limitations, or degraded hardware can invalidate assumptions used during planning. Exposing these conditions allows the world model and planner to update their understanding of what the robot can currently achieve rather than repeatedly issuing infeasible commands.
The separation should not be interpreted as complete isolation. High-level AI benefits from actuator feedback and control-state information, while low-level controllers require goals generated by perception and planning. The architecture therefore combines strong functional separation with carefully controlled information exchange. This creates a hierarchy in which each layer operates at the abstraction, computational complexity, and timing scale appropriate to its responsibilities.
Different embodiments require different implementations of the same principle. An AMR may separate navigation and trajectory planning from wheel velocity control, a manipulator may separate task and motion planning from joint servo loops, and a quadruped may separate locomotion reasoning from high-frequency balance and torque control. Aerial robots similarly separate mission and trajectory intelligence from attitude, rate, and motor stabilization loops.
The boundary is especially important when increasingly powerful learned models generate physical actions. Reinforcement-learning policies, world-model planners, VLA systems, and generalist robot policies may produce sophisticated behavior, but their outputs still encounter mechanical limits and uncertain real-world dynamics. A deterministic execution layer provides a controlled path from probabilistic intelligence to physical action without requiring learned models to replace every established control mechanism.
Verification and development also become easier when responsibilities are separated. Control engineers can validate timing, stability, actuator limits, and fallback behavior independently from continual AI model updates, while AI developers can improve perception, prediction, and planning without rewriting motor-control software. Stable interfaces allow processors, models, sensors, and actuators to evolve at different rates while preserving system-level integration.
Ultimately, separating high-level AI from low-level control is not intended to reduce the authority or capability of intelligence, but to place intelligence within an executable physical hierarchy. High-level AI determines goals and adapts behavior using increasingly rich models of the world, while low-level control preserves timing, stability, constraint enforcement, and safe execution. Their coordinated interaction enables capable AI behavior without sacrificing deterministic control of the machine.
08.04. Embedded Control plus Edge GPU Architecture
An Embedded Control plus Edge GPU architecture separates deterministic machine control from computationally intensive Physical AI while keeping both domains physically close to the robot. Embedded controllers handle hard real-time functions, hardware interfaces, and safety-critical execution, while an edge GPU executes perception, sensor fusion, world modeling, and learned intelligence. This division combines predictable control with high-performance AI computation.
The embedded control domain typically consists of MCUs, real-time CPUs, motor controllers, programmable I/O devices, and dedicated safety processors. These components interface directly with encoders, IMUs, motor drives, steering systems, joints, brakes, force sensors, and emergency circuits. Their primary responsibility is not sophisticated reasoning but reliable execution with bounded latency, low jitter, and deterministic responses to physical events.
The edge GPU domain provides the parallel computation required by modern neural networks and multimodal processing. Camera images, LiDAR point clouds, radar measurements, and other high-dimensional sensor streams can be processed through detection, segmentation, depth estimation, tracking, localization, occupancy estimation, and sensor-fusion networks. GPU acceleration allows these workloads to operate locally without continuously transmitting high-bandwidth sensor data to external infrastructure.
A heterogeneous robot therefore contains at least two distinct timing domains. Embedded controllers may execute motor and stabilization loops from hundreds of hertz to kilohertz rates, whereas GPU perception and reasoning workloads generally operate at lower and more variable frequencies. The architecture must preserve this distinction so that fluctuations in neural-network inference time cannot interrupt motor regulation, vehicle stability, manipulation control, or other time-critical physical functions.
Communication between the embedded controller and edge GPU becomes a critical architectural interface. The controller can provide robot pose, velocity, joint states, actuator feedback, battery status, fault information, and synchronized sensor metadata to the AI domain. In the opposite direction, the GPU can provide trajectories, target velocities, poses, waypoints, motion primitives, or behavior commands rather than unrestricted electrical commands to individual actuators.
Shared time is particularly important when multiple sensors and processors participate in the same perception-to-action loop. Hardware timestamps, synchronized clocks, deterministic communication, and consistent coordinate frames allow GPU-generated environmental estimates to correspond to the physical state observed by the embedded controller. Without accurate temporal alignment, even highly accurate AI models can generate commands based on outdated robot motion or incorrectly fused sensor observations.
The command boundary should include validation rather than functioning as a transparent communication channel. Commands generated by GPU-based planning or learned policies can be checked for validity, freshness, velocity and acceleration limits, actuator constraints, collision conditions, and feasible operating envelopes before execution. The embedded domain can therefore retain final authority over safety-critical actuation even when sophisticated behavior originates from probabilistic AI models.
Failure containment is another major advantage of this architecture. GPU workloads may experience overload, memory pressure, thermal throttling, delayed inference, software faults, or model failures. An embedded controller can use watchdogs and command timeouts to detect these conditions and continue a previously validated trajectory, reduce speed, hold position, execute a controlled stop, or enter another predefined safe state instead of allowing an AI-side failure to destabilize the machine.
The edge GPU should similarly monitor the embedded domain rather than treating it as an ideal actuator interface. Tracking errors, wheel slip, joint saturation, excessive motor temperature, reduced battery capability, actuator faults, or unexpected contact forces can indicate that a planned action is no longer feasible. Feeding these states into perception, world modeling, and planning allows high-level intelligence to adapt its behavior to the robot\'s actual physical capability.
Data movement must be carefully designed because modern sensors can generate more information than processors and networks can efficiently transfer. High-bandwidth cameras and LiDAR are commonly connected directly or through high-speed interfaces to the edge computing domain, while compact control and state messages are exchanged with embedded processors. Avoiding unnecessary copies and transformations reduces memory bandwidth consumption, latency, processor load, and overall system power.
Power and thermal constraints further couple the two computing domains. A high-performance GPU can consume substantial electrical power and generate significant heat, while actuators may simultaneously create large transient loads on the robot\'s battery and power distribution system. Compute performance, cooling capacity, battery runtime, peak actuator demand, and thermal throttling must therefore be considered together rather than sizing the AI computer independently from the physical platform.
The embedded controller can remain responsible for essential autonomy when GPU capability is intentionally reduced. Under low battery, thermal stress, compute overload, or degraded sensing, the system may lower inference frequency, reduce model complexity, disable nonessential AI functions, or transition to simpler navigation behavior. Basic stabilization, braking, actuator protection, and safe stopping should remain available independently of these performance-management decisions.
Software partitioning should reflect the hardware boundary. Real-time control software, hardware abstraction, safety monitoring, and deterministic communication belong primarily to the embedded domain, whereas deep-learning runtimes, perception pipelines, world models, and computationally demanding planning belong to the GPU domain. Middleware connects them through versioned interfaces so that AI models and control firmware can evolve without creating unnecessary dependencies between their internal implementations.
This architecture scales across multiple Physical AI embodiments. An AMR can combine an embedded vehicle controller with GPU-based localization and navigation, while a mobile manipulator can add joint controllers and manipulation perception. Quadrupeds can retain high-frequency balance and actuator control on deterministic processors while using GPU computation for terrain understanding and locomotion planning. Similar separation applies to humanoids and autonomous aerial robots.
Redundancy can also be introduced according to system criticality. The embedded domain may independently monitor emergency inputs and actuator health, while the edge GPU performs richer environmental risk assessment. Some architectures can maintain simplified perception or obstacle-detection paths outside the primary AI pipeline, ensuring that failure of a large neural model does not remove every mechanism capable of detecting dangerous operating conditions.
The Embedded Control plus Edge GPU architecture ultimately creates a practical bridge between conventional real-time robotics and modern Physical AI. The embedded domain guarantees deterministic interaction with physical hardware, while the edge GPU supplies computationally intensive perception, prediction, and intelligence. Their carefully controlled interface allows powerful AI to influence robot behavior without making fundamental stability and safety dependent on variable-latency AI computation.
08.05. Edge plus On Premise Architecture
An Edge plus On-Premise architecture extends Physical AI beyond a single robot by combining autonomous on-robot intelligence with shared computing infrastructure located within the operational site. The edge system remains responsible for immediate perception, planning, control, and safety, while on-premise resources provide heavier reasoning, fleet intelligence, data aggregation, model services, and computational workloads that do not require hard real-time execution.
The edge layer is physically integrated with each robot and normally contains embedded controllers, CPUs, GPUs, NPUs, local storage, and communication interfaces. It processes high-bandwidth sensor streams from cameras, LiDAR, radar, IMUs, encoders, and other sensors without depending on external connectivity. Functions directly associated with safe motion must remain executable locally so that network degradation cannot remove the robot\'s essential autonomy.
Real-time perception is a primary edge responsibility. Sensor synchronization, detection, segmentation, localization, tracking, occupancy estimation, local mapping, and obstacle understanding must operate close to the physical platform because their outputs influence immediate behavior. Transmitting every raw sensor stream to an external server before making basic navigation decisions would introduce communication latency, bandwidth dependence, and an unacceptable external failure point.
Local world modeling and planning should similarly preserve enough environmental understanding for independent operation. The robot can maintain its immediate map, dynamic object states, traversability information, local predictions, and short-horizon trajectories on the edge. Even when richer global models exist on-premise, the robot should continue sensing, predicting, avoiding obstacles, and executing safe actions when communication becomes temporarily unavailable.
On-premise infrastructure introduces a second computational tier with substantially greater shared compute, memory, and storage capacity. GPU servers or AI clusters can perform large-scale map construction, fleet-level optimization, centralized analytics, complex world-model processing, model evaluation, and other workloads that would be inefficient to replicate on every robot. Multiple robots can therefore share expensive computational capabilities while retaining independent real-time control.
Fleet intelligence is particularly well suited to the on-premise tier. Individual robots can upload compact state information, local maps, detected hazards, traversability observations, task progress, and selected sensor data. The infrastructure can integrate these observations into shared maps or fleet memory and redistribute useful information, allowing one robot\'s experience to improve the operational awareness of other robots without requiring every platform to rediscover the same conditions.
Task allocation and global planning can also exploit this shared perspective. An on-premise fleet manager may consider robot location, battery state, payload capability, current workload, traffic conditions, mission priority, and infrastructure constraints when distributing tasks. The resulting assignments or strategic plans are sent to individual robots, while local planners retain authority to modify execution when newly observed obstacles or safety conditions conflict with the global plan.
The communication boundary between edge and on-premise systems must distinguish critical from noncritical information. Robot control should not depend on continuous transmission of high-rate actuator commands across the network. Instead, communication can exchange goals, missions, maps, model outputs, fleet states, compressed features, selected observations, software updates, and diagnostics, while high-frequency feedback loops and emergency responses remain entirely within the robot.
Network latency, jitter, packet loss, and temporary disconnection must therefore be treated as normal architectural conditions rather than exceptional failures. Messages should contain timestamps, validity periods, sequence information, and clearly defined timeout behavior. When communication quality decreases, robots can continue local operation, cache data, defer nonessential uploads, and synchronize accumulated information after connectivity recovers instead of immediately losing autonomy.
Bandwidth management becomes increasingly important as fleet size grows. Continuous upload of raw multi-camera video and dense LiDAR from every robot can overwhelm wireless networks and on-premise storage. Edge systems can filter, compress, summarize, or selectively record information based on operational importance. Events, anomalies, uncertain observations, failures, or representative samples can receive higher upload priority than redundant observations from predictable operating conditions.
On-premise infrastructure can additionally support models that are too large or computationally expensive for continuous edge execution. Complex reasoning, large world models, foundation-model inference, global optimization, or retrospective analysis may be executed remotely when latency requirements permit. Their results can provide strategic guidance to robots, but the architecture should prevent delayed remote intelligence from overriding newer local observations or immediate safety decisions.
The same infrastructure provides a natural foundation for the Physical AI data loop. Robots collect operational experience and transfer selected logs, sensor samples, execution outcomes, failure cases, and performance metrics to centralized storage. These datasets can support model training, fine-tuning, evaluation, simulation, and validation, after which approved model versions can be distributed back to the fleet for controlled deployment and continuous improvement.
Model lifecycle management should distinguish experimentation from operational deployment. New perception, planning, or world-model components can be evaluated on on-premise resources before being released to edge computers. Version control, compatibility checks, staged deployment, rollback mechanisms, and fleet monitoring reduce the risk that a defective model update simultaneously affects every robot and provide traceability between software changes and physical behavior.
Security and data sovereignty are important reasons for choosing on-premise infrastructure. Sensitive sensor data, facility maps, production information, human observations, or operational records can remain inside the organization\'s controlled network rather than being continuously transferred to external cloud services. Authentication, encryption, network segmentation, access control, secure update mechanisms, and audit logging should protect communication between robots and infrastructure.
Resource scheduling allows the on-premise tier to serve many robots efficiently. GPU capacity can be allocated dynamically among fleet analytics, large-model inference, simulation, map processing, training, and evaluation according to operational priority. During periods of high robot activity, infrastructure can prioritize mission-critical inference and fleet services, while computationally intensive training or offline analytics can be delayed or assigned to available resources.
This architecture creates a deliberate hierarchy rather than making the robot a thin client of a centralized server. The edge provides low-latency autonomy, safety, and resilience, while on-premise infrastructure contributes computational depth, shared knowledge, and fleet-scale optimization. The correct partition is determined by latency, bandwidth, compute cost, power, privacy, availability, and the consequence of losing communication for each function.
An Edge plus On-Premise architecture therefore enables Physical AI systems to scale from intelligent individual robots toward coordinated intelligent fleets. Local computing preserves immediate perception and physical autonomy, while shared infrastructure aggregates experience and supplies heavier intelligence across the site. Their combination creates a resilient architecture in which robots can operate independently when necessary and become collectively more capable when infrastructure connectivity is available.
08.06. Edge On Premise Cloud Architecture
A three-tier Edge--On-Premise--Cloud architecture organizes Physical AI computation according to latency, autonomy, bandwidth, privacy, and computational scale. The edge remains closest to physical action, on-premise infrastructure coordinates robots and provides site-level intelligence, while the cloud supplies elastic resources for large-scale training, analytics, model management, and cross-site learning. Together, these tiers form a hierarchical intelligence platform.
The edge tier is embedded directly within each robot and contains real-time controllers, CPUs, GPUs or NPUs, local storage, and sensor interfaces. It processes cameras, LiDAR, radar, IMUs, encoders, force sensors, and other observations close to their source. Immediate perception, localization, obstacle avoidance, short-horizon planning, stabilization, control, and safety remain local because physical autonomy cannot depend on a continuously available external network.
Local world modeling allows the robot to maintain an operational representation of its surroundings even when infrastructure connectivity is degraded. Occupancy, nearby objects, traversability, robot state, dynamic obstacles, and short-term predictions can be continuously updated at the edge. The robot therefore retains enough intelligence to make time-critical decisions while selectively exchanging higher-level information with external computing tiers.
The on-premise tier provides shared computational capability within the factory, warehouse, hospital, campus, mine, construction site, or other operational environment. GPU servers, storage systems, networking equipment, and fleet services can aggregate information from multiple robots. Because these resources are physically closer than remote cloud infrastructure, they can support workloads requiring substantial computation while maintaining relatively predictable network latency and organizational control.
Fleet intelligence is a central function of the on-premise tier. Robot positions, task status, battery levels, local maps, detected hazards, traffic information, and operational constraints can be combined into a shared representation of the site. Fleet managers can perform task allocation, traffic coordination, global route planning, shared map maintenance, and resource optimization while allowing each robot to independently resolve immediate local conditions.
On-premise computing can also host larger world models, VLA services, optimization algorithms, simulation workloads, and centralized inference that exceed the practical compute or power budget of individual robots. Such services can provide strategic recommendations, long-horizon predictions, or complex reasoning. However, remote results must be timestamped and validated because delayed infrastructure intelligence should never override newer edge observations or immediate safety constraints.
The cloud forms the third tier and provides computational elasticity beyond the capacity of a single operational site. Large GPU clusters can support foundation-model training, large-scale simulation, dataset processing, global analytics, hyperparameter optimization, and long-running experiments. Cloud infrastructure is particularly valuable when computational demand changes significantly over time and permanently installing equivalent resources at every site would be economically inefficient.
Cloud services can also integrate experience across geographically distributed fleets. Selected operational data from multiple sites can contribute to larger datasets, enabling models to learn from diverse environments, robot configurations, tasks, and failure conditions. Improvements derived from this aggregated experience can be evaluated, versioned, and eventually distributed back through on-premise infrastructure to appropriate edge platforms, creating a fleet-wide learning cycle.
The three tiers should not be treated as interchangeable computing locations. Each function should be placed according to its maximum acceptable latency, required availability, data volume, compute demand, privacy constraints, and consequences of communication failure. Millisecond-level stabilization belongs at the edge, site-wide coordination naturally fits on-premise infrastructure, and computationally intensive training or global analysis is generally better suited to cloud-scale resources.
Communication between tiers should therefore be hierarchical and asynchronous whenever possible. Edge robots can transmit states, events, selected observations, compressed features, diagnostics, and task progress to on-premise services. On-premise infrastructure can aggregate and filter this information before transferring appropriate datasets or analytics to the cloud, reducing unnecessary wide-area bandwidth consumption and preventing every robot from independently communicating massive raw data streams.
The reverse path distributes intelligence toward physical execution. Cloud-generated models, policies, configuration packages, or global knowledge can first pass through validation and deployment management. On-premise infrastructure can test compatibility, stage releases, monitor performance, and coordinate fleet deployment before approved versions reach edge devices. This hierarchy provides stronger control than directly updating every robot from a remote cloud service.
Connectivity loss must be incorporated into the architecture from the beginning. If cloud connectivity disappears, the on-premise tier should continue supporting local fleet operation. If communication between a robot and on-premise infrastructure also fails, the edge should retain sufficient autonomy to continue safely, complete appropriate local actions, reduce functionality, or enter a controlled safe state. Intelligence therefore degrades progressively rather than collapsing after a single network failure.
Data placement follows similar hierarchical principles. Immediate sensor buffers and operational state reside at the edge, site-level logs and fleet datasets can be maintained on-premise, and long-term datasets or large training corpora may be stored in cloud infrastructure. Filtering, compression, event selection, retention policies, and data lifecycle management prevent uncontrolled growth while preserving information valuable for debugging, validation, learning, and regulatory requirements.
Security and data sovereignty must span all three tiers. Authentication, encryption, access control, secure boot, signed software, network segmentation, audit logging, and controlled model deployment protect the path from cloud services to physical actuators. Sensitive facility data or human observations may be restricted to edge or on-premise systems, while only anonymized, summarized, or explicitly approved information is transferred to external cloud environments.
Resource and energy optimization can also exploit hierarchical computing. Robots have strict battery, cooling, weight, and compute limitations, whereas on-premise servers can provide greater sustained performance and cloud resources can scale dynamically. Workloads can therefore migrate or be partitioned according to urgency and cost, keeping latency-sensitive functions local while moving noncritical heavy computation toward infrastructure with more appropriate power and thermal capacity.
The architecture supports a continuous Physical AI ...
![Physical AI Hardware Software Co Design — Ch.08: Reference Architectures [EN]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1neeq6b7ev4xkxc8i80i.png)





