AI & ML
From Video Frames to Robot Meta-Frames: A Unified Latent Interface for World Action Models
arXiv cs.RO0 views
arXiv:2611.14365v1 Announce Type: cross Abstract: Pretrained video diffusion models (VDMs) offer a powerful foundation for World Action Models (WAMs), which jointly predict robot actions and future observations. Building such models requires injecting low-dimensional robot signals (e.g. proprioception, actions, and values) into a latent space shaped by natural videos, and a key design choice is the interface through which these signals enter: existing schemes treat it as a formatting detail, spending a separate latent frame on each signal or bolting on dedicated encoders and decoders. We argue that signal injection is instead a distribution-alignment problem: how well a representation matches the prior of the pretrained VDMs during denoising. Using the relative reconstruction error of a frozen VDM denoiser as a probe, we find that conventional spatial encodings sit far off this prior, whereas frequency-domain packing yields latents that the denoiser corrects far less. Motivated by this diagnostic result, we propose MetaWAM, which folds all robot signals into a single unified meta-frame: each signal is assigned to a disjoint frequency-domain channel band and lifted to the spatial latent by an inverse DCT, yielding a decodable interface aligned with the pretrained VDM prior without any added module or architectural change. A fixed band layout together with slot-aware routing lets the same meta-frame support policy, world-model, and value prediction over a shared set of robot variables. On RoboCasa, MetaWAM reaches 68.9% success while collapsing the per-signal frames into one and cutting inference latency by 39%, and attains 85.0% success on three real-world Franka tasks. Code and models are available at https://github.com/showlab/MetaWAM
Source:https://arxiv.org/abs/2611.14365PreviousHiPFuse: Real-Time LiDAR Semantic Segmentation for Robotic Perception on Edge DevicesNextUGG: Unified Generative Geometry Model
Related posts

Five Details That Make a Bedroom Feel French, From Someone Who Sells French Bedding
0dev.to Design
User-Friendly Mobile App UX Design – Best Practice & Process
0dev.to Design
The empty states in my app blamed the user, so I wrote a linter for that
0dev.to Design
Generic CRMs made me hate CRM, so I built one with only my trade's stages
0dev.to Productivity
Think Before You Solve: The Importance of Planning in Software Engineering
0dev.to Productivity
Attention Fusion for Bridge Deck Delamination Detection
0arXiv eess.IV