AI & ML
From Video Frames to Robot Meta-Frames: A Unified Latent Interface for World Action Models
arXiv cs.RO0 views
arXiv:2611.14365v1 Announce Type: cross Abstract: Pretrained video diffusion models (VDMs) offer a powerful foundation for World Action Models (WAMs), which jointly predict robot actions and future observations. Building such models requires injecting low-dimensional robot signals (e.g. proprioception, actions, and values) into a latent space shaped by natural videos, and a key design choice is the interface through which these signals enter: existing schemes treat it as a formatting detail, spending a separate latent frame on each signal or bolting on dedicated encoders and decoders. We argue that signal injection is instead a distribution-alignment problem: how well a representation matches the prior of the pretrained VDMs during denoising. Using the relative reconstruction error of a frozen VDM denoiser as a probe, we find that conventional spatial encodings sit far off this prior, whereas frequency-domain packing yields latents that the denoiser corrects far less. Motivated by this diagnostic result, we propose MetaWAM, which folds all robot signals into a single unified meta-frame: each signal is assigned to a disjoint frequency-domain channel band and lifted to the spatial latent by an inverse DCT, yielding a decodable interface aligned with the pretrained VDM prior without any added module or architectural change. A fixed band layout together with slot-aware routing lets the same meta-frame support policy, world-model, and value prediction over a shared set of robot variables. On RoboCasa, MetaWAM reaches 68.9% success while collapsing the per-signal frames into one and cutting inference latency by 39%, and attains 85.0% success on three real-world Franka tasks. Code and models are available at https://github.com/showlab/MetaWAM
Source:https://arxiv.org/abs/2611.14365PreviousHiPFuse: Real-Time LiDAR Semantic Segmentation for Robotic Perception on Edge DevicesNextUGG: Unified Generative Geometry Model
Related posts
Pyrite benchmarks protein-ligand pose ranking where experts perform near chance
0dev.to Machine Learning

How I Connected Muse to Feishu So It Works When My Computer Is Off
0dev.to Tutorial
Your README proves the work. It doesn't prove what it felt like before it worked.
0dev.to Career
Delaware Layoffs November 2026: WARN Numbers Explained
0dev.to Career
Your Mentee Wants You to Look at Their Take Home Test Before They Submit It
0dev.to Career
I lost an invoice once and decided billing should be a form, not a task
0dev.to Beginners