Open Source
Render the Robot, Generate the World: One Video World Model Across Embodiments
arXiv cs.RO0 views
arXiv:2611.24925v1 Announce Type: new Abstract: Predicting the consequences of an agent's actions requires modeling both the body's motion and the environment's response. Most video world models attempt to learn both from numerical actions, even though geometry and kinematics already specify the intended motion. We introduce Doppel, an embodiment-agnostic video world model conditioned on camera-aligned renderings of future body configurations and numerical action trajectories. By explicitly providing the intended motion, Doppel can focus on predicting object motion and interaction-driven changes in the environment. Rendered motion also provides a common visual interface across different embodiments and action spaces. To train on robot datasets without camera extrinsics, we recover camera poses and align rendered trajectories with recorded observations. We jointly train a single shared video backbone and conditioning adapter on eight datasets spanning eight robot and human embodiments, from single- and bimanual robot arms to a dexterous humanoid and human hands and bodies, using about 600 hours of video without embodiment labels. Across three robot-arm datasets, Doppel improves PSNR over action-only baselines by 1.00 dB on held-out trajectories and 1.20 dB under out-of-distribution evaluation, while predicting more consistent object motion. Training on all eight embodiments (Doppel-8) improves robot predictions over the three-robot model (Doppel-3) on four of six evaluation sets. Project page: https://doppelwm.github.io/.
Source:https://arxiv.org/abs/2611.24925PreviousTeleoperated Haircutting with Delay Compensation and Passivity-Based Control for Unilateral Internet Teleoperation under Real Communication UncertaintiesNextGrana: Compositional Reward Modeling for Long-Horizon Robot Manipulation
Related posts
Nintendo takes Picofly firmware down from GitHub [video]
0Hacker News
NVFP4 Won't Save Your VRAM: Quantization Tradeoffs on a 12GB Laptop GPU
0dev.to OpenSource
DrawScore: Evaluating Generated CAD Models against Engineering Drawings
0arXiv cs.SE
Predicting Process-Level Behavior: A Benchmarking Study
0arXiv cs.SE
Characterizing GitHub Security Actions Alerts For Web Application Tutorials
0arXiv cs.SE
AeroRSI: Recursive Self-Improvement for Aerial Vision-Language Navigation
0arXiv cs.RO