Backend
JEPA Dots: Dual-Latent World Model for Language-Guided Robot Control
arXiv cs.RO0 views
arXiv:2611.24062v1 Announce Type: new Abstract: Predictive world models for language-guided robot control face a fundamental conditioning challenge: instructions should influence action selection without altering the predicted consequences of fixed actions under identical observations. To address this challenge, we introduce JEPA Dots (Dynamics-Oriented Task Separation), a compact dual-latent framework that jointly learns visual dynamics and language-guided control through separate dynamics and task representations. Language-conditioned task features guide a flow-matching policy, while masked attention restricts world prediction to instruction-independent dynamics history and supplied actions. In closed-loop evaluation on 40 trained LIBERO tasks, JEPA Dots achieves 84.9% overall success with approximately 39.1M parameters, compared with the published 78.7% of NanoVLA-S with 161.0M parameters under a different evaluation protocol. In the reference experiment, JEPA Dots also improves overall success from 76.0% for our task-ID-conditioned adaptation baseline to 84.9%, with LIBERO-Goal success increasing from 67.1% to 88.5%. Instruction-switching experiments further confirm that task representations and policy outputs respond to language changes, while dynamics representations and world predictions remain invariant under fixed observations and supplied actions.
Source:https://arxiv.org/abs/2611.24062PreviousWAM-Policy: World Action Models with On-Policy Distillation for Quadrupedal Loco-ManipulationNextSemPhys: Semantic-Physical Co-Learning for Humanoid Motion Generation via Differentiable World Models
Related posts
Learning to Decide with AI Assistance under Human-Alignment
0arXiv cs.HC
From Defense to Advocacy: Empowering Users to Leverage the Blind Spot of AI Inference
0arXiv cs.HC
Noise-induced enhancement of regime lifetimes -- A data-driven approach using deterministic trajectories
0arXiv cs.NA
fOGA: An Orthogonal Greedy Algorithm for Fractional Laplacian Problems
0arXiv cs.NA
Bad Genius: Counterfactual-Guided Harness Evolution Beyond Task-Specific Shortcuts
0arXiv stat.ML
Kinetic Langevin Splitting Schemes for Constrained Sampling
0arXiv stat.ML