AI & ML
$Z$-sight: Unifying Vision-Language-Action and Latent World Modeling
arXiv cs.RO0 views
arXiv:2611.24940v1 Announce Type: new Abstract: We introduce $Z$-sight, a unified robot foundation model that couples vision-language-action (VLA) and generative latent world modeling. Video-based world action models (WAMs) show the value of joint future-and-action generation for control, yet inherit the heavy cost of video generators. Recent latent-space policies avoid pixel generation, yet typically align intermediate features or rely on frozen target encoders, decoupling future prediction from the VLM's native feature space. $Z$-sight instead predicts the future directly in the visual token space in which its VLM perceives the present. Coupling a foresight expert and a continuous action expert with the VLM in a mixture-of-transformers, $Z$-sight jointly denoises future spatial tokens and actions at inference without pixel reconstruction or external encoders. Trained on human video and robot trajectories, $Z$-sight achieves 81.9% success on SIMPLE (+9.6 points over the strongest baseline) and 73.3% across six Unitree G1 tasks (+20.0 points) at 110 ms per chunk, while remaining competitive on tabletop benchmarks.
Source:https://arxiv.org/abs/2611.24940PreviousPhoenix: Learning Scene Structure Representations for Generalizable World-Action ModelsNextNeural Circuit Architectural Priors for Rat Locomotion
Related posts

How to Interpret Logistic Regression and Odds Ratios
0dev.to
Kubernetes Service Account Tokens Do Not Need to Ride Along
0dev.to Kubernetes
Export a calendar day to ICS for your system calendar
0dev.to OpenSource
Comparing Axleta Repository Packer with Gitingest
0dev.to Programming

Animating and rigging by hand is sometimes hard....
0dev.to Programming
Anthropic is bullshitting you about subagents and loops.
0dev.to Programming