AI & ML
PILOT: Guiding Long-Horizon World-Model Planning for Robot Manipulation with Imagined Subgoals and Policy Priors
arXiv cs.RO0 views
arXiv:2610.14335v2 Announce Type: replace Abstract: Latent world-model planning has shown promising results in goal-conditioned robotic manipulation, but remains challenging for long-horizon planning due to limited temporal guidance and inefficient action search. A distant goal image offers little guidance for preparatory actions, capturing intermediate subgoal images by hand is impractical, and uninformed sampling makes action search inefficient. We propose PILOT (Policy-Initialized Local Optimization with Temporal subgoals), which brings two foundation models into latent world-model planning: a video model turns a language instruction into visual subgoals, and a vision-language-action policy initializes the cross-entropy method (CEM). CEM refines action sequences toward each subgoal with an action-conditioned world model, and no goal image is needed at test time. With both models fine-tuned on 40 demonstrations, PILOT achieves 68.8% success on four real UR5e tasks, compared with 55.0% without policy initialization, 13.8% for policy-only execution, and 0.0% for direct final-goal planning. In simulation, PILOT exceeds direct execution of the same policy by 30 percentage points on long Push-T tasks and by 22 percentage points on LIBERO-10. Subgoals and the policy prior are complementary: better subgoals and stronger priors each raise success, and even a weak policy that succeeds alone on only 3.3% of medium Push-T tasks is an effective prior.
Source:https://arxiv.org/abs/2610.14335PreviousVideo2SwimFish: An Automated Pipeline for Reconstructing Controllable Fish Models and Biological Locomotion from Real Fish VideosNextRollout then Recycle: 3,718 Labelled Real-Robot Episodes from Benchmarking 7 Policies on 12 Tasks
Related posts
One Leaked Prefix Can Take Down Three Clouds: Route Filtering for Multi-Cloud Estates
0dev.to
Global stock markets slump as Anthropic's record IPO fails to stem AI sell-off
0Hacker News

The Download: inside the quest to make our minds immortal
0MIT Technology Review
The Birdfy Gifts I’d Buy for Backyard Birders
0Wired

Rocket Report: A weekend of ups and downs; Stoke celebrates engine milestone
0Ars Technica
California AG subpoenas Nvidia over worries about Huang’s ties with China
0TechCrunch