AI & ML
Objectives as Policies: Vision-Language Model Cost Programs for Contact-Rich Manipulation
arXiv cs.RO0 views
arXiv:2611.18298v1 Announce Type: new Abstract: Vision-language models (VLMs) have shown strong semantic and spatial reasoning capabilities in 3D environments. Recently, connecting VLMs with model-predictive controllers (MPC) has emerged as a paradigm for combining strong semantic reasoning with physics-based planning. However, connecting these two for contact-rich robotic manipulation requires a task representation that bridges the two and captures how physical interactions can be reasoned by VLMs. In this work, we introduce Objectives as Policies (OaP), a framework that connects a pretrained VLM to a shared sampling-based MPC through an editable objective program. Each program specifies a sequence of stages for completing a task, combining running and terminal costs from a cost library with stage completion conditions. Given a natural-language task instruction and scene observation, we build a simulation-based digital twin through real-to-sim, and the VLM generates an objective program that is executed by the MPC in the simulation. The program is iteratively updated by the VLM based on MPC rollouts, before being deployed in the real world. We evaluate our framework in both simulation and the real world on four tabletop manipulation tasks, and show that it outperforms strong baselines. Additional videos and examples are available on the project website: https://zihaolu001.github.io/oap-project/
Source:https://arxiv.org/abs/2611.18298PreviousNPU-Accelerated GNSS Signal AcquisitionNextPROOF: Perceptive Robustness via Out-of-Distribution Fallback for Legged Locomotion
Related posts
Stopping models closed under pgf composition, and the stability of randomly stopped model extensions
0arXiv stat.TH
Testing the equality of the copulas of two multivariate populations with Bernstein polynomials
0arXiv stat.TH
Quantifying Dependence Effects on $k$-out-of-$n$ Reliability Relative to the Independent Benchmark
0arXiv stat.TH
Causal and conditional tail coefficients: asymptotics, inference and graph recovery
0arXiv stat.TH
Characterizing M-estimators using canonical divergences from information geometry
0arXiv stat.TH
Statistical guarantees for signature kernel ridge regression
0arXiv stat.TH