Open Source
ManiTaskGen: A Comprehensive Task Generator for Benchmarking and Improving Vision-Language Agents on Embodied Decision-Making
arXiv cs.RO0 views
arXiv:2505.20726v3 Announce Type: replace Abstract: Building embodied agents capable of accomplishing arbitrary tasks is a core objective towards achieving embodied artificial general intelligence (E-AGI). While recent work has advanced such general robot policies, their training and evaluation are often limited to tasks within specific scenes, involving restricted instructions and scenarios. Existing benchmarks also typically rely on manual annotation of limited tasks in a few scenes. We argue that exploring the full spectrum of feasible tasks within any given scene is crucial, as they provide both extensive benchmarks for evaluation and valuable resources for agent improvement. Towards this end, we introduce ManiTaskGen, a novel system that automatically generates comprehensive, diverse, feasible mobile manipulation tasks for any given scene. The generated tasks encompass both process-based, specific instructions (e.g., "move object from X to Y") and outcome-based, abstract instructions (e.g., "clear the table"). We apply ManiTaskGen to both simulated and real-world scenes, demonstrating the validity and diversity of the generated tasks. We then leverage these tasks to automatically construct benchmarks, evaluating high-level embodied decision-making through a grounded skill interface for agents built upon existing vision-language models (VLMs). Furthermore, we propose a simple yet effective method that utilizes ManiTaskGen tasks to enhance embodied decision-making. Overall, this work presents a cross-scene task generation framework facilitating both benchmarking and improvement of embodied decision-making agents. Our code is available at https://manitaskgen.github.io.
Source:https://arxiv.org/abs/2505.20726PreviousHumanSense AI — Turning Human Signals into Real-Time InsightsNextVLD: Visual Learned Distance as an Interface for Reinforcement Learning Navigation
Related posts
Accuracy potential of visual localization exploiting high-end street-level imagery
0arXiv cs.RO
BLaFA: Bridging Language to 3D Functional Affordances for Dexterous Robotic Grasping in Tool Use
0arXiv cs.RO
DexHOT: Learning Hand-Object Configuration Transfer for Dexterous Manipulation
0arXiv cs.RO
CompVLA: A Variable Compliance Vision-Language-Action Model for Contact-rich Manipulation
0arXiv cs.RO
DexTeleop-0: Force-Aware Bimanual Dexterous Teleoperation with Ego-Centric Perception towards Shared Autonomy
0arXiv cs.RO
ObjView-Bench: Rethinking Difficulty and Deployment for Object-Centric View Planning
0arXiv cs.RO