Hey HN, I'm Salman, a research engineer at GDM Robotics on the team that built this robotics plugin.

Whether you're a robotics founder, data company, making RL environments, or just getting FOMO seeing all the cool robotics demos on X, you can just plug in a robot and start prompting away. The plugin we've written up includes skills that steer the model toward following best practices.

The benefits of using the plugin are: - fewer tokens (and therefore thinking time) needed to do bring-up the robot and to reach a successful attempt on your task - higher fidelity real-to-sim-to-real when creating MuJoCo assets and environments - leveraging strong motion planning, visual servoing, and collision avoidance

We have tested on SO-101, YAM, and Franka Duo, but you don't even need real arms to test it out. You can just create cool environments in sim and then prompt the model to control a simulated robot in the environment! (Just make sure to tell it not to use privileged sim state if you want to actually evaluate how good the perception and model intelligence is).

Just point your agent to the repo, good luck, and happy tinkering!

Let us know if you have any questions! Will answer anything here or feel free to post on the GitHub repo directly.


Comments URL: https://news.ycombinator.com/item?id=50553913

Points: 7

# Comments: 0