Open Source
Video2SwimFish: An Automated Pipeline for Reconstructing Controllable Fish Models and Biological Locomotion from Real Fish Videos
arXiv cs.RO0 views
arXiv:2609.38966v2 Announce Type: replace Abstract: We present Video2SwimFish, an automated pipeline and benchmark for building controllable fish assets from real-fish videos for underwater embodied AI. Given synchronized multi-view videos of an individual fish, the pipeline generates a metrically scaled deformable mesh from a VLM-selected canonical frame, builds internal articulation adapted to that individual's morphology through a VLM actor-critic loop, and extracts a Biological Locomotion Manifold (BLM). The BLM provides a low-dimensional action space bounded by real-fish motion. Compared with learning directly in joint space, it lets the individual swimming policy of each reconstructed fish converge faster and reach a higher task success rate. We release two paired datasets: synchronized top- and front-view recordings of 120 individual fish across 6 species, and the controllable assets and individual swimming policies derived from them. Because every asset is tied to the animal it came from, the dataset supports a benchmark that evaluates locomotion learning not only on task success but on fidelity to that individual's locomotion behavior. We evaluate on two tasks, 3D trajectory following and target reaching, and a case study in which a simulated BlueROV underwater robot captures the assets. We further validate biological realism with a human study in which fish experts rate the simulated fish, as a complement to the automated metrics. We find that task success and locomotion fidelity do not necessarily improve together, and we identify faithful reproduction of individual animal locomotion as an open challenge for the community. Project website: https://hangongchen.github.io/video2swimfish-web/
Source:https://arxiv.org/abs/2609.38966PreviousRAVEL: Asynchronous Rolling Inference for Flow-Based Vision-Language-Action ModelsNextPILOT: Guiding Long-Horizon World-Model Planning for Robot Manipulation with Imagined Subgoals and Policy Priors
Related posts

Are AI-assisted commits reverted more often? I checked 663 matched pairs. The honest answer is 'can't tell'
0dev.to GitHub
Your AI Coding Assistant Can Now Reach Your Files and Database. Who Decides What It Touches?
0dev.to GitHub

7 things to check before an AI coding agent touches your repo
0dev.to GitHub
Polyurethane Adhesive Manufacturer
0dev.to News
Use Your AI Agent to Activate Your "Weak Ties"
0dev.to Productivity
Building a snippet keyboard for iPhone and a paste picker for the Mac menu bar
0dev.to Productivity