AI & ML
One Verdict Is Not Enough: Regression Testing Learned World Models and Their Test Harnesses
arXiv cs.SE0 views
arXiv:2610.41130v1 Announce Type: new Abstract: Learned world models ship as software components whose model versions, preprocessing, and test harnesses change between releases. Work on model updates already counts per-sample regressions, or negative flips, that an accuracy check hides. Two problems remain: a test can pass under one configuration and fail under another, and a harness fault can produce erroneous scores and wrong verdicts without a crash. We present TetraState, a regression-testing protocol for paired possible/impossible video tests. Each test runs under four fixed model-native configurations; its number of correct verdicts assigns one of four outcome classes. Inconclusive tests and failed executions are reported separately alongside the verdict rate. Revisions are compared by per-test outcome transitions, and the harness is versioned as part of the system under test. On all 506 IntPhys 2 Main tests and two heterogeneous systems, Cosmos 3 Nano and V-JEPA 2, two V-JEPA 2 harness revisions move modal accuracy by only 0.2 and 2.6 points yet change 11 and 214 test labels; in the second, accuracy rises although fewer tests are judged correctly, and 43 regressions keep the modal verdict, so a modal-verdict flip check misses them. These results outline a research agenda on configuration-level regression oracles for learned components.
Source:https://arxiv.org/abs/2610.41130PreviousRepGen: Reproducing Deep Learning Bugs through Context-Aware Code Generation and VerificationNextLineage Is Memory: Parent Inheritance and Feedback Disclosure in Short-Budget LLM-Guided Optimization
Related posts

You can edit your videos with AI on Instagram’s new desktop Edits app
0The Verge

You can post AI music straight to SoundCloud from ElevenLabs
0The Verge

The hidden supply chain costs of the AI boom
0The Verge
Aina’s smart ring has a screen, a joystick and a mic so you can talk to AI agents
0TechCrunch
ElevenLabs now lets users publish AI-generated music directly on SoundCloud
0TechCrunch
Dropbox acquires Tigris Data for $350M to bolster its AI infrastructure
0TechCrunch