AI & ML
One Verdict Is Not Enough: Regression Testing Learned World Models and Their Test Harnesses
arXiv cs.SE0 views
arXiv:2610.41130v1 Announce Type: new Abstract: Learned world models ship as software components whose model versions, preprocessing, and test harnesses change between releases. Work on model updates already counts per-sample regressions, or negative flips, that an accuracy check hides. Two problems remain: a test can pass under one configuration and fail under another, and a harness fault can produce erroneous scores and wrong verdicts without a crash. We present TetraState, a regression-testing protocol for paired possible/impossible video tests. Each test runs under four fixed model-native configurations; its number of correct verdicts assigns one of four outcome classes. Inconclusive tests and failed executions are reported separately alongside the verdict rate. Revisions are compared by per-test outcome transitions, and the harness is versioned as part of the system under test. On all 506 IntPhys 2 Main tests and two heterogeneous systems, Cosmos 3 Nano and V-JEPA 2, two V-JEPA 2 harness revisions move modal accuracy by only 0.2 and 2.6 points yet change 11 and 214 test labels; in the second, accuracy rises although fewer tests are judged correctly, and 43 regressions keep the modal verdict, so a modal-verdict flip check misses them. These results outline a research agenda on configuration-level regression oracles for learned components.
Source:https://arxiv.org/abs/2610.41130PreviousRepGen: Reproducing Deep Learning Bugs through Context-Aware Code Generation and VerificationNextLineage Is Memory: Parent Inheritance and Feedback Disclosure in Short-Budget LLM-Guided Optimization
Related posts
Merge builds product and business tools with OpenAI technology
0OpenAI Blog
How to Survive (Not Thrive) as a Salaryman
0HackerNoon
Debugging macOS LaunchAgent Failures With launchctl and Exit Code 78
0HackerNoon

Building Data Observability for Banking Pipelines
0HackerNoon
OpenMatter Network Positions Decentralized Computing as an Alternative To Data Centers for AI
0HackerNoon
BMW bumps battery for the better: The 2027 i7 60 xDrive, driven
0Ars Technica