AI & ML
One Verdict Is Not Enough: Regression Testing Learned World Models and Their Test Harnesses
arXiv cs.SE0 views
arXiv:2610.41130v1 Announce Type: new Abstract: Learned world models ship as software components whose model versions, preprocessing, and test harnesses change between releases. Work on model updates already counts per-sample regressions, or negative flips, that an accuracy check hides. Two problems remain: a test can pass under one configuration and fail under another, and a harness fault can produce erroneous scores and wrong verdicts without a crash. We present TetraState, a regression-testing protocol for paired possible/impossible video tests. Each test runs under four fixed model-native configurations; its number of correct verdicts assigns one of four outcome classes. Inconclusive tests and failed executions are reported separately alongside the verdict rate. Revisions are compared by per-test outcome transitions, and the harness is versioned as part of the system under test. On all 506 IntPhys 2 Main tests and two heterogeneous systems, Cosmos 3 Nano and V-JEPA 2, two V-JEPA 2 harness revisions move modal accuracy by only 0.2 and 2.6 points yet change 11 and 214 test labels; in the second, accuracy rises although fewer tests are judged correctly, and 43 regressions keep the modal verdict, so a modal-verdict flip check misses them. These results outline a research agenda on configuration-level regression oracles for learned components.
Source:https://arxiv.org/abs/2610.41130PreviousRepGen: Reproducing Deep Learning Bugs through Context-Aware Code Generation and VerificationNextLineage Is Memory: Parent Inheritance and Feedback Disclosure in Short-Budget LLM-Guided Optimization
Related posts
Federated AI Still Has a Central Coordinator Problem
0HackerNoon

The iPhone Duo has more than 100,000 optimized apps, Apple says
0The Verge
Product JSON-LD for WooCommerce: the fields AI shopping answers can actually use
0dev.to
It's a weird time to buy a laptop, but the new MacBook Pro doesn't skimp on quality
0Ars Technica
Could potential AI clones of GTA VI force a quicker official port?
0Ars Technica

AI is the new dating dealbreaker
0The Verge