AI & ML
A Taxonomy of Failing Bug Reproduction Tests on SWT-Bench
arXiv cs.SE0 views
arXiv:2611.13395v1 Announce Type: new Abstract: Automatic Program Repair (APR) tools require high-quality test generation models to validate candidate patches. Despite the advancements of automated bug reproduction testing tools, they continue to struggle with issues of higher complexity. These struggles may be caused by a framework's lack of problem-specific context. We address the unclassified nature of these framework failures by providing a taxonomy of bug reproduction tests. This study uses the AssertFlip and OpenHands frameworks, the two highest-scoring open-source tools on SWT-Bench Verified. The significance of model selection on the causes of failure is also explored by utilizing GPT-5-mini and GPT-4o-mini on both tools. The error traces of these frameworks are categorized to determine granular modes of failure. Through this process, mechanical and implementation failures were the most common failure classes. Overall, a tool's generation approach has a significant impact on precision and coverage, while the selection of the model alters the quality and quantity of generated tests. Specifically, more powerful models with higher reasoning capability tended to present fewer failures due to a lack of technical or problem-specific context. Our experiment also shows evidence that our taxonomy can be used as concrete guidance to augment the issue description to improve the generation quality.
Source:https://arxiv.org/abs/2611.13395PreviousDEPUTYBENCH: Evaluating Contextual Safety in Enterprise AgentsNextThe AI Era's Lines of Code? A Case Study on What an Engineer's Token Spend Measures
Related posts
Voice layer for any website – does it worth?
0Hacker News New
Brazil's AI Development Strategy Needs an Edge Layer – TechPolicy.Press
0Hacker News New
Generative AI and the temporary upskilling of knowledge workers
0Hacker News New
Anthropic Unveils Claude Whiteboards, Claude Sheets, and Claude Data
0Hacker News New
AI Cracks a Quantum Computing Bottleneck
0Hacker News New
Jack and Jill Cut Candidate Screening Costs by 88% with Jev
0Hacker News New