AI & ML
On the Limits of Execution Traces for Repairing Assertion Failures in LLM-Generated Code
arXiv cs.SE0 views
arXiv:2611.09212v1 Announce Type: new Abstract: Iterative repair returns a failed program's error to a large language model (LLM) and asks it to fix it. After iterative repair, the remaining failures are mostly assertion failures: with four Qwen2.5 models, they make up 79% of residual failures on BigCodeBench, all on HumanEval and 93% on LiveCodeBench. We argue, through an information view of the feedback, that an assertion failure says little about where the fault lies. We propose Breakpoint Debugging (BPD), which returns the failed program annotated inline with its variables' runtime values. We take the tasks that iterative repair failed to fix with five repair attempts and compare two ways of spending five further generations on them: BPD, and resampling. On BigCodeBench, BPD repairs 26.7% of these tasks against 22.4%, and its advantage grows with model size. On HumanEval, BPD outperforms resampling but only slightly, and on LiveCodeBench, fresh sampling repairs more (6.5% against 4.9%). Furthermore, BPD reduces unchanged-code resubmissions from 28% to 8% on BigCodeBench. Using only the first failing test's trace, it outperforms tracing all failing tests while using fewer tokens than both all-trace BPD and plain iterative repair. On LiveCodeBench, BPD gives no gain even on the public tests, so a visible failing input is not enough for a trace to help. These results match the information view: a trace helps only when the test exposes a failing input, and the fault is an implementation slip; otherwise, a repair loop should resample rather than trace.
Source:https://arxiv.org/abs/2611.09212PreviousGreyZone: Production-quality Agentic Application ThreatsNextBOXBOX: Recursive Harness Self-Improvement for Long-Running Performance Optimization
Related posts

Agents draft. People decide.
0dev.to UX

โค้ดเพิ่ม 400% แต่คุณไม่ได้เร็วขึ้น 400%: GitHub ชี้คอขวดใหม่อยู่ที่การรีวิว
0dev.to GitHub

Drago: Your GitHub Dragon
0dev.to GitHub
Case Study: Automating Tableau to Power BI Migration with Up to 90% Accuracy.
0dev.to Data
Vom Bluttest ins Molekül: Warum die Alamar/Nightingale-Studie räumliches Denken braucht...
0dev.to Data
10x Genomics가 Anthropic의 Claude AI를 Chromium 플랫폼에 통합한 'Sentira'를 발표했습니다. 단일세포 데이터 분석을 가...
0dev.to Data