AI & ML
What Does a Repair Agent Gain from Remembering Its Failure? An Exploratory Study on Real Python Bugs
arXiv cs.SE0 views
arXiv:2610.41433v1 Announce Type: new Abstract: When a coding agent fails a repair task, it is often retried with a record of the failure: the transcript, a self-written lesson, or extracted test signals. We ask what such failure memory changes in the retry. On 75 BugsInPy bugs, we retried each of 181 failed attempts of gpt-5.4-mini under five conditions, with a predeclared analysis whose deviations we report. Two predeclared follow-ups repeated the plain-retry, transcript and lesson conditions with the larger gpt-5.4 and added a notice-only control. On the bug-revealing tests, every memory condition gained over a plain retry in point estimate (+5.5 to +9.9 points over 29.8%), but none significantly. Five further findings stand out. First, a model-written lesson showed no advantage over the transcript it summarises (-1.7 points, 95% CI [-8.7, +4.9]; -3.5 for gpt-5.4), although it costs an extra model call. Second, the transcript may do more than tell the agent that it failed: a notice alone produced no significant change, while the transcript resolved 12.2 points more failures than the notice for gpt-5.4-mini (CI [+4.2, +20.0], partial run of 156 failures; the only significant one of twelve predeclared contrasts, and not significant for gpt-5.4). Third, post hoc, fewer retries with memory ended without modifying any Python file, while the share of editing retries that touched a file the fix changes was near ceiling. Fourth, in a post hoc full-test-suite check, retries that had also seen the previous attempt's test results more often failed other tests that the reference fix passes, which shrinks their apparent gains. Fifth, outcomes varied far more between failures than between conditions, and we give the sample sizes a confirmatory study would need.
Source:https://arxiv.org/abs/2610.41433PreviousArchitectural Debt Enforcement under Incomplete Observation: A Controlled Study of Ratchets and Evidence QualificationNextPost-SQLFix: Property-Guided Repair of LLM-Generated SQL with Runtime Verification
Related posts

'AI security controls are only as strong as the wider application environment around them': OpenAI forced to patch after experts bypass ChatGPT 'lockdown mode' to gain access to entire chat history
0TechRadar

ExpressVPN’s browser extension now blocks malicious sites – even when the VPN is off
0TechRadar

This AI enhanced property listing almost drove me insane
0TechRadar

T-Mobile chairman says Starlink Mobile needs $200 billion to compete
0TechRadar

Two more Garmin smartwatches are down to record-low prices
09to5Google
Four Hidden Config Settings That Stop Token Leaks and Unapproved Writes in OpenAI Codex and Claude Code
0dev.to