AI & ML
Pdb or Not to Be: Comparing Debugging Interfaces With Pass$_{X}$@$k$
arXiv cs.SE0 views
arXiv:2611.08666v1 Announce Type: cross Abstract: Interactive debuggers are indispensable to human software development, yet coding agents overwhelmingly rely on edit-execute-observe loops, using common bash tools, test scripts, and print statements to debug via repeated shell commands. Stepping through a debugger provides live access to runtime state, which may enable different problem-solving capabilities. However, prior work typically introduces the debugger as an additional tool or through a new subagent, making its individual contribution difficult to isolate from the associated scaffold changes. To address this, we compare a debugger-only agent against a bash-only agent on all 500 SWE-bench Verified instances, holding the rest of the scaffold fixed. We evaluate both architectures on two models, GPT-OSS-120B and Nemotron-3-Ultra-550B, with three independent runs per architecture $\times$ model configuration. Crucially, to distinguish interface specialization from expanded coverage under repeated sampling, we introduce pass$_X$@$k$, a cross-agent pass@$k$ that provides an unbiased estimator of the probability that at least one of $k$ agents is successful. Although the bash and debugger agents solve partially disjoint subsets of the benchmark in any single pair of runs, our pass$_X$@$2$ estimator reveals that this is largely due to run-to-run variance rather than tool complementarity. Across both models, we find that two bash rollouts outperform the combination of one bash and one debugger rollout. Our findings highlight the importance of accounting for stochastic variance in evaluating agent and tool diversity, and we hope that pass$_X$@$k$ will serve as a useful framework for the community.
Source:https://arxiv.org/abs/2611.08666PreviousAegis OS: A Transactional Multi-Tenant Operating System Runtime for Eliminating Authorization Fatigue in Autonomous Coding AgentsNextAnalyzing Code Provenance in LLMs for Programming Education
Related posts
![AMR AI and Embodied Intelligence — Ch.11: Self Supervised Robot Learning [EN]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fh8y4egw0kdqtzf6m3hzx.png)
AMR AI and Embodied Intelligence — Ch.11: Self Supervised Robot Learning [EN]
0dev.to
![AMR AI and Embodied Intelligence — Ch.13: Human Robot Interaction [EN]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fjm36fpwnygomah6qsu4n.png)
AMR AI and Embodied Intelligence — Ch.13: Human Robot Interaction [EN]
0dev.to
![AMR AI and Embodied Intelligence — Ch.14: VLM and VLA [EN]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fo06nxmfdm3t6o1oznau9.png)
AMR AI and Embodied Intelligence — Ch.14: VLM and VLA [EN]
0dev.to
370 Books in 20 months: South African author defends her use of AI
0Hacker News
The climate case for AI has not been made
0MIT Technology Review
I Let Pangram Loose on My Gmail Inbox. Here’s How Much AI It Found
0Wired