Open Source
AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluation
arXiv cs.SE0 views
arXiv:2605.12925v5 Announce Type: replace Abstract: Evaluation of software engineering (SWE) agents is dominated by a binary signal: whether the final patch passes the tests. This outcome-only view treats a principled solution and a chaotic trial-and-error process as equivalent. We show that this equivalence is empirically false. We evaluate 2,614 OpenHands trajectories from eight model backends on 60 SWE-bench Verified tasks. Of these, 47 have enough passing trajectories to construct task-level process references, yielding a 1,815-trajectory evaluation subset. Among passing trajectories in this subset, 10.7% exhibit behavior we call a Lucky Pass: regression cycles, blind retries, missing verification, or temporally disordered exploration, implementation, and verification. We introduce AgentLens, a framework for process-level assessment of SWE-agent trajectories, and release AgentLens-Bench, a dataset of 1,815 trajectories annotated with quality scores, waste signals, divergence points, and 47 task-level Prefix Tree Acceptor (PTA) references. AgentLens builds PTA references by merging multiple passing solutions for the same task, and uses a context-sensitive intent labeler to assign actions to Exploration, Implementation, Verification, or Orchestration based on trajectory history rather than tool identity alone. On AgentLens-Bench, the quality score separates passing trajectories into Lucky, Solid, and Ideal tiers and further decomposes Lucky Passes into five recurring mechanisms. Across the eight model backends, Lucky rates range from 0.5% to 23.2%, and some models move by as many as five rank positions when ranked by quality score instead of pass rate. We publicly release AgentLens-Bench and the AgentLens SDK at https://github.com/agentlens-framework/agentlens, with a project page at https://agentlens-framework.github.io/agentlens/.
Source:https://arxiv.org/abs/2605.12925PreviousReBoT-MiCS: Reliability-Aware Bounding Time for Enhancing Timing Behavior of Embedded Mixed-Criticality SystemsNextWhere Is the Cost of Third-Party API Routers in Agentic Software Development?
Related posts

AGPL vs MIT vs Open Core: Choosing Open Source Collaboration Software
0dev.to OpenSource
From exposed secrets to managed credentials: Reducing credential risk across the application lifecycle
0HashiCorp Blog
Approximation of the L\'evy-driven stochastic heat equation on the sphere
0arXiv cs.NA
EgoSwarm: Learning Decentralized Robot Swarm Intelligence from Human Swarm Demonstrations
0arXiv cs.RO
EasyNav: A Modular Architecture for Real-Time Navigation Across Heterogeneous Robot Environments
0arXiv cs.RO
DynLimit: A Simulation Benchmark for Dynamics-Critical Planning and Control
0arXiv cs.RO