AI & ML
Build-and-Find: An Effort-Aware Protocol for Evaluating Agent-Managed Codebases
arXiv cs.SE0 views
arXiv:2605.06136v2 Announce Type: replace Abstract: Most coding-agent benchmarks ask whether generated code behaves correctly. That remains essential, but repository-level engineering is increasingly agent-managed: one agent writes a repository, and later agents inspect, audit, or extend it as working context. In that setting, a generated repository is not only an answer to a task but also a communication artifact for future work. Even when strong agents nearly satisfy the visible behavioral objective, repositories can differ in how clearly they expose the intended behavior and design choices behind that behavior. We introduce Build-and-Find to evaluate whether downstream agents can efficiently recover intended choices from generated repositories. For each task, a builder creates a codebase from a hidden specification; a finder receives only the codebase and a specification-traced question bank. The protocol separates behavioral correctness from artifact-side recovery and reports recovery accuracy, repeatability, implementation coverage, and inspection effort. Evaluation proceeds in two stages: the first uses recovery accuracy and stability as gates to establish that finders can reliably recover the intended choices; the second compares inspection effort by the same finder across artifacts that pass these gates and allow the same intent to be recovered. Lower effort suggests that the artifact makes that intent easier to locate. We find that several frontier models approach saturation in intent recovery on our task pack, yet second-stage inspection effort reveals differences among their generated artifacts, providing a dimension beyond accuracy for comparing artifact legibility. We also audit artifacts to distinguish unimplemented specification claims from finder failures and check whether correct answers are supported by artifact evidence. We release the harness and benchmark data for reproducibility and extension.
Source:https://arxiv.org/abs/2605.06136PreviousCloud-OpsBench: A Reproducible Benchmark for Agentic Root Cause Analysis in Cloud SystemsNextSCDBench: A Benchmark for LLM-Based Smart Contract Decompilers
Related posts
Our Route Planner Sent Crane Lorries to the Middle of Each Postcode and Some Rural Postcodes Are Miles Wide
0dev.to Data
How much of GPT-6 Astra's weight-editing demo is the harness?
0dev.to Machine Learning
How Pet Groomers Can Track Appointments, Pet Notes and Rebookings
0dev.to Tutorial
Symmetrical Components in Plain English: Fortescue Without the Fog
0dev.to Tutorial
When P2P Pays You and the Account Freezes Anyway
0dev.to Career
Your Company Will Pay for the Course if You Agree to Pay It Back When You Leave
0dev.to Career