AI & ML
Build-and-Find: An Effort-Aware Protocol for Evaluating Agent-Managed Codebases
arXiv cs.SE0 views
arXiv:2605.06136v2 Announce Type: replace Abstract: Most coding-agent benchmarks ask whether generated code behaves correctly. That remains essential, but repository-level engineering is increasingly agent-managed: one agent writes a repository, and later agents inspect, audit, or extend it as working context. In that setting, a generated repository is not only an answer to a task but also a communication artifact for future work. Even when strong agents nearly satisfy the visible behavioral objective, repositories can differ in how clearly they expose the intended behavior and design choices behind that behavior. We introduce Build-and-Find to evaluate whether downstream agents can efficiently recover intended choices from generated repositories. For each task, a builder creates a codebase from a hidden specification; a finder receives only the codebase and a specification-traced question bank. The protocol separates behavioral correctness from artifact-side recovery and reports recovery accuracy, repeatability, implementation coverage, and inspection effort. Evaluation proceeds in two stages: the first uses recovery accuracy and stability as gates to establish that finders can reliably recover the intended choices; the second compares inspection effort by the same finder across artifacts that pass these gates and allow the same intent to be recovered. Lower effort suggests that the artifact makes that intent easier to locate. We find that several frontier models approach saturation in intent recovery on our task pack, yet second-stage inspection effort reveals differences among their generated artifacts, providing a dimension beyond accuracy for comparing artifact legibility. We also audit artifacts to distinguish unimplemented specification claims from finder failures and check whether correct answers are supported by artifact evidence. We release the harness and benchmark data for reproducibility and extension.
Source:https://arxiv.org/abs/2605.06136PreviousCloud-OpsBench: A Reproducible Benchmark for Agentic Root Cause Analysis in Cloud SystemsNextSCDBench: A Benchmark for LLM-Based Smart Contract Decompilers
Related posts
![Path Planning and Autonomous Navigation — Ch.24: Advanced Autonomous Behaviors [KR]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fgmggiae4nyxpukfvgqba.png)
Path Planning and Autonomous Navigation — Ch.24: Advanced Autonomous Behaviors [KR]
0dev.to
화면이 사람을 붙잡는 힘 — AB 1709 소송이 개발자에게 던지는 질문
0dev.to
![Path Planning and Autonomous Navigation — Ch.12: Reversing and Parking [EN]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1k9m20qhwzb1csyeu5v1.png)
Path Planning and Autonomous Navigation — Ch.12: Reversing and Parking [EN]
0dev.to
![Path Planning and Autonomous Navigation — Ch.13: Docking and Charging [EN]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ff66zllmrqlif2vsyczxb.png)
Path Planning and Autonomous Navigation — Ch.13: Docking and Charging [EN]
0dev.to
![Path Planning and Autonomous Navigation — Ch.16: Rough Terrain Driving [EN]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmltd2hs9q5yutx3myjia.png)
Path Planning and Autonomous Navigation — Ch.16: Rough Terrain Driving [EN]
0dev.to

Implementing the Modern Data Strategy: Microsoft Fabric Blueprint
0dev.to