Frontend
Time-Consistent Evaluation of Hierarchical Code Graphs for SWE Agents
arXiv cs.SE0 views
arXiv:2611.17388v1 Announce Type: new Abstract: Evaluating repository-aware software engineering (SWE) agents requires strict separation between pre-existing repository knowledge and future code evolution, yet standard benchmarks frequently suffer from temporal contamination and prompt-level solution leakage. Building on the time-consistent benchmark protocol of Sun (2026), we enforce strict zero-leakage isolation across DragonFly in C++ and React in JavaScript/TypeScript, locking repository snapshots and knowledge graphs at historical cutoff commits $T_0$ and evaluating 100 post-$T_0$ pull requests across four prompt granularities: Minimal, Concise, Contextual, and Guided. Under this zero-leakage protocol, we conduct a matched evaluation of three context-augmentation regimes: (1) an unaugmented Baseline, (2) a flat syntax-level Tree-sitter AST graph, and (3) ServiceExplorer, a deterministic two-level hierarchical dependency graph capturing Level-1 system/package containers and Level-2 architectural modules. Across 7,200 paired evaluations spanning six foundation models across the MAI-Code Flash, Claude, and GPT-Sol families, ServiceExplorer consistently outperforms both Baseline and Tree-sitter AST across every model, repository, and prompt granularity ($+15.6\%$ to $+86.2\%$ relative $F_1$ lift, $p < 10^{-4}$), while tripling exact full-PR localization on DragonFly. Crucially, equipping an efficiency-tier or prior-generation model with ServiceExplorer consistently surpasses the unaugmented Baseline $F_1$ of the next-generation model across all evaluated families.
Source:https://arxiv.org/abs/2611.17388PreviousProdPilot: A Layered Model Context Protocol System for Automated Production Readiness and Deployment of Vibe-Coded ProjectsNextInterpreting Coding Agents Through Their Behavioral Trajectories
Related posts
OOPs in Java
0dev.to Java
Taking over a codebase you don't know: where to start
0dev.to Git
Automating Price Validation Across 633 Static Pages Before Every Deploy
0dev.to TypeScript
How We Use Flight Data APIs to Track Every Airport Pickup
0dev.to TypeScript
How to export a Framer site to TanStack Start
0dev.to TypeScript
LUNAR BROWSER :
0dev.to TypeScript