AI & ML
SeVerus: Context-Aware, Gamified Benchmarking for Rust Verification
arXiv cs.SE0 views
arXiv:2611.07194v1 Announce Type: new Abstract: Coding agents achieve high success rates on existing benchmarks for Verus, a verifier for Rust programs. These benchmarks typically provide the auxiliary lemmas needed to complete a proof. In practice, however, such lemmas must be discovered and proved. To capture this challenge, we present SEVERUS, a context-aware technique to generate Verus benchmarks that organizes verification tasks into quests, each divided into levelsin a gamified manner. Using the original proof dependencies in verified projects, SEVERUS constructs one quest per verification target and progressively removes proof context to create individual levels. These levels range from the easiest, requiring completing only a single missing lemma, to the hardest, requiring proving the target without any supporting lemmas. An independent checker rejects changes to executable code, weakened specifications, and unsound proof constructs. A noisy binary search estimates the highest level that an agent can solve reliably. Applied to six open-source Verus projects, SEVERUS generates 819 quests comprising 10,095 levels. Experiments with Codex using GPT-6 Sol and GPT-6 Luna, and with Claude Code using Sonnet 4.6, show that success decreases and cost increases as proof context is withdrawn. The strongest agent solves the final level of 52 of 60 sampled quests, yet can only reliably solve 35% of their levels, with failures concentrated in longer quests. Successful GPT-6 Sol proofs rarely reproduce the withheld lemmas, instead, using alternative proof decompositions. By systematically varying proof context, SEVERUS provides a foundation for constructing verification benchmarks that can continue to challenge increasingly capable coding agents.
Source:https://arxiv.org/abs/2611.07194PreviousMulti-step SWE: Can Coding Agents Keep Building on Their Own Code Over Long Horizons?NextHow Should Agents See Code? Rethinking Code Representations for Agentic Software Engineering
Related posts
Snakes and Ladders as a Markov chain: 39.23 throws, computed and simulated
0dev.to WebDev
The 5 Best Next.js RAG and Chat-With-Your-Documents Templates in 2026
0dev.to WebDev
Booking Systems That Fill Calendars: How Codeair Software Solutions Builds Appointment Software for Service Businesses
0dev.to WebDev
Sidero Omni shows why zero-result scans need an evidence record
0dev.to Security
I masked a dataset for an LLM and kept every checksum valid — here's the approach
0dev.to Python
Designing Circuit Breakers for Autonomous Agent LLM Token Burn
0dev.to DevOps