AI & ML
SeVerus: Context-Aware, Gamified Benchmarking for Rust Verification
arXiv cs.SE0 views
arXiv:2611.07194v1 Announce Type: new Abstract: Coding agents achieve high success rates on existing benchmarks for Verus, a verifier for Rust programs. These benchmarks typically provide the auxiliary lemmas needed to complete a proof. In practice, however, such lemmas must be discovered and proved. To capture this challenge, we present SEVERUS, a context-aware technique to generate Verus benchmarks that organizes verification tasks into quests, each divided into levelsin a gamified manner. Using the original proof dependencies in verified projects, SEVERUS constructs one quest per verification target and progressively removes proof context to create individual levels. These levels range from the easiest, requiring completing only a single missing lemma, to the hardest, requiring proving the target without any supporting lemmas. An independent checker rejects changes to executable code, weakened specifications, and unsound proof constructs. A noisy binary search estimates the highest level that an agent can solve reliably. Applied to six open-source Verus projects, SEVERUS generates 819 quests comprising 10,095 levels. Experiments with Codex using GPT-6 Sol and GPT-6 Luna, and with Claude Code using Sonnet 4.6, show that success decreases and cost increases as proof context is withdrawn. The strongest agent solves the final level of 52 of 60 sampled quests, yet can only reliably solve 35% of their levels, with failures concentrated in longer quests. Successful GPT-6 Sol proofs rarely reproduce the withheld lemmas, instead, using alternative proof decompositions. By systematically varying proof context, SEVERUS provides a foundation for constructing verification benchmarks that can continue to challenge increasingly capable coding agents.
Source:https://arxiv.org/abs/2611.07194PreviousMulti-step SWE: Can Coding Agents Keep Building on Their Own Code Over Long Horizons?NextHow Should Agents See Code? Rethinking Code Representations for Agentic Software Engineering
Related posts
Show HN: Grid – A Hold'em arena where your AI agent plays your strategy (Beta)
0Hacker News Show
Show HN: Mercury 3 cost calculator (launch promo vs. list pricing)
0Hacker News Show

'Gen Z uses AI the way their grandparents smoked': Students say they feel pressured to use AI, and many would ban it if they could
0TechRadar

Apple Store now offers extended holiday returns for all products
09to5Mac
I bought the knock-off Pixel Watch and Fitbit Air bands, and it’s crazy how good they are [Gallery]
09to5Google

Amazon will let you text Alexa your shopping list
0The Verge