AI & ML
Programmable Distributions of Semantics-Preserving Variants for Evaluating Code Agents
arXiv cs.SE0 views
arXiv:2608.18389v2 Announce Type: replace-cross Abstract: Code agents are increasingly deployed to resolve real software issues, yet their reliability under superficial code variations remains poorly understood. We evaluate whether coding agents that repair repository-level issues remain reliable when the surrounding codebase is rewritten into a semantically equivalent form. We sample semantically equivalent variants of a repository from an explicitly defined distribution and compare the agent's resolve rate on them with that on the original. Different distributions reveal different aspects of an agent, such as its sensitivity to routine refactoring or to misleading code near the location of a fix. We develop VDL, a probabilistic programming language for expressing such distributions, whose programs randomly apply semantics-preserving transformations (SPTs) to a repository. For our evaluation, we use it to define a generic distribution that is not targeted at any particular failure mode. We evaluate two agentic scaffolds (mini-SWE agent and OpenCode), each backed by one of four frontier models (Claude Opus 4.5, Kimi K2.5, MiniMax M2.5, and Qwen 3.6-27B), on instances from SWE-bench Verified and SWE-bench Pro. Resolve rates drop by up to 6.7 percentage points, significantly in 6 of 16 configurations. No ranking of models by robustness holds across scaffolds, revealing a jagged robustness frontier.
Source:https://arxiv.org/abs/2608.18389PreviousThe $\alpha$-Limit Problem: Convergence of a Linear Degenerate Interface Transmission ProblemNextCross-Benchmark Transfer from RL on Agentic Coding Tasks
Related posts
Software jobs are back, but not for juniors. What I'd build if I were starting in 2026.
0dev.to Discuss
How to Calculate Real Throughput in an Automated Filling Line
0dev.to Productivity
WeightBench Tests Language Models as Weight-Space Programmers
0dev.to Machine Learning
Data Cleaning for messy dates dd-mm-yyyy
0dev.to Machine Learning

How Much GPU Memory Do I Need?
0dev.to Machine Learning
The Hugging Face Ecosystem in 2026: Transformers, Datasets, Trainer, LoRA/PEFT, Gradio & Spaces (Real Code)
0dev.to Machine Learning