AI & ML
Reducing Inference Cost of Coding Agents with Timeout Cascades
arXiv cs.SE0 views
arXiv:2611.24324v1 Announce Type: cross Abstract: AI coding agents have become one of the most popular applications of large language models. Yet, agentic AI comes at the cost of a tremendous amount of GPU compute resources, which limits its profitability and its viability at scale. In this paper, we investigate whether smaller, cheaper models can cut this cost without sacrificing accuracy. We evaluate 36 popular large and small open-weight models released since 2024 on a software engineering benchmark, and find that recent small models are closing the accuracy gap compared to their large counterparts. We investigate the structure of this gap and find that the gap is highly concentrated around hard tasks that small language models fail to solve while achieving remarkably good accuracy for easy and medium complexity tasks. Following this observation, we show that by utilizing a hybrid inference strategy that tries a small model first and escalates to a large one only when needed, we can achieve up to 33% reduction in total GPU-hour costs and 16.4% on all tasks from SWE-bench Verified, without sacrificing accuracy. Strikingly, this simple cascade nearly matches an oracle-style task router, highlighting a practical path for deploying cost-aware coding agents in production without complex learned task routers and with minimal engineering complexity.
Source:https://arxiv.org/abs/2611.24324PreviousInference-Time Scaling in Agentic Hardware Verification Workflows: A Multi-Strategy Characterization Across Protocol ComplexityNextPredicting Process-Level Behavior: A Benchmarking Study
Related posts

How to Interpret Logistic Regression and Odds Ratios
0dev.to
Kubernetes Service Account Tokens Do Not Need to Ride Along
0dev.to Kubernetes
Export a calendar day to ICS for your system calendar
0dev.to OpenSource
Comparing Axleta Repository Packer with Gitingest
0dev.to Programming

Animating and rigging by hand is sometimes hard....
0dev.to Programming
Anthropic is bullshitting you about subagents and loops.
0dev.to Programming