BudgetMix RL: What Multi-Turn Reinforcement Learning Really Improves in LLM Agents
Teams training LLM agents with multi-turn reinforcement learning usually report one number: success rate on a long-horizon benchmark. A new paper, What Does Multi-Turn Reinforcement Learning Really Improve in LLM Agents?, asks what sits behind that number. Its answer matters for anyone shipping agents: the gains depend heavily on how many turns the agent is allowed at inference time, and they come from a specific skill, recovering from mistakes. The authors also propose BudgetMix RL, a training change meant to make agents robust when the turn budget shrinks.
Why a single success rate hides the story
Agentic RL trains a model on whole interaction trajectories: the model reasons, calls a tool, reads the result, and repeats until it finishes or runs out of turns. The reward arrives at the end, and the policy gradient pushes up every action in successful trajectories.
That setup makes the final score a function of several things at once: how good each individual decision is, how well the agent corrects itself, and how many attempts and turns the evaluation grants. A benchmark that fixes one turn limit and one sample count collapses all of these into one figure. Two training recipes can tie on that figure while behaving very differently once you change the limits in production, where latency and cost often force tighter caps than the ones used in research.
Measuring with Pass@(K,T)
The authors evaluate along two axes. K is sampling breadth, the number of independent attempts per task. T is interaction depth, the maximum number of turns per attempt. Their metric, Pass@(K,T), counts a task as solved if any of K attempts succeeds within T turns.
This extends the familiar Pass@K by making the turn budget explicit. Plotting results as a surface over (K, T) lets you see where an RL-trained model beats its base model and where it does not. It is also easy to reproduce: if your agent harness already logs turns per trajectory, you can compute the metric from existing evaluation runs by truncating trajectories at different values of T and resampling attempts for different values of K.
What the analysis found
Two findings stand out.
First, the advantage of trajectory-level RL depends on having enough turns. With a generous budget, the RL-trained agent clearly outperforms the baseline. As the budget tightens, that advantage shrinks significantly. The training signal rewarded behavior that pays off over long interactions, and cutting interactions short removes the payoff.
Second, agentic RL does not uniformly improve single-step decision quality. If you look at one decision in isolation, the trained model is not reliably better than the base model. Its strength concentrates in error recovery and long-horizon resilience: when a tool call fails or a step goes wrong, the trained agent is better at noticing and trying something else. This reframes what the training buys. It is less about a sharper first move and more about staying on course across many moves.
For practitioners the consequence is practical. If you evaluate an RL-trained agent only under the turn cap used in training, you can overestimate its value for deployments with stricter caps. And if your tasks are short, with few opportunities to fail and recover, multi-turn RL may deliver less than the headline results suggest.
Process supervision and BudgetMix RL
A natural fix for budget sensitivity is denser feedback. Process supervision scores intermediate steps instead of only the final outcome, so credit reaches good decisions even when the trajectory is cut short. The paper gives this a preliminary discussion of benefits and limitations rather than a full treatment. Step-level signals can help, but they require a way to judge intermediate steps, and a reward model for that is another component that can be wrong or be exploited. The authors treat it as one option, not a solution, and move to a cheaper intervention.
BudgetMix RL changes one thing in training: instead of fixing the interaction budget, it varies the budget across rollouts. Some trajectories run with a tight turn limit and others with a loose one, so the policy is optimized for a mixture of execution settings rather than a single one.
The idea is simple to implement. In a typical rollout loop, the maximum turn count is a constant passed to the environment. Under BudgetMix RL you sample it per rollout from a set of budgets. The reward and optimizer stay the same. The agent learns that it cannot always count on a long runway, which pushes it to make progress earlier while keeping its recovery behavior for the cases where turns are plentiful.
According to the paper's experiments, BudgetMix RL achieves robust performance across diverse execution settings, meaning its results hold up across the (K, T) grid better than training with a fixed budget. The authors present it as an alternative that addresses the deployment risk created by budget dependence. Treat the specific numbers as something to verify in the released code before adopting the method, since agent environments differ in how costly an extra turn is.
Three habits follow from the paper.
Report results across several turn budgets and attempt counts rather than a single point. Check whether your agent's gains survive at the turn cap you will run in production. And if they do not, try randomizing the budget during training before reaching for more complex reward machinery.
The code is available at Rethinking-Agent-RL, and the project page has the paper's supporting material.
The broader lesson is about evaluation. Agent benchmarks bundle decision quality, recovery, and budget into one score, and the choices made in training decide which of these improves. Separating them tells you what your RL run actually bought.
