Originally published at Trida AI.
The dangerous moment in an agent run is not the first failed tool call. It is the fortieth.
A person whose lookup tool times out twice tries something else, or goes and makes tea. An agent does neither. It replans, and the new plan starts with the same lookup. It rewords the arguments and calls again, because from where it sits, each attempt looks like a fresh idea. The retries built into the tool layer add their own attempts underneath. None of these callers is misbehaving. Every one of them is doing exactly what it was told to do, which is to get the information and finish the task. Together they aim a steady stream of calls at a service that has already said, several times, that it cannot answer.
That stream is not harmless. Google's SRE book defines a cascading failure as a failure that grows through positive feedback: part of a system fails, the load shifts onto what remains, and the remainder becomes more likely to fail in turn. A failing tool under agent traffic fits the pattern neatly. The calls hold threads, connections and time. The service spends its shrinking capacity telling callers no, slowly. And the run itself burns its budget on attempts that were settled before they were sent.
Retries answer a different problem. A retry assumes the failure might be momentary, and sometimes it is. What an agent deployment needs alongside retries is a way for the route to a tool to remember that the last several calls failed, and to stop sending new ones for a while. That memory has a name in distributed systems, and it is older than agents by decades.
A breaker gives the route a memory
The Circuit Breaker pattern in Microsoft's Azure Architecture Center puts a proxy in front of an operation that might fail. The proxy counts recent failures. While the count stays below a threshold set for a time window, calls pass through — the closed state. Cross the threshold and the proxy opens: calls fail immediately, without being sent, and a timer starts. When the timer expires, the proxy turns half-open and lets a limited number of calls through as probes. If they succeed, the breaker closes and the counter resets. If one fails, it opens again and the wait restarts.
Two details in that design do most of the work for agents. The first is that an open breaker answers at once. A call that would have taken eight seconds to time out now takes milliseconds to refuse, and the refusal itself is information the agent can act on. The second is the half-open probe. Recovery is tested by a trickle, not by the full weight of every waiting run arriving in the same second, which can knock a recovering service straight back down.
Put one breaker on the path to each tool, inside the gateway or runtime the calls already pass through — never inside the model's discretion. The model should not decide whether a tool is healthy any more than it decides whether its own retries happened. Health is a counted fact about recent calls, and counting belongs in code.
One breaker per tool, on the path every call takes. Each breaker counts only its own tool's failures, so the open records lookup leaves search and email working. The agent reads the state; it does not set it.
The open state is a tool result
Here is where an agent deployment has to go further than the classic pattern. A conventional caller receives an exception and its code decides what happens next. An agent receives whatever the tool layer hands it, and reasons about it in language. So the open response has to be written for that reader.
It should say which tool is unavailable, when it is worth trying again, and what to do in the meantime. Park this step and continue with the parts of the task that do not need it. Work from the records already fetched. Or stop and report, if the missing call is the task. Given that, a competent agent does something sensible with the refusal. Given a bare timeout error repeated ten times, it will mostly produce an eleventh attempt in different words.
The retry layer has to respect the same signal. The Azure guidance makes the point directly: a retry policy invoking an operation through a breaker should stop when the breaker indicates the fault is not a passing one. If retries keep firing underneath an open breaker, the breaker is decoration.
Keying matters just as much. Give each tool its own breaker, and split further where a tool fronts genuinely separate resources — by destination, by shard, by customer environment. The Azure guidance warns against merging independent providers behind one breaker: one sick shard can block healthy ones, and the failures you do count stop describing anything real. Count the failures that implicate the tool, too. A call rejected because the agent supplied a malformed argument is the agent's fault, and letting it trip the breaker teaches the system that a bad prompt is an outage.
One outage, as the run recorded it. Example log, with example settings (a threshold of three failures in a minute, a ninety-second wait). Across the whole outage the service received six calls. The step that needed it was queued, not repeated, and the run spent the wait on work that was still possible.
Where a breaker stops helping
A breaker is a blunt instrument, and the bluntness cuts the wrong way often enough that the limits deserve stating plainly. The Amazon Builders' Library is candid about the trade. Circuit breakers are widely promoted for this problem, it says, but they introduce modal behaviour into systems: behaviour that can be difficult to test, and that can add significant time to recovery. Its own mitigation for runaway retries is plainer. Limit retries locally with a token bucket: calls retry while tokens remain, and when the bucket empties they retry only at a fixed rate. Amazon added that behaviour to the AWS SDK in 2016. For a tool that fails partially, answering most calls and dropping some, that gentler shape may serve better than a switch.
Breakers shared across many agent instances have their own trap. One fleet-wide breaker opening is at least honest information, but if every instance probes in the same instant when the wait expires, the probe becomes a small flood. Stagger the probes. And a breaker protects a tool, not a task: it cannot tell you whether the answer that finally comes back is right, only that the route to it is answering again. If the agent has no useful work to do while a breaker is open and no honest way to report the gap, all you have bought is a faster failure. The refusal, the alternative work, and the report are part of the design, not polish added later.
None of this is exotic. That is rather the point. Agents inherited the failure modes of distributed systems along with the tools, and the countermeasures transfer — as long as someone remembers that the caller, this time, will not get discouraged on its own. Persistence is the quality you bought. The breaker is how you aim it.
Work with us
If your agents keep calling tools that have already failed, the fix is rarely a cleverer prompt. It is a breaker on each tool's route, an open response the agent can reason about, and work it can do while it waits. Tell us where your runs stall through Trida AI's intake.
