Originally published at Trida AI.
A failed agent run usually ends in the worst place it can: a log line. The run retried, spent its budget, gave up, and the only evidence is a stack trace in a stream that scrolls past thousands of other lines. Nobody owns it. A week later the same failure has happened forty more times, and each one was news.
Messaging systems settled this problem years ago. When a message can't be processed, it doesn't vanish and it doesn't block the queue behind it. It goes to a dead letter queue, a holding place built for inspection. Amazon SQS describes dead-letter queues as the place to isolate messages that failed processing so you can work out why. Google Cloud Pub/Sub forwards a message to a dead-letter topic after roughly a configured number of delivery attempts (five by default), wrapping the original with attributes that identify where it came from. Azure Service Bus adds a reason and a description to every dead-lettered message, and is blunt about the other half of the contract: there is no automatic cleanup, so messages sit there until someone retrieves them.
Agent runs need the same destination, and most don't have one. The shape of the problem is familiar (retries exhausted, failure made permanent, work set aside for a human), but the unit is different. A message is a payload. A run is a piece of work that may have read three systems, written to one of them, and stopped halfway through a plan. Parking that safely takes more than a queue. It takes a record.
A log is not a queue
Logs answer "what happened?" for someone who already knows where to look. A dead letter queue answers a different question: "what work is still owed, and who owes the next move?" That difference is operational, not cosmetic. A queue has an owner, a count, and an age for its oldest entry. It can alarm when it grows. Items leave it one of two ways, replayed or resolved by a person, and both are deliberate acts.
So the first design decision is boring and important: terminal failure is a state, not an event. When a run exhausts its retries or its budget, the runner doesn't just report an error and exit. It writes a dead letter record and stops. Everything after that (inspection, the fix, the replay, the handoff) works from that record.
Where a failed run goes. The runner parks a terminal failure as a record; a person either clears it for replay from its checkpoint or takes it over with the record attached.
What the record has to hold
A dead letter record for a message can be thin: the payload, a reason code, a delivery count. A run's record has to carry enough for a stranger to make a safe decision about work they didn't watch happen.
The original task, first: the instruction and inputs the run started from, snapshotted rather than referenced, because the source may have changed since. Then the failure, described in a small fixed vocabulary (tool timeout, invalid input, permission denied, output rejected by a checker), because free-text errors can't be counted, and uncounted failures can't be fixed in batches. The attempt history belongs beside it: how many tries, at what, and where the run stopped, with a pointer to its checkpoint and its trace.
The field people skip is the side-effect register: what the run already changed in the world before it stopped. Which writes landed, under which idempotency keys, and, just as explicitly, which ones are unknown. That register is what separates a replayable failure from a dangerous one, and it's cheap to write at the moment of failure and nearly impossible to reconstruct a day later.
Two smaller fields earn their space. An owner, because an unowned queue is a landfill with good tooling. And an expiry, because a failure nobody has looked at in thirty days is rarely worth replaying. The world it belonged to has moved on.
One dead letter record. The replay decision is written down with the reason, and the unknown side effect — not the timeout — is what sends this one to a person.
Replay is the dangerous part
Getting a failed message back is a solved, almost casual operation: SQS calls it redrive, and the message returns to its source queue. Replay an agent run the same casual way and you can do real damage. The run may have already created the draft, sent the first email, or placed the hold. Starting it again from the top repeats whatever landed.
So replay has a gate, and the gate reads the side-effect register. If every landed effect is recorded and the replay resumes from the checkpoint reusing the original idempotency keys, it can go. If any effect is marked unknown (a tool call that timed out after it may have committed), the record doesn't replay at all. It goes to a person, with the unknown named in plain language, and the person settles that one question before anything resumes. One ambiguous write should cost a human two minutes, not a duplicate payment.
The queue also pays rent in aggregate. Read as a pile, dead letter records are a bug report against your own system: the same failure class forty times is a broken tool contract or a bad instruction, not forty accidents. The fix lands in the tool or the prompt, the waiting records replay against the fixed system, and the class empties. A queue that only ever gets replayed, never read, is a retry loop with extra storage.
Where this stops helping: small deployments. If your agents fail terminally twice a week, a table and a named owner give you most of this; the full apparatus is for fleets where failures arrive faster than anyone can read logs. And the record itself needs care — task snapshots can hold personal or commercial data, so redact on the way in and keep retention short. A dead letter queue that becomes a second, unguarded copy of your inputs has traded one problem for a worse one.
Work with us
If your agents' failures currently end as log lines, retried, dropped, and rediscovered weeks later, we can help you give them a destination: records a person can decide from, and replays that are safe to run. Tell us about the deployment through Trida AI's intake.
