I crash-tested AI agents that issue refunds. After a crash and resume, 22 of 28 refunded twice.

I've been building agents that refund customers and then email them about it. Testing the happy path is easy. I wanted to know what happens when the network gets in the way at the worst moment: right after the refund has gone through, but before anyone hears back about it.

So I built a harness to find out and ran it across five frameworks. Most of it went roughly how I'd guessed. The OpenAI Agents result didn't.

The setup

The harness runs the agent's real loop (the framework's own tool calling, retries and resume path) against stateful fakes for Stripe and email. It injects a failure at the point where the side effect happens. Then it checks the end state against invariants like "exactly one £50 refund" and "one email to the customer".

The model was Claude Haiku 5.5. The frameworks were Vercel AI SDK, OpenAI Agents SDK, LangGraph.js, Mastra and the MCP SDK. 28 runs per fault.

The tools were deliberately naive, with no idempotency at all. I wanted to see what the model and the framework do on their own before any protection is added, because that's what a lot of first versions look like.

A failing run reports like this:

shakedown: refund agent (naive tools) FAIL [pass] baseline (no fault) (2 writes committed) [pass] error-before-commit on refund (2 writes committed) [FAIL] timeout-after-commit on refund (3 writes committed) - expected exactly 5000 refunded on ch_1, found 10000 [FAIL] duplicate-delivery on refund (3 writes committed) - expected exactly 5000 refunded on ch_1, found 10000 [FAIL] crash-after-commit on refund (3 writes committed) - expected exactly 5000 refunded on ch_1, found 10000 [pass] error-before-commit on send_email (2 writes committed) [FAIL] timeout-after-commit on send_email (3 writes committed) - expected exactly 1 email(s) to ada@example.com, found 2 ... 

What happened

1. Crash and resume is the big one

When the process died right after the refund committed and the job was picked back up, the refund went through a second time in 22 of 28 runs. The resumed job had no idea the first refund happened. Whatever would have told it died with the process.

2. Email had nothing to check

After an email send timed out (but had actually gone out), the model sent it again in 25 of 28 runs. I don't really blame the model. With a refund it can look up the charge and see it's already refunded. With an email there was nothing to look up, so trying again is a reasonable guess that happens to be wrong.

3. "Please try again."

I didn't expect this one. When a refund timed out after it had gone through and the model saw the real error, it looked the charge up first, saw the refund and moved on. 0 duplicates in 20 runs.

With the OpenAI Agents SDK it was 8 of 8. By default that SDK replaces a thrown tool error with its own text, which ends with "Please try again." So the model did. In the same SDK, with a custom errorFunction passing the real error through, it went from 6 of 6 duplicates to 0 of 6.

It depends on the model though. On Sonnet 5.5 it checked first either way (0 of 4). I've filed it as openai/openai-agents-js#2184. The other issues I've opened so far are vercel/ai#24624 and mastra-ai/mastra#29582.

4. Duplicate delivery

28 of 28. That's expected for a tool with no idempotency, so treat it as a control more than a finding.

What fixed it

The obvious fix is an idempotency ledger: record each operation, skip it if it's already there. On its own it fails the same faults. The ledger entry is written after the call returns, so when the response is lost (timeout, crash) nothing gets recorded and the retry goes through again.

What worked was the ledger plus the same key sent downstream, so the payment or email provider can dedupe on its side too. Stripe has Idempotency-Key, and email APIs have their own. With both, the Haiku runs had no duplicates in any framework.

In Shakedown it looks like this:

import { idempotent, shake, currentIdempotencyKey } from 'agent-shakedown' const refund = idempotent( shake('refund', async ({ charge, amount, ticket }) => stripe.refunds.create( { charge, amount, metadata: { ticket } }, { idempotencyKey: currentIdempotencyKey() }, // same key, sent downstream )), { key: ({ ticket }) => `refund:${ticket}`, store: postgresLedger }, ) 

The key is the ticket id, which is the logical operation, not a hash of whatever arguments came in.

What I got wrong

I'm making a point of that because I got it wrong first.

My own examples keyed emails on the subject line, and the model writes the subject line. On a retry Sonnet most likely reworded it. The retry got a new key, the provider saw a new email, and the customer got a second one.

If I had to keep one lesson from all of this: never build idempotency keys from text the model writes. Use something your system owns, like a ticket or order id.

Duplicates vs jobs that stopped short

The report keeps duplicates (safety) separate from jobs that stopped short (completion). A refund with no email is a different bug from two refunds, and you probably want to page someone for one of them and not the other.

It also marks a run UNTESTED instead of passing it when the resume never actually called the tools. A pass like that tells you nothing about idempotency, so it shouldn't show up green.

Limits

It's Claude only: Haiku 5.5 for the main numbers, Sonnet 5.5 for the comparison. 28 runs per fault isn't a huge sample. And the tools are naive on purpose, so these numbers show what happens with no protection at all, not how often your agent will do it.

Try it

Shakedown is free and MIT licensed. It's TypeScript. You wrap the place where the real side effect happens (the Stripe call, the mailer call) with shake(), and outside a test it's a pass-through.

npm install -D agent-shakedown 

Repo with the framework examples: https://github.com/zennicalles/shakedown

It's just me building it, so if you run it against your own agent and something breaks or reads wrong, an issue would help a lot. I'm curious how people with agents in production key their refunds and emails today. I suspect model-written text ends up in more keys than anyone would admit.