Part 2 of **Shipping AI Agents to Production. Part 1 was about the second check: the eval gate judges the version, the smoke eval judges the environment.

I promised bundles, canaries and rollback for this part. Then something happened that no canary could have caught, so those move to Part 3.

A refund call timed out on a Friday evening. No deploy, no prompt change. The customer asked, "Did my refund go through?" So our support agent called the refund tool again. 9 customers were refunded twice in 14 minutes.

We already had an idempotency key on every refund. We built it from the tool call ID. The second call had a new ID, so it got a new key, and the payment provider saw a brand-new request.

The fix: one key per business intent, never per attempt. And for a tool with side effects, an error means "unknown", not "failed". A reader put it better than I had: a retry is only safe when the second attempt can be recognised as the same attempt.

This post covers the key, the tool wrapper and what the model should see.

Why does an agent retry more than a normal service?

In Part 1, one prompt line ("If a tool call fails, retry once") refunded 212 customers twice. We removed the line, added a smoke eval and put an idempotency key on side-effect tools. Three weeks later it happened again, with only a slow payments API.

The wrong model was "one layer retries". In a normal service, that's the HTTP client. In an agent, at least four layers can retry the same action, and none of them know about the others:

  • The model. A prompt line, or its own judgement, calls the tool again after an error.
  • The agent framework. A LangGraph RetryPolicy defaults to 3 attempts.
  • The HTTP client. Many SDKs retry timeouts and 5xx on their own.
  • The user. "Did it go through?" is a retry. So is pressing the button again.

Four stacked layers that can retry the same tool call: the model (2 attempts), the agent framework (3), the HTTP client (3) and the user ("did it go through?"). 2 × 3 × 3 = up to 18 calls for one refund, before the user asks again.
Each layer is reasonable on its own. Stacked, one refund request can become up to 18 calls.

You won't remove every retry, and you shouldn't. The goal is that every retry, from any layer, lands on the same answer.

What does an error from a refund tool actually mean?

Leslie Lamport described a distributed system as "one in which the failure of a computer you didn't even know existed can render your own computer unusable." An agent with tools is one. The payments API fails in five specific shapes:

Table of five failure shapes, showing what the agent sees and what actually happened. Slow success: timeout, but done. Lost response: network error, but done. Error after commit: 500, but maybe done. Partial failure: error, but half done. Duplicate delivery: success twice, done twice.
In four of the five shapes, "error" doesn't mean "didn't happen".

So the rule:

For a tool with side effects, an error means "unknown", not "failed".

Not every tool needs this. Put each tool in one group and store it as a field the wrapper reads, so nobody has to remember which tools are dangerous:

effect Examples Retry policy
read get_order, check_refund_status Retry freely
idempotent set_ticket_status, update_address Retry with the same arguments
side_effect refund_order, charge_card, send_email Never retry blindly

In our support agent that was 9 read-only, 2 idempotent and 3 side-effecting tools out of 14. Everything below applies to those 3.

What should an idempotency key identify?

Stripe's docs say that with an Idempotency-Key header "you can safely repeat the request without risk of creating a second object or performing the update twice." Three details matter for agents:

  • Stripe saves the first response for a key and replays it for repeats, including 500 errors.
  • Reusing a key with different parameters returns an error. That catches an agent retrying with a different amount.
  • Keys can be removed once they're at least 24 hours old. After that, the same key can create a new request.

Our key was fine. What it identified was wrong.

Two flows side by side. Wrong: the key comes from the tool call ID, so two calls get keys call_7f2 and call_9a1 and the provider creates two refunds. Right: both calls use the key refund:order_8812:49.00, and the provider returns the same single refund.
A key from the tool call ID changes on every retry. A key from the business intent doesn't.

One key must mean one business intent: "refund $49.00 for order 8812". Not one attempt, one tool call or one turn. Derive it from the intent (refund:order_8812:49.00) or generate a UUID once and store it with the intent. Either works, as long as it's created once per intent.

What does a retry-safe tool wrapper look like?

Every side-effecting call goes through one wrapper that owns keys, timeouts and retries:

Flowchart of the wrapper. Tool call, then look up intent: if already done, return already_done. Otherwise record the intent as pending with its key, lock it, call the provider with the key. On success, mark done. On timeout or error, check the provider for the refund: found means done, unclear means pending_unknown, handed to reconciliation.
Record the intent, call with its key, and check before acting again.

Here it is as Python-style pseudocode. payments and intents stand in for your payment client and database table:

def refund_order(order_id: str, amount: Decimal) -> dict: key = f"refund:{order_id}:{amount}" # one key per intent, not per call intent = intents.get_or_create(key, status="pending") if intent.status == "done": return already_done(intent) with intents.lock(key): # a concurrent call waits here intent = intents.get(key) if intent.status == "done": return already_done(intent) try: refund = payments.refund(order_id, amount, idempotency_key=key, timeout=10) intents.mark_done(key, refund.id) return {"status": "done", "refund_id": refund.id} except (Timeout, ConnectionError, ServerError): found = payments.find_refund(order_id, amount) # check, don't retry if found: intents.mark_done(key, found.id) return {"status": "done", "refund_id": found.id} intents.mark(key, "pending_unknown") return {"status": "pending_unknown", "message": "Refund may have gone through. Do not call " "refund_order again for this order."}

The line that matters is the first one. Everything else depends on key being the same on every attempt.

Two more details:

  • Set the timeout from the real API's p99, not the mock's. In Part 1, the mock answered in 50 ms, the timeout was 3 seconds, and the real API's p95 was 4.8 seconds.
  • Bounded retries with the same key are fine for errors that clearly happened before anything was processed, like a rate limit. A new key for the same intent is never fine.

What should the model see when a refund is unclear?

When a tool returns Error: timeout, the model has to guess, and a helpful model guesses "try again". So side-effecting tools return one of three statuses, each with an instruction:

{"status": "done", "refund_id": "re_123"} {"status": "already_done", "refund_id": "re_123", "message": "Completed earlier. Tell the customer it went through."} {"status": "pending_unknown", "message": "Refund may have gone through. Do not retry. Tell the customer it is being confirmed."}

Then change two pieces of text:

  • The prompt. "If a tool call fails, retry once" becomes "Never call refund_order more than once for the same order. If it returns pending_unknown, tell the customer the refund is being confirmed."
  • The tool description. "Safe to call once per order. Repeated calls return the original result." Models read tool descriptions as instructions.

The wrapper makes retries safe. The statuses make them unnecessary.

When does this rule break?

Seniors will ask this, so here are the gaps:

  • Keys expire. After 24 hours, a key alone won't stop a late retry. The intent lookup in step 1 is what protects that case.
  • An intent-derived key blocks a legitimate repeat. If a customer can really get two $49.00 partial refunds on one order, refund:order_8812:49.00 will treat the second as a duplicate. Use a stored UUID per intent instead.
  • The provider can't always answer. If find_refund also times out, you're in pending_unknown. Something has to resolve it later.
  • Smoke evals only run when you deploy. Part 1's smoke eval checks that a side-effect tool runs at most once per conversation, but this incident had no deploy. Between releases, protection lives in the tool layer.

That "something" is an hourly reconciliation job that compares your intent table with the provider's records:

Reconciliation flow: your intent table and the provider's refunds are compared every hour and sorted into three buckets. Matched: nothing to do. Duplicate: page on-call and open a finance ticket. Orphan or pending_unknown over 30 minutes: resolve by hand. Below: second incident timeline, alert at 14 minutes, tool off at 16, all 9 duplicates confirmed within the hour.
Compare your intents with the provider's records every hour, and route each mismatch to an owner.

After the first incident, we added an alert on duplicate tool calls. This time it fired at 14 minutes, a feature flag switched off refund_order at 16, and reconciliation confirmed all 9 duplicates within the hour. The first time, 212 customers were hit and we found out from a customer ticket. The incident still happened, but the system caught it and told us exactly who was affected.

What should you check in the next 20 minutes?

  1. List every tool and mark it read, idempotent or side_effect.
  2. For each side-effecting tool, find where the idempotency key comes from. If it contains a tool call ID, a request ID or a timestamp, that's your bug.
  3. Search your prompts for "retry". Delete any line that applies to a side-effecting tool.
  4. Add one smoke case: request a refund, then ask "did it go through?", and assert exactly one refund exists. This single case would have caught our second incident.
  5. Check you have a flag that disables one tool without stopping the agent.

Part 1's rule was about checks: the eval gate judges the version, the smoke eval judges the environment. This part's rule is about tools: a retry is only safe when the second attempt is recognised as the same attempt. Neither check runs between deploys. The tool layer does.

Next in this series, the promise from Part 1: why an agent version is five things, not a code diff, and how a 5% canary would have cut 212 double refunds to about a dozen.

Our AI agent passed its eval. Then it refunded 212 customers twice
rakesh_kumar_04012c337851 profile

Our AI agent passed its eval. Then it refunded 212 customers twice

#discuss #ai #python #architecture
2
5 min read

Which of your side-effecting tools builds its idempotency key from something that changes per attempt? Tool call ID, request ID, timestamp? I'm most curious how many teams key on the tool call ID. Tell me in the comments.