Yesterday I fixed my webhook duplicates with one claimable row per delivery key. A slow receiver took 2.5 seconds to answer, my sweeper ticked every second, and the same payment went out three times before the first acknowledgement landed. One row in a table, claimed atomically, and three deliveries became one. I wrote it up feeling pleased with myself. Then rudratosh left a comment that undid the whole feeling: the claim row stops the resend, but the agent still believes the step failed. Fix it quietly and the agent plans its next move from a world where no key was ever assigned, when a key exists.

He was right, and I ran it twice to find out how right. Same setup as yesterday, same slow receiver, same claim row, same key, evt-1042. The only difference between the two runs was what the planner was allowed to know.

Here is the shape of the problem. The planner watching the delivery has the patience of one second. One second after the send starts it checks for an acknowledgement, hears nothing, the receiver needs two and a half seconds, and records the step as UNKNOWN. The acknowledgement lands at about 2.5 seconds, the claim row moves to done, and the sender writes "ACK received" in its own log. All of that is true in both runs. The question is only whether the planner ever learns it before deciding what to do next.

My claim is simple, and I expect some of you to push back on it. Stopping the duplicate is the easy half of this fix. The other half is telling the planner the step succeeded, out loud, in a place it actually looks. A fix the planner never hears about is a fix that only works until the planner's next decision.

The two runs

In the first run, the planner reads the claim row before planning anything. It asks for the status of evt-1042 and gets back done. The late discovery arrives as a state change it can see: UNKNOWN to SUCCEEDED, on that key, at half past three seconds into the run. Its next plan is step two, send the customer confirmation for evt-1042, and do not send evt-1042 again. The receiver logged one delivery. Zero duplicate payments.

In the second run, the planner reads nothing. The sender's log line exists, the claim row says done, but the planner never opens either one. Its belief is still UNKNOWN, the same belief it formed at one second, and it plans from that: evt-1042 must have failed, so do the step again, with a new key, evt-1043, and send the payment again. And it does. The receiver logged two deliveries, and the second one is a duplicate payment under a fresh key. Same fix, same receiver, same timings. The only difference is one read.

Told the planner Kept it quiet
Belief at 1.0 s UNKNOWN UNKNOWN
Transition UNKNOWN to SUCCEEDED seen Yes, from a claim-status read No
Next plan Step 2: confirm evt-1042, do not resend Redo step 1: new key evt-1043, send again
Deliveries the receiver logged 1 2
Duplicate payments 0 1

I ran the whole thing twice. Both runs gave the same numbers and the same plans, to the second.

Problem 1: beliefs form early and then freeze

The planner did nothing wrong at one second. With the information it had, UNKNOWN was the honest answer: no acknowledgement, receiver slow, step might have failed. The mistake is not the belief. The mistake is that nothing ever revisits it. A belief formed in the first second of a run is still steering the plan at the fourth second, long after the world changed underneath it, unless something goes back and checks.

This is the part people miss when they talk about retries. They argue about timeouts and intervals, the same argument as yesterday, and they treat the planner as if it watches the whole run the way a person would. It does not. It watches the moment it last looked, and then it plans. If the last look was at one second, the plan is built on a one-second-old world.

Problem 2: a log line nobody reads is not feedback

This is the finding that stung most, because the second run had the information. The sender wrote "ACK received for delivery_id=evt-1042" in its own log. The claim row sat there saying done. Both facts existed before the planner decided. Neither one mattered, because the planner never looked at either of them.

A log line is a diary entry. A state change the planner reads before deciding is feedback. The difference is not where the fact lives, it is whether the decision reads it. Every system I have built has a log full of facts nobody ever read again, and I have caught myself pointing at those logs as if their existence proved something was communicated. It proves something was written down. Those are not the same thing.

Problem 3: the redo uses a new key, and the claim row cannot save you from that

Here is the part that makes the quiet fix actively dangerous rather than merely incomplete. When the planner redoes the step, it does not reuse evt-1042. It assigns a new key, evt-1043, because from its point of view the first key belongs to a failed step. The claim row protects a key. It cannot protect a payment the planner believes was never made. The duplicate in my second run did not slip past the claim row. It walked around it, through the front door, wearing a new key.

That is why "fix it quietly when nothing downstream depends on the agent knowing" is a narrower rule than it sounds. The payment confirmation, the next step in the chain, the customer email, all of them depend on the agent knowing. The only step where the quiet fix is safe is one with no next step at all, and those are rarer than they look.

Problem 4: telling the planner costs one read

The fix for this half is embarrassingly small, which is probably why it gets skipped. Before the planner decides the next step, it reads the claim row for the key it is deciding about. One read, one status, done or not. In my run that read is the entire difference between one delivery and two, between a confirmation email and a duplicate payment.

There is a temptation to make this fancier, to build a whole notification channel, an event bus, a watcher. Resist it, at least at first. The planner already knows how to read the row. The row already holds the truth. The missing piece was never a channel. It was the habit of reading before deciding.

Two honest limits, because the test was small. First, this was a tabletop model of a planning loop on my own machine, not a real agent framework: one key, one receiver, a planner with exactly one rule, plan from the freshest state you actually read. A real agent has more inputs, but the failure is the same shape, a belief formed early and never corrected. Second, the claim row still needs a lease in real life, a crashed claimer holds its row forever, and that part is still untested here, same as yesterday.

If you run agents against webhooks, or against anything where a step can succeed slowly, you have met this bug. What did your agent do with the late success, redo the work, or carry on as if nothing had happened? And where does your planner read the state back, before it plans, or never?

P.S. Everything I write stays free — that part doesn't change. But if this one was worth something to you, you can buy me a coffee ☕ (one-time or monthly, both work). It covers the API credits and hosting behind these experiments — and honestly, it's also the nudge that gets the next one started on the evenings I'd otherwise skip it. — Salam

Buy me a coffee

AheadKit on Gumroad — Beginner-Friendly Books & Packs