The receiver was not down. That is the part I keep coming back to. It was answering every single delivery, taking its time about it, two and a half seconds from first byte to done. My sweeper checked once a second for an acknowledgement, heard nothing after one second, and decided the delivery must be lost. It was not lost. It was still being written down at the other end. By the time the first acknowledgement arrived, the same order had already gone out three times.

That was this morning, on my own machine, with one delivery key and a receiver I had deliberately slowed down. krusenas raised this in the comments on "My Cron Job Fired 50 Times While the Receiver Was Down. 10 Webhooks Vanished." He wanted the slow case, not the dead one. What happens when the receiver is up, healthy, and simply a second too slow for the thing watching it? A dead receiver fails loudly. A slow one looks exactly like a dead one for two and a half seconds, and a sweeper that resends on silence will happily manufacture duplicates out of a delivery that was never in trouble.

Here is the setup, so you can argue with it. One delivery, key evt-1042, a payment confirmation for a single order. A local receiver that sleeps 2.5 seconds before answering 200. A sweeper that ticks every 1.0 second and resends whenever no acknowledgement has arrived. The sender waits up to 4 seconds for each reply, so the first acknowledgement does land, at about 2.5 seconds. I counted duplicates on the receiving side, in the receiver's own log. Then I ran the whole thing twice.

Before the fix After the fix
Sends 3 1
Deliveries the receiver logged 3 1
Extra copies 2 0

Both runs gave the same numbers. Before the fix, the sweeper fired at 0 seconds, at 1 second, and at 2 seconds. The first acknowledgement arrived at 2.5 seconds, a half second after the third copy had already left. After the fix, one send, one delivery, one acknowledgement, and the sweeper spent its next two ticks doing nothing at all. That nothing is the whole point of this piece.

My claim is simple, and I expect some of you to disagree with it. Longer timeouts and longer sweep intervals do not fix webhook duplicates. Only claiming the work once does. Everything else is a new number on the same trap.

Problem 1: silence is not loss

A sweeper that resends on no acknowledgement treats two completely different situations as one. The delivery is gone, or the delivery is slow. From the sender's chair those look identical for the first few seconds. There is no signal in the silence that tells you which one you are in. So the sweeper guesses "lost", because that is the guess a retry loop is built to make. In my run the guess was wrong three times in a row, and each wrong guess was a real copy landing on a real receiver.

People wave this away. "It was only a duplicate, the receiver can ignore it." The receiver in my test was a simple order inbox. It logged three deliveries for one order. If that inbox charges a card, sends a confirmation email, or decrements stock, three deliveries is three of those, unless something downstream catches it. The sweeper created work the receiver never asked for, and then the receiver gets blamed for handling it badly.

Problem 2: the sweeper cannot learn patience from a longer timeout

My first instinct, and probably yours, was to stretch the numbers. Give the sender a longer wait, or the sweeper a longer gap between ticks. I did the arithmetic instead of the change, and the arithmetic is unkind. A 4-second wait already covered the 2.5-second receiver. The timeout was never the problem. The sweeper fired at 1 second and 2 seconds because that is its job: check every second, resend on silence. A longer timeout changes nothing about the second tick.

Stretch the sweep interval instead, say from 1 second to 3 seconds, and this exact receiver stops duplicating. The acknowledgement lands at 2.5 seconds, inside the gap, and the sweeper never fires twice. That feels like a fix. It is a fix for a 2.5-second receiver, on a quiet morning. The next slow day the receiver takes 4 seconds, or 7, or a lock holds it for 12, and the 3-second gap manufactures the same duplicates again. You have not removed the trap. You have moved it to a number you have not hit yet.

Problem 3: every duplicate is a second bill

Count what three deliveries actually cost, even when the receiver is polite enough to ignore two of them. Three sends, three lots of work at the receiving end, three chances for a side effect to slip through. On a busy day that multiplier lands on everything at once, because the slow day is the day every delivery is slow, and the sweeper is resending every one of them. A retry loop that cannot tell slow from lost multiplies the load on the exact receiver that is already struggling, at the exact moment it is struggling. That is a strange way to help.

The fix has to live where the decision lives. The decision is "send this again", and it has to be made once, by one owner, with a record everyone else can see. A longer interval is a hope that the receiver hurries up. A claim is a fact that the work is taken.

Problem 4: a flag in memory is not a claim

The obvious cheap version is a flag in the sender's memory: sending equals true, do not send again. That works until the sender restarts, or until there are two senders, or two copies of the sweeper running side by side, which is exactly how these things end up running. Each one holds its own flag, each one is sure it is the only sender, and the receiver gets two copies anyway. A flag nobody else can see is a private opinion. A claim has to be a shared fact.

So the fix is one row per delivery key, in a table every sender can see, claimed atomically. One row, one key, one owner at a time.

CREATE TABLE claims ( delivery_key TEXT PRIMARY KEY, status TEXT, claimed_at REAL ); 

The sweeper does not check and then claim. It claims, in one step, and the table decides who won. The insert succeeds for the first caller and fails for everyone else, because the key is the primary key and there can only be one row for it. The winner sends. The losers see status in_flight and skip their tick. When the acknowledgement arrives, the winner marks the row done. A slow run cannot be re-claimed while it is in flight, because the row is already there, already owned, already visible to the next tick.

That is the entire change. Same receiver, same 2.5-second delay, same 1-second sweep, same 4-second wait. Three deliveries became one. The sweeper still ticked every second. It just had nothing to do on the second and third ticks, because the row told it the truth the silence could not: this delivery is already being handled.

Two honest limits, because the test was small. First, this was local: one key, one receiver, no real network, no crash in the middle of a send. A claimer that crashes while holding a row would hold it forever unless the row carries a lease that expires, and I did not test a lease. Do not ship a claim without one. Second, the receiver here was slow, never down. The down case was the previous piece. The slow case is the sneakier of the two, because everything looks fine until you count the copies.

If you run webhooks, you have met this bug, probably on a day you were already busy. What did your duplicate cost you, a double charge, a double email, a customer asking why they got three copies of the same thing? And what fixed it for you, a claim like this one, a key check at the receiving end, or something else entirely?

P.S. Everything I write stays free — that part doesn't change. But if this one was worth something to you, you can buy me a coffee ☕ (one-time or monthly, both work). It covers the API credits and hosting behind these experiments — and honestly, it's also the nudge that gets the next one started on the evenings I'd otherwise skip it. — Salam

Buy me a coffee

AheadKit on Gumroad — Beginner-Friendly Books & Packs