A renewal reminder for a subscription box can't go out whenever a worker happens to pick it up. Merchant policy puts it between 72 and 24 hours before the card is charged, consumer-protection rules in several jurisdictions put their own floor under that window, and the whole point of the notice is that the user still has time to cancel. So the job sits somewhere for days — and every day it sits is another chance for a queue retry, a redeploy, or an operator replaying a backlog to send the same message twice. Use a durable claim ledger keyed on the business fact, written before the send and committed after it, and treat the broker's delivery guarantee as at-least-once rather than as your deduplication layer.
That ledger becomes the only component allowed to answer "did this notice already go out?"
Examples here are Python. A Node.js worker needs the same three steps in the same order — pg instead of psycopg, node:crypto instead of hmac — because the ordering, not the language, is what keeps duplicate processing out of the customer's inbox.
What makes a delayed reminder queue send the same message twice after a retry?
Three different mechanisms produce the same symptom, and they need different answers.
The first is plain at-least-once redelivery. The worker renders the reminder, hands it to the mail provider, and dies before it acknowledges the message; the broker has no way to know how far the handler got, so it redelivers. Managed queues say this out loud — Pub/Sub documents at-least-once as its default delivery semantic — and any system built on visibility timeouts behaves the same way.
The second is subtler and it's the one that burns people: the visibility timeout expires while the send is still in flight. A slow TLS handshake with the SMTP relay, a provider throttling you to a few hundred messages a second, one retry on a 4xx — thirty seconds gone, ack deadline blown, and the broker hands the identical message to a second worker while the first is still talking to the provider. Both workers believe they are the first. Any dedupe check that lives in worker memory loses here, because the two workers don't share memory.
The third source isn't the queue at all. It's recovery. The scheduler gets paused for a database failover, the backlog builds, and someone re-enqueues everything due between 09:00 and 15:00 to catch up. Nothing in the broker remembers which of those notices already went out before the pause.
Broker-native deduplication doesn't reach any of this. A FIFO deduplication ID covers a five-minute window; a renewal reminder is enqueued up to 30 days before its deadline. The window closed weeks before the duplicate happened.
Deriving the idempotency key from the business fact
The identity of a reminder is a business tuple: this subscription, this billing period, this kind of notice. Not the message id the broker assigned, not the attempt number, not scheduled_for, and definitely not the worker hostname. Anything that changes when the job is replayed mints a fresh key and quietly re-enables the duplicate you were trying to stop.
I use a keyed digest rather than a bare hash — HMAC-SHA256 as described in RFC 2104 — mostly because the key ends up in places I don't fully control: log aggregators, the Message-ID header, provider metadata fields. A keyed digest keeps raw subscription identifiers out of those places. If your ledger never leaves your own database, a plain SHA-256 over the same tuple is fine and I wouldn't argue with it.
import hashlib import hmac import os NOTICE_SECRET = os.environ["NOTICE_KEY_SECRET"].encode() def notice_key(subscription_id: str, period_start: str, kind: str, revision: int = 0) -> str: """Stable identity for one user-facing notice. Business identity only. No broker message id, no attempt counter, no send timestamp: a replay days later must derive exactly the same key. """ material = "|".join([subscription_id, period_start, kind, str(revision)]) return hmac.new(NOTICE_SECRET, material.encode(), hashlib.sha256).hexdigest() notice_key("sub_8812", "2026-10-01", "renewal_notice_t72") The revision field is there for the case that trips up most ledgers built without one. A customer downgrades the plan, the renewal amount changes, and the merchant decides a corrected notice has to go out before the same deadline. That is a different notice, not a retry, so it bumps to revision=1 and gets sent on purpose. Keep that decision in the producer, where someone can reason about it, and let the worker stay dumb.
Claim, send, commit — and the window where the worker dies
The order is not negotiable: claim the key and commit that claim, send, commit the result, then ack the broker. Each step exists because of a specific crash position.
CLAIM = """ insert into notice_ledger (notice_key, subscription_id, kind, state, claimed_at) values (%s, %s, %s, 'claimed', now()) on conflict (notice_key) do nothing returning notice_key """ def deliver(conn, job) -> str: key = notice_key(job["subscription_id"], job["period_start"], job["kind"]) claimed = conn.execute(CLAIM, (key, job["subscription_id"], job["kind"])).fetchone() conn.commit() # durable before anything leaves the process if claimed is None: return "suppressed" # another attempt owns it; ack and move on receipt = send_notice( to=job["email"], body=render(job), message_id=f"<{key[:32]}@notices.example.com>", ) conn.execute( "update notice_ledger set state = 'sent', provider_id = %s, sent_at = now()" " where notice_key = %s", (receipt.provider_id, key), ) conn.commit() return "sent" Crash between the claim and the send and the row sits at claimed forever while the customer gets nothing. Crash between the send and the update and the row also sits at claimed — except this time the mail is already in flight. Those two states are indistinguishable from inside the worker, which is exactly why a stale claim must never be released automatically. An auto-release sweeper is a duplicate generator with a cron schedule.
Send them to a quarantine view instead and reconcile against the provider's event stream.
select notice_key, subscription_id, claimed_at from notice_ledger where state = 'claimed' and claimed_at < now() - interval '15 minutes' order by claimed_at limit 500; Setting your own Message-ID is what makes that reconciliation cheap. Delivery, bounce and complaint webhooks echo it back, so a quarantined claim can be resolved against what the provider actually accepted instead of a guess. The quarantine stays small in practice, since it only fills from crashes inside a window of a few seconds, and clearing it is a lookup rather than a judgement call.
Never ack the queue message before the ledger commit. Ack-first loses notices; commit-first duplicates them, and duplicates are what the ledger absorbs.
| Where the dedupe check lives | Survives a multi-day delay | Survives an operator replay | Cost on the send path |
|---|---|---|---|
| Broker deduplication ID | No — the window is minutes | No | Free, already in the queue |
| In-process cache | No | No | Nanoseconds, lost on restart |
| Shared cache claim with TTL | Only if the TTL outlives delay plus retries | While the key is alive | One round trip |
| Durable ledger row | Yes | Yes | A write plus a commit |
Replaying a backlog without re-sending yesterday's reminders
Treat a replay as an ordinary producer. It re-enqueues jobs, the ledger refuses the ones that already happened, and the operator never has to compute a safe time range by hand — which they will get wrong at 02:00 during an incident.
Before running it for real, count what it would mint: select the derived keys against the ledger and report how many are new. If a six-hour replay finds 4 unknown keys out of 900, the backlog was already delivered and the safest action is to stop.
Two things make replays go sideways even with a correct ledger. The first is throughput: 40,000 reminders released in one burst hits per-domain rate limits, and receiving mail systems read a sudden identical-content spike as exactly what it looks like. Push replay traffic through the same per-domain token bucket as normal sends and expect the pipeline, not the ledger, to be the bottleneck. The second is embedded credentials — if the reminder carries a one-time code or a signed cancellation link, a suppressed duplicate is correct behaviour, but a deliberate resend has to reuse the stored code. Mint a new one and the first message stops working, which produces a support ticket that is much harder to explain than a duplicate.
Put the replay batch id in logs and metrics. Never in the key.
Rolling this out on a live reminder pipeline
Run it in shadow first. Write claims, log the ones that would have been suppressed, enforce nothing, and let it ride through one full renewal cycle so that monthly and annual plans both pass through it. The suppression rate you observe is your real duplicate rate, and it's usually higher than the bug reports suggest, because customers silently ignore a second reminder more often than they complain about it. Backfill keys for notices already in flight before you flip enforcement on, otherwise the first enforced run treats every pending job as new.
Four metrics are enough: claims minted, duplicates suppressed, claims in quarantine, and the p99 gap between claim and commit. Alert on the quarantine count crossing zero and on that gap growing, since a widening gap means the crash window is getting wider too.
The catch is that every send now costs a durable write plus a commit before the network call, and that write is on the critical path. If the notice is harmless to repeat — an internal webhook, a metrics ping, a cache warm — stick with the broker's at-least-once guarantee and skip all of this. At millions of sends an hour, a shared cache claim with a TTL longer than the full delay window is the pragmatic middle ground, though it doesn't support the reconciliation story above; once the key expires, so does your evidence. And if your reminders are generated fresh from a nightly scan of the subscriptions table rather than queued weeks ahead, most of this collapses into a unique index on the table you already have, which is a trade I'd take every time.
