A media subscription renewal reminder has a deadline, and the queue does not know about it. The queue only knows delivery. With an at-least-once background job queue, retries and duplicate jobs are normal; the engineering decision is how replay is governed so one renewal cycle creates one reminder intent.
Short answer: make replay a recorded business decision. Derive one idempotency key from the renewal cycle, store the decision and outbound intent together, and route every later delivery through that record. Acknowledging a message is not evidence that the reminder happened.
This framing matters during incidents. A consumer can disappear after committing storage but before acknowledging its delivery. The next consumer must be able to say “already decided” without calling the mail provider again.
Define the replay subject before tuning the queue
For a renewal due at 2026-09-01T09:00:00Z, the reminder subject is one subscription, one renewal cycle, and one reminder kind. That tuple is the replay subject. A delivery ID, worker ID, or attempt number is not; each changes when the transport retries.
The producer calculates remindAt once and publishes it with the business data. It also canonicalizes the cycle string. 2026-09 and September-2026 must not become two identities for the same event.
One key. One intent.
type RenewalReminder = { subscriptionId: string; renewalCycle: string; remindAt: string; recipient: string; }; function idempotencyKey(job: RenewalReminder): string { return ["renewal-reminder", job.subscriptionId, job.renewalCycle].join(":"); } I start with this key in design review because it makes the later audit trail legible. “Duplicate suppressed” should answer which renewal was protected, not which opaque queue message happened to arrive first.
When is a replay safe for a deadline-bound reminder?
The worker needs a small durable ledger. Its states can be claimed, queued, and sent, with an outbox ID attached to the business key. The exact labels are less important than the invariant: a second delivery reads the first decision and returns the same intent.
type Claim = { key: string; outboxId: string | null; state: "claimed" | "queued" | "sent" }; interface Tx { lockClaim(key: string): Promise<Claim | null>; insertClaim(key: string): Promise<Claim>; insertOutbox(job: RenewalReminder): Promise<{ id: string }>; markQueued(key: string, outboxId: string): Promise<void>; } interface Db { transaction<T>(fn: (tx: Tx) => Promise<T>): Promise<T>; } async function accept(db: Db, job: RenewalReminder) { const key = idempotencyKey(job); return db.transaction(async (tx) => { let claim = await tx.lockClaim(key); if (!claim) claim = await tx.insertClaim(key); if (claim.outboxId) { return { outboxId: claim.outboxId, duplicate: true }; } const outbox = await tx.insertOutbox(job); await tx.markQueued(key, outbox.id); return { outboxId: outbox.id, duplicate: false }; }); } The database supplies the guarantee here: a unique constraint on the business key, a unique constraint on the outbox representation, and a transaction that locks the claim. TypeScript describes the contract; it does not serialize two workers. The outbox sender can retry independently while preserving the same outbox ID.
Picture the failure window in detail. Worker A claims renewal-reminder:sub_418:2026-09, writes outbox row out_91, and commits at 08:59:59. It dies before the queue acknowledgment reaches the broker. Worker B receives the redelivery at 09:00:01, waits for the claim lock, sees out_91, and acknowledges without sending. If the key had included A's delivery ID, B would have created a second row even though the business event had not changed. That is the distinction an incident review needs to capture: transport repetition was expected; the ledger made the business effect singular.
How should a Node.js background job queue handle at-least-once retries and duplicate processing?
Treat retry as a policy with four explicit outcomes: retry now, retry later, quarantine, or acknowledge as a duplicate. That vocabulary is useful in dashboards and runbooks because it separates transport behavior from business state.
When a downstream HTTP service returns 429 Too Many Requests, MDN documents Retry-After as the signal for when to try again. Parse either delay seconds or an HTTP date, add bounded jitter, and keep the original idempotency key. If the header is absent, use a documented local ceiling rather than an unbounded loop.
function retryAt(response: Response, now: Date): Date | null { if (response.status !== 429) return null; const value = response.headers.get("retry-after"); if (!value) return null; const seconds = Number(value); if (Number.isFinite(seconds)) { return new Date(now.getTime() + seconds * 1000); } const date = new Date(value); return Number.isNaN(date.getTime()) ? null : date; } Run the sender from the outbox, not from the queue callback. If the callback sends first and records later, a crash can leave an unrecorded email. If it records first and the sender retries by creating a new key, the ledger loses its purpose.
The ledger is the incident artifact
AWS describes a dead-letter queue as a destination for messages that could not be processed successfully after repeated attempts. For renewal reminders, that destination is a review boundary, not a delete button. Preserve the stable business key, original remindAt, attempt count, and a bounded error summary. A replay should re-enter the same claim check.
The useful metrics are decision metrics: duplicate acknowledgments, claim-to-outbox latency, outbox-to-send latency, deadline misses, and dead-letter count. Keep subscription IDs out of metric labels. High-cardinality labels turn a useful incident graph into expensive noise.
| Outcome | Durable record | Next action |
|---|---|---|
| Duplicate delivery | Existing outbox ID | Acknowledge without sending |
| Temporary rate limit | Same outbox ID and notBefore | Retry after the server signal |
| Repeated processing failure | Original key and bounded error | Quarantine for review |
I benchmark the ugly path first: two consumers racing on one key, a worker stopping after the transaction commits, and a provider returning both forms of Retry-After. Clean throughput is a weak proxy for this system's risk.
Set retention from the replay horizon
Storage forgets. If the claim disappears before the maximum dead-letter replay or dispute window, an old delivery can create a second reminder. Keep the key at least as long as that horizon, then test backup restoration against it. Your mileage may vary because the horizon comes from policy and customer support, not queue defaults.
The catch is operational weight. A ledger and outbox require a poller, retention jobs, replay permissions, and an audit view. They are not suitable for a harmless internal update where repeating the write has no effect. Stick with a single idempotent database update in that case. For a customer-facing renewal reminder, the extra machinery buys a clear, inspectable rule: one business intent per cycle.
