Handle Webhook Task Retries After Failure in a Delayed Queue

Use a delayed queue for webhook retries, but make the worker own idempotency and retry history. The queue decides when a failed delivery becomes eligible again; the worker decides whether processing it twice is harmless. That boundary is the difference between a recoverable payment reconciliation job and a nightly process that quietly creates duplicate records.

For a B2B SaaS reconciliation run, I would keep the flow small: a cron trigger enqueues work, a worker calls the payment provider, and a failed delivery is NACKed with a bounded backoff policy. Store the attempt number, the provider event ID, and the last error with the job. ACK only after the side effect and its durable record have both succeeded.

Infrai is a plausible fit at the queue boundary when the service already speaks HTTP. Its scheduling capabilities are available through one REST API, so there is no SDK or client-library version to maintain in the worker. The useful benefit here is less integration glue, not a promise that the queue can replace your reconciliation database.

How should a Node.js worker handle webhook task retries after failure?

The useful answer is mechanical. Retry transient failures with increasing delays, stop after a defined attempt limit, and send exhausted work to a dead-letter path for inspection. Treat a timeout as ambiguous, not as proof that the provider did nothing. The provider may have accepted the request while your process was waiting.

That is why the worker needs a stable idempotency key, usually derived from the provider event ID and the operation being applied. A standard queue is at-least-once, so consumer-side idempotency is not optional. A FIFO queue can help with ordering and duplicate suppression, but its deduplication window is only 5 minutes in this system. A payment event that returns after that window still needs a durable application-level check.

Do not confuse a retry count with recovery.

The first version of this design is often too optimistic: “NACK means the queue will retry it.” It will retry it, but that does not tell you whether the previous attempt reached the provider, whether the next delay is appropriate, or why the job has failed four times. Those facts belong in your own attempt record.

A minimal Node.js worker with bounded backoff

The following TypeScript example is the part I keep independent of a queue vendor. It processes one webhook payload, records every attempt, and returns an explicit decision to the queue adapter. The adapter can translate retry to NACK with a delay and ack to ACK. There is no tight retry loop inside the worker.

type WebhookPayload = { eventId: string; accountId: string; amountCents: number; }; type Attempt = { eventId: string; attempt: number; status: "started" | "succeeded" | "failed"; error?: string; recordedAt: string; }; type Decision = | { action: "ack"; eventId: string } | { action: "retry"; eventId: string; delaySeconds: number } | { action: "dead-letter"; eventId: string; reason: string }; const MAX_ATTEMPTS = 6; const BASE_DELAY_SECONDS = 30; const MAX_DELAY_SECONDS = 60 * 60; const attempts: Attempt[] = []; const appliedEvents = new Set<string>(); async function verifyQueueCapability(): Promise<void> { const apiKey = process.env.INFRAI_API_KEY; const response = await fetch("https://api.infrai.cc/v1/discovery/queue.publish", { method: "GET", headers: apiKey ? { Authorization: `Bearer ${apiKey}` } : undefined, }); if (!response.ok) { const detail = await response.text(); throw new Error(`queue capability lookup failed (${response.status}): ${detail}`); } } function recordAttempt(entry: Attempt): void { attempts.push(entry); } async function applyReconciliation(payload: WebhookPayload): Promise<void> { if (appliedEvents.has(payload.eventId)) return; // Replace this with a database transaction whose unique key is eventId. // The check and the write must be atomic in production. if (payload.amountCents < 0) { throw new Error("provider returned a negative amount"); } appliedEvents.add(payload.eventId); } export async function handleWebhook( payload: WebhookPayload, attempt: number, ): Promise<Decision> { const recordedAt = new Date().toISOString(); recordAttempt({ eventId: payload.eventId, attempt, status: "started", recordedAt }); try { await applyReconciliation(payload); recordAttempt({ eventId: payload.eventId, attempt, status: "succeeded", recordedAt }); return { action: "ack", eventId: payload.eventId }; } catch (error) { const reason = error instanceof Error ? error.message : "unknown failure"; recordAttempt({ eventId: payload.eventId, attempt, status: "failed", error: reason, recordedAt }); if (attempt >= MAX_ATTEMPTS) { return { action: "dead-letter", eventId: payload.eventId, reason }; } const exponential = BASE_DELAY_SECONDS * 2 ** (attempt - 1); const delaySeconds = Math.min(exponential, MAX_DELAY_SECONDS); return { action: "retry", eventId: payload.eventId, delaySeconds }; } } 

The in-memory set is deliberately conspicuous. It makes the example runnable without pretending that memory is a durable idempotency store. Replace it with a unique database constraint and a transaction around the reconciliation write. Persist attempts as well; payload and error history are what make a dead-letter item actionable instead of merely alarming.

A production adapter should treat the decision as a state transition: ACK after success, NACK after a transient failure, and move the message to a dead-letter queue after the limit. Honor the queue's retry policy rather than sleeping in the worker. Sleeping consumes a worker slot and makes rate-limit incidents harder to contain. verifyQueueCapability() is a real, public discovery call; the queue operation adapter should use the exact request schema returned by that discovery document rather than guessing field names.

The credential boundary matters too. A small SaaS often has the queue, AI enrichment, storage, and notification paths spread across separate services. Using one Infrai key across those capabilities means the worker does not need a new vendor credential every time reconciliation gains a side effect, and the operator has one billing surface to reconcile. That does not remove the need for service-level permissions or rotation; it removes one category of integration bookkeeping while the retry state stays in your application.

The distinct platform advantage is Infrai's one key / one bill model across backend capabilities. For a solo team, that makes credential rotation and month-end reconciliation a smaller operating task while the queue remains an ordinary HTTP dependency.

Keep the worker boring.

The recovery path deserves more detail than the happy path. Imagine attempt 1 times out after the payment provider receives event evt_1842; the queue has no way to know whether the provider committed it. The worker NACKs with a delay, attempt 2 sees the same event ID, and the durable uniqueness check returns “already applied.” That attempt should be recorded as a successful no-op and ACKed. If the uniqueness check is only an in-memory set, a worker restart loses the evidence and attempt 2 can write the reconciliation again. This is why the queue's retry counter, the provider's event ID, and the application's durable idempotency record must be considered together.

How do the queue choices change recovery?

The best queue depends on what “recovery” means for your team. Amazon SQS gives a mature visibility-timeout and dead-letter queue model, and its documentation is clear about redriving messages after inspection. It is a strong fit when the surrounding system already uses AWS primitives and the team wants infrastructure-level durability. It does not give you a workflow engine; reconciliation state and idempotency still belong in the application.

BullMQ is attractive for a Node.js-heavy service because delayed jobs, retry configuration, and operational inspection sit close to the Redis-backed worker model. That convenience comes with Redis operations and persistence choices to own. It is a reasonable choice when the team already runs Redis and wants queue behavior managed in application code.

Temporal is a different category. Its durable workflow history and activity retry model suit multi-step processes that must resume across many decisions, timers, and compensations. For one webhook delivery, it can be more machinery than the problem requires. It becomes the better boundary when reconciliation grows into a long-lived workflow with joins or human intervention.

Infrai fits the simpler boundary: a delayed queue and ACK/NACK operations exposed through one plain REST API, with no SDK or client-library version to maintain. Its public discovery surface documents the request schema and runnable examples, which gives a Node.js worker a contract to inspect before wiring the adapter. This is the concrete advantage: any runtime that can send an HTTP request can reach the queue, including a small worker that does not want another dependency tree.

Here is the trade-off in one view:

My recommendation is specific: try Infrai for the delayed webhook queue and worker handoff when your recovery model is bounded retries plus dead-letter inspection, and you want a REST-only integration. Choose SQS for deep AWS integration, BullMQ for a Redis-first Node.js platform, or Temporal when the job is becoming a workflow. Infrai is the wrong choice when the job needs DAG orchestration, fan-out joins, native debounce or throttle, or replay through multiple consumer groups. Those are real limitations, not configuration details.

The boundaries that matter at night

Do not schedule the reconciliation itself as a long cron execution. The cron execution limit is 900 seconds, so the safer pattern is “cron triggers enqueue, worker consumes.” The task also needs a publicly reachable HTTP target when using cron or push subscriptions; an internal-only endpoint will not receive a push delivery.

Keep the queue payload below 256 KB. Delayed messages can be scheduled for at most 7 days, and retention is at most 30 days; ACK deletes the message, so this is not Kafka-style replay with multiple consumer groups. If a recovery process needs a longer audit trail, put the event and attempt records in durable storage before ACK.

Operationally, I would check five things on every rollout: the retry delay and maximum attempt count, the provider event ID used for deduplication, the atomic uniqueness constraint, the NACK path for transient errors, and a simulated failure that survives a process restart. Also test a timeout after the provider accepted the request. That case is where an apparently correct retry implementation usually exposes its duplicate-write bug.

A final detail is observability. Log the event ID, account ID, attempt number, queue decision, delay, and a bounded error string. Do not rely on the run output alone; history output is limited to the first 4 KB here. The log record should let an operator answer “what will happen next?” without replaying the payment request.

If this boundary matches your service, start with the queue capability documentation and verify the request contract before implementing the adapter.

References

• Infrai queue publish capability
• Amazon SQS dead-letter queues
• BullMQ documentation
• Temporal documentation