Node.js Queue or Cron? HTTP Worker Recovery After Webhook Failures

Short answer: use a durable queue to retry failed webhook jobs, let an HTTP worker make one delivery attempt at a time, schedule delayed retries without sleeping, and move exhausted work to a dead-letter queue; keep cron for periodic reconciliation, not per-job timing.

The deciding constraint is the retry budget. Each delivery needs an attempt count, a next eligible time, a maximum age, and a terminal outcome that an operator can inspect. A queue already gives that state a natural lifecycle. A cron loop can be simpler for a tiny workload, but only if coarse timing and application-owned claiming logic are acceptable.

The language is secondary. A Node.js producer and a Python worker can follow the same job contract, which is useful when a notebook experiment is becoming a production service and the delivery policy needs evals before it needs another framework.

How should Node.js HTTP workers retry failed webhook jobs: queue or cron?

Choose by asking where one webhook's state lives while it waits. With a queue, the job is available, delayed, claimed, acknowledged, or dead-lettered. Workers scale independently and concurrency has one obvious control point. With cron, a scheduled process queries a table for due rows, claims a batch, sends requests, and updates those rows. The table is now acting as a queue, while the application owns leases, competing pollers, indexes, and recovery after a process exits between the HTTP call and the update.

That's the real comparison.

Cron is still suitable when the work is genuinely periodic: refresh a cache every hour, generate a daily report, or scan for discrepancies on a loose recovery window. It can also be reasonable for a very small callback system when the team already operates a transactional database and is willing to build and test row claiming. The catch is that a one-minute poll creates up to a minute of scheduling granularity, and many rows becoming due together can produce a burst unless the scanner deliberately limits its batch and send rate.

A durable queue is suitable when retries must be independent, delay precision matters, destinations need separate backpressure, or failed deliveries need an auditable terminal state. It adds a broker and operational work: queue-age alerts, dead-letter review, redrive controls, and retention choices. Don't add that machinery to a weekly best-effort task whose missed run can be handled safely by the next cron tick.

In-process timers are smaller still, but only for disposable development runs. They disappear with the process, don't coordinate across replicas, and make a restart part of the delivery policy. Once a callback matters, memory is the wrong ledger.

Start with a delivery state machine, not a scheduler

The smallest production shape has a producer, durable retry state, stateless HTTP workers, a dead-letter destination, and a low-frequency reconciler. The producer creates a stable delivery ID. The worker claims a job, resolves the destination and signing secret, builds the request, performs one network attempt, and returns a decision. The queue adapter then acknowledges the job, makes it eligible after a delay, or dead-letters it.

One attempt. Then stop.

The worker must never sleep for backoff. Sleeping consumes a concurrency slot and loses the delay if the process ends. Store the next eligible time in durable infrastructure instead. Use exponential backoff with jitter to avoid a synchronized retry wave, but cap both the delay and total delivery age. Exact values depend on the receiver's contract and the event's usefulness; I'm not sure a universal attempt count exists, because a stale login callback and an analytics notification have different consequences. An explicit policy version resolves that uncertainty better than a magic constant hidden in worker code.

Delivery also needs an honest duplicate model. A receiver can accept and commit a request while the sender loses the response. The sender cannot prove whether the side effect happened, so a later attempt may repeat it. Keep the stable delivery ID across retries and require receivers to deduplicate harmful side effects. “Exactly once” in a diagram doesn't remove this ambiguity.

Sign the exact request bytes rather than parsing and re-serializing them after signing. RFC 2104 defines HMAC as keyed hashing for message authentication. A practical webhook contract also has to define what bytes are signed, which algorithm is used, how keys are identified, and how replay protection works. Those contract details should be fixtures in the eval harness, not prose that every SDK interprets differently. Secrets belong in secret storage and are resolved at send time; putting them in queue payloads widens access and complicates rotation.

The producer has its own failure window. If business data commits and the process exits before publishing the job, no retry policy can recover the missing intent. A transactional outbox is appropriate when the business write and the delivery intent must commit together: write both in one database transaction, then publish unsent outbox rows asynchronously. The reconciler can compare durable intent with queue state, but it should remain a safety net rather than the primary retry clock.

Keep the HTTP worker boring and testable

This focused Python function models the part I want to move from notebook to production: deterministic request bytes and a policy decision. It doesn't know which queue implementation is underneath it. The adapter owns attempts, delayed scheduling, age limits, acknowledgements, and dead-letter writes.

from dataclasses import dataclass from hashlib import sha256 import hmac import json import urllib.error import urllib.request @dataclass(frozen=True) class Delivery: delivery_id: str url: str event: dict @dataclass(frozen=True) class Result: action: str reason: str def make_request( job: Delivery, secret: bytes, timestamp: int ) -> urllib.request.Request: body = json.dumps(job.event, separators=(",", ":")).encode("utf-8") signed_bytes = b".".join( [str(timestamp).encode(), job.delivery_id.encode(), body] ) signature = hmac.new(secret, signed_bytes, sha256).hexdigest() return urllib.request.Request( job.url, data=body, method="POST", headers={ "Content-Type": "application/json", "X-Webhook-Delivery": job.delivery_id, "X-Webhook-Timestamp": str(timestamp), "X-Webhook-Signature": signature, }, ) def deliver_once( job: Delivery, secret: bytes, timestamp: int, retryable_statuses: set[int], ) -> Result: request = make_request(job, secret, timestamp) try: with urllib.request.urlopen(request, timeout=10) as response: status = response.status except urllib.error.HTTPError as error: status = error.code except (urllib.error.URLError, TimeoutError): return Result("retry", "transport") if 200 <= status < 300: return Result("acknowledge", "accepted") if status in retryable_statuses: return Result("retry", f"status_{status}") return Result("dead_letter", f"status_{status}") 

The retryable set is injected because HTTP status alone doesn't describe every webhook contract. For one eval fixture I might classify 429 as retryable and 400 as terminal, but the receiver's documented contract must decide. That keeps policy changes reviewable and prevents a generic library from silently retrying a permanent schema error.

The same boundary works with a Node.js worker. Decode the common job schema, call one delivery function, and map its result to the broker. BullMQ's documentation shows delayed jobs, retries, workers, concurrency, and job schedulers as queue primitives; it is evidence for the model, not a reason to couple the delivery function to one library. A database-backed adapter or managed queue can implement the same three outcomes.

Measure the retry budget before copying this design

The useful dashboard starts with queue age, not raw queue length. Length can rise during an expected burst while workers keep up; age shows whether the oldest eligible webhook is missing its delivery objective. Add attempts per delivery, success by destination, worker duration, delayed-job count, dead-letter growth, and redrive outcomes. Log the delivery ID, destination identifier, attempt, policy version, duration, and result class. Don't log signing secrets or full payloads.

For AI applications, count downstream work too. A duplicate webhook that starts an agent run can repeat model calls and prompt cost even when the queue metrics look normal. The receiver's idempotency check should happen before expensive work, and the eval suite should verify that ordering.

My pre-production eval matrix would include a fixed HMAC fixture, duplicate jobs with one delivery ID, two workers competing for one job, worker termination after receiver acceptance, a transport timeout, response 429, a permanent response, exhausted attempts, expired delivery age, and dead-letter redrive. The duplicate case deserves more than a checkbox: send delivery evt_1042, let the receiver persist that ID and its side effect, then end the connection before the worker receives a response. Make the job eligible again and send it through a different worker. The expected result is two HTTP attempts but one receiver-side effect, followed by acknowledgement of the retry; the logs should connect both attempts through evt_1042 without exposing the payload. If the second request launches another model call, the idempotency boundary is too late. None of these cases requires a large benchmark. Each needs an observable expected state.

Load tests should separate cold and warm workers, vary destination latency, and include one slow destination beside healthy ones. The question is not merely peak throughput — it is whether one receiver can consume all concurrency and push unrelated webhooks past their delivery budget. Per-destination limits or fair scheduling may be necessary when that isolation matters, though they add configuration and can reduce utilization.

Use cron when periodic scanning is actually the job and a loose recovery window is honest. Use a queue when every failed webhook needs durable, independently delayed state. Before choosing either, measure the oldest-job age, duplicate effect rate, redrive burden, and downstream cost under failure. Those results turn “simplest” from an aesthetic preference into an operational decision.

References

• RFC 2104: HMAC keyed-hashing for message authentication
• BullMQ documentation