A Small SaaS Background Job Queue: HTTP Webhook Workers, DLQ, Cron, and Delays

For a small Node.js SaaS app, a simple background job queue should keep HTTP workers, retries, delayed jobs, cron triggers, and final failures outside the request process. Use a durable work record and a worker boundary that can retry without repeating the business effect; a timer inside the web process is rarely enough.

That answer is deliberately narrower than a feature checklist. The evaluation constraint is operational: a public webhook endpoint for bounded HTTP work, delayed execution, cron-triggered occurrences, retries, a dead-letter path, and a data path that can stay in either the US or EU. If any one of those needs a special exception, the design is not simple anymore.

I would start with the failure I can explain at 2 a.m. A request accepts an account change, records the work durably, and returns. A separate worker attempts the side effect. If the process restarts between those steps, the record remains. If delivery arrives twice, the effect still happens once.

What should a simple Node.js SaaS HTTP worker queue provide for retries, DLQ, delayed jobs, cron triggers, public webhooks, and US/EU placement?

It needs a durable acceptance boundary. A browser-facing route should report success only after the requested work survives a process exit. For work coupled to an application write, an outbox row in the same database transaction is a useful boundary: the application state and the intention to do follow-up work commit together. A dispatcher can deliver the committed row later.

The first tempting implementation is setTimeout in the web server. It looks fine in a demo, then a deploy or restart removes work that exists only in memory. That is a design property, not a Node.js quirk. A durable table can be sufficient for a low-volume app; a separate message system earns its complexity when independent worker scaling, leases, or high sustained throughput are real requirements.

Delivery should be treated as at-least-once unless its contract says otherwise and the whole path proves it. The worker receives a stable work ID, attempts an atomic claim, performs the effect only for the winning claim, and records completion. The provider, application, and downstream API are separate boundaries, so a successful callback response does not magically reveal whether an email, billing request, or model invocation already took effect.

Short version: make the effect idempotent before tuning the queue.

For regional requirements, map the full payload path rather than accepting a regional label. Include the intake endpoint, message body, retry records, dead-letter storage, worker logs, analytics exports, and backups. US or EU placement is a hard eligibility gate. It is not a score to trade against a slightly nicer dashboard.

The experiment: replace process timers with a persisted delivery contract

The simple approach failed the only test that mattered: kill the process after the application accepts work but before the deferred callback runs. A memory timer has no recovery story. The chosen approach stores a small, versioned payload with its next attempt time and lets a generic dispatcher invoke a public HTTP handler.

The useful record is more than a payload blob. It carries an immutable work ID, an occurrence key when the source is cron, a version, an attempt counter, a next-attempt timestamp, and a terminal reason once retries stop. On intake, write it with the user-facing transaction when the two outcomes must agree. On dispatch, claim it using a lease that identifies the attempt without claiming completion. On successful delivery, record the completed work ID before returning success. On a temporary refusal, calculate a future attempt and release the claim. On an invalid request or an exhausted budget, write the terminal state and retain enough context for a person to decide whether replay is safe. Those transitions make the behavior inspectable during a deploy, a duplicate webhook, or a regional audit; without them, the apparent simplicity comes from hiding the operations that still need to happen.

Here is the focused part of that handler. The interfaces are intentionally generic because the state transition matters more than a particular delivery service.

type ReportJob = { version: 1; workId: string; accountId: string; occurrenceKey: string; }; type WorkStore = { claimOnce(workId: string): Promise<boolean>; complete(workId: string): Promise<void>; reschedule(workId: string, nextAttemptAt: Date): Promise<void>; }; type Dependencies = { work: WorkStore; verifyWebhook(request: Request, rawBody: string): Promise<boolean>; generateReport(job: ReportJob): Promise<Response>; }; export async function handleReport( request: Request, dependencies: Dependencies, ): Promise<Response> { const rawBody = await request.text(); if (!(await dependencies.verifyWebhook(request, rawBody))) { return new Response(null, { status: 401 }); } const job = JSON.parse(rawBody) as ReportJob; if (job.version !== 1 || !job.workId || !job.accountId) { return new Response(null, { status: 400 }); } if (!(await dependencies.work.claimOnce(job.workId))) { return new Response(null, { status: 204 }); } const result = await dependencies.generateReport(job); if (result.status === 429) { const retryAfter = Number(result.headers.get("retry-after") ?? "30"); const seconds = Math.min(Math.max(retryAfter, 1), 900); await dependencies.work.reschedule( job.workId, new Date(Date.now() + seconds * 1_000), ); return new Response(null, { status: 202 }); } if (!result.ok) return new Response(null, { status: 400 }); await dependencies.work.complete(job.workId); return new Response(null, { status: 204 }); } 

This example needs a real lease policy in production. A claim that never expires can strand work after an interrupted worker, while a lease that is too short can permit overlapping attempts. Keep the lease, completion record, and idempotency key visible in storage; otherwise a retry investigation turns into log archaeology.

HTTP 429 means the client sent too many requests in a period, and a response can include Retry-After. Treat that header as an input, then clamp it to an application-level limit and apply a retry budget. Authentication failures and invalid payloads should become terminal immediately. Repeating them wastes capacity and makes the actual signal harder to find.

There is a trade-off. HTTP delivery is not suitable for long-running imports, continuously leased work, or handlers that cannot safely expose and authenticate a public endpoint. Use polling workers when the workload needs controlled concurrency and lease renewal; keep a durable table when the workload is small and tightly coupled to database writes. The right choice changes with the failure boundary, not with a marketing comparison.

Delayed work and cron are two different clocks

A delayed item has one intended execution time. A cron expression generates a series of intended occurrences. Mixing them creates duplicate work when a scheduler retries or two scheduler instances briefly overlap.

The scheduler should create ordinary work and exit. For example, the daily usage calculation can use an occurrence key such as usage-rollup:2026-08-07; the worker claims that key through the same idempotency path as an event-driven request. It does not matter whether the trigger arrives early, late, or twice. Only one completed effect is permitted for that occurrence.

Keep schedules in the application’s chosen time zone and persist the intended timestamp, not merely the local clock reading. Daylight-saving changes and deployment windows otherwise create hard-to-reconstruct gaps. I'm not sure a single rule covers every reporting requirement; the decisive question is whether the report is keyed by a business date, a UTC interval, or a customer-local date. Write that choice next to the schedule.

A dead-letter path is an operating procedure, not a bin

A dead-letter queue or terminal-failure table quarantines work after its attempt threshold. It does not repair the cause. AWS documents dead-letter queues with a redrive policy and maxReceiveCount, and cautions that a dead-letter queue can affect ordering in a FIFO workload. Those details point to a general operating question: who examines the item, changes the data or code when needed, and replays it using the original work ID?

The recovery loop should have explicit states: pending, claimed, retry-waiting, completed, and terminal. The last state should include a concise failure category, the attempt count, the original payload reference, and a replay action that does not mint a new idempotency key. Don't make operators reconstruct state from scattered logs.

For an LLM feature, a duplicate is also a cost event. One repeated inference may charge twice, create two user-visible artifacts, or overwrite an earlier result. Track it. The basic dashboard can stay small:

Before adopting this shape, run a small failure drill: submit the same webhook twice, interrupt a worker after the side effect boundary, return a 429 with Retry-After, then replay a terminal item. Measure queue delay, attempts, and the time needed to explain every state transition. Those numbers are more useful than a feature matrix because they expose the work the team will own.

Keep it boring.

Further reading

• https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html
• https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429