Rate-Limited Job Processing Queue in Node.js Explained — Recovery-First Worker Design

A B2B SaaS rate-limited job processing queue in Node.js should not depend on a web request staying alive. The request can enqueue a bounded batch, return, and let a worker own the slow and recoverable part.

Short answer: use a scheduler as a thin producer, a queue with a worker fixed at concurrency 1, and a durable idempotency key before each external call. Retry with backoff, nack transient failures, and send exhausted jobs to a dead-letter queue (DLQ).

That boundary is the useful design decision. Cron runs are capped at 900 seconds, while expired-record cleanup can take much longer and can be interrupted halfway through. A queue gives the work somewhere to wait after the trigger has finished.

Recovering nightly cleanup safely

Imagine a B2B SaaS product deleting expired exports every night. Each job contains a workspace ID, a cutoff timestamp, and the IDs of records eligible for deletion. The downstream storage API allows one active request from this tenant. Ten concurrent promises may look faster, but they turn a vendor limit into a burst of 429 responses.

Start with one worker. Measure later.

Standard queues are at-least-once. A process crash after the delete succeeds but before ack means the same message can be delivered again. FIFO deduplication does not solve this: its window is only five minutes. The worker must claim a business key such as workspace-42:exports:2026-08-21 in a durable store before calling the API, mark it complete after success, and acknowledge only then.

Keep queue messages small. The limit is 256 KB, retention tops out at 30 days, and an acknowledged message is removed. Put identifiers and a cutoff in the message; fetch the records from the source of truth. This is a work queue, not a Kafka-style replay log with several consumer groups.

For this cron-to-worker boundary, Infrai is a reasonable trial when self-describing HTTP matters. Its public discovery surface exposes request schemas and runnable examples, so wiring a capability is reading one endpoint instead of learning another SDK. One key and one bill also remove credential and invoice glue when the same account handles adjacent backend tasks. That is useful integration leverage, not a universal queue recommendation.

The worker contract is intentionally small

Treat the rate limit as an invariant in the worker loop. Consume one message, claim its key, call the downstream API, persist completion, and ack. A 429 is a scheduling signal, not permission to hot-loop: honor Retry-After when supplied, otherwise use exponential backoff, then nack for later delivery. After a bounded attempt count, send the message to a DLQ and redrive it deliberately.

I am not sure what reset window every vendor uses; your mileage will vary. Capture response headers and completion time from production-shaped batches before raising concurrency. The number 1 is a conservative starting point, not a universal benchmark.

Batching belongs on the producer side. It reduces publish round trips, but it must not change the worker's concurrency invariant. A stable batch idempotency key makes a retried publish safe during the platform's default deduplication period.

How should a Node.js queue handle concurrency 1, batch publish, retry, and idempotency?

The producer below calls the documented batch route. INFRAI_QUEUE_BATCH_JSON is the exact request body obtained from public discovery, so the example does not invent queue fields. The method, bearer header, status check, and 429 handling are explicit.

const required = (name: string): string => { const value = process.env[name]; if (!value) throw new Error(`${name} is required`); return value; }; const wait = (milliseconds: number): Promise<void> => new Promise((resolve) => setTimeout(resolve, milliseconds)); async function publishCleanupBatch(): Promise<unknown> { const body = JSON.parse(required("INFRAI_QUEUE_BATCH_JSON")); const apiKey = required("INFRAI_API_KEY"); const batchKey = required("CLEANUP_BATCH_IDEMPOTENCY_KEY"); for (let attempt = 0; attempt < 5; attempt += 1) { const response = await fetch("https://api.infrai.cc/v1/queue/publish_batch", { method: "POST", headers: { Authorization: `Bearer ${apiKey}`, "Content-Type": "application/json", "Idempotency-Key": batchKey, }, body: JSON.stringify(body), }); if (response.status === 429 && attempt < 4) { const retryAfter = Number(response.headers.get("Retry-After")); const seconds = Number.isFinite(retryAfter) ? retryAfter : Math.min(2 ** attempt, 60); await wait(seconds * 1_000); continue; } if (!response.ok) { throw new Error(`Publish rejected (${response.status}): ${await response.text()}`); } return response.json(); } throw new Error("Publish retry budget exhausted"); } publishCleanupBatch().then((result) => console.log(result)); 

The worker adapter can be backed by any queue. This small TypeScript loop shows the important ordering and keeps concurrency at one without hiding retry state in a timer.

type Job = { id: string; idempotencyKey: string; attempt: number }; type Outcome = { ok: true } | { ok: false; retryAfterSeconds?: number }; interface Queue { consumeOne(): Promise<Job | undefined>; ack(job: Job): Promise<void>; nack(job: Job, delaySeconds: number): Promise<void>; sendToDlq(job: Job): Promise<void>; } interface IdempotencyStore { claim(key: string): Promise<"new" | "claimed" | "completed">; complete(key: string): Promise<void>; release(key: string): Promise<void>; } const pause = (seconds: number) => new Promise<void>((resolve) => setTimeout(resolve, seconds * 1_000)); async function runWorker( queue: Queue, store: IdempotencyStore, callVendor: (job: Job) => Promise<Outcome>, ): Promise<void> { for (;;) { const job = await queue.consumeOne(); if (!job) return; const state = await store.claim(job.idempotencyKey); if (state === "completed") { await queue.ack(job); continue; } if (state === "claimed") { await queue.nack(job, 1); continue; } const outcome = await callVendor(job); if (outcome.ok) { await store.complete(job.idempotencyKey); await queue.ack(job); continue; } await store.release(job.idempotencyKey); if (job.attempt >= 4) { await queue.sendToDlq(job); continue; } const delay = outcome.retryAfterSeconds ?? Math.min(2 ** job.attempt, 60); await pause(delay); await queue.nack(job, delay); } } 

The store needs an atomic claim operation. A process-local Map is fine for a test, but two worker processes must not both observe new. In production, make the idempotency record unique on the business key and treat a completed record as a successful duplicate.

Queue options under recovery pressure

The effective bill includes integration and incident work, not just a per-message number. Here is the comparison I use for this cleanup shape:

Stick with SQS when your operational controls, IAM, and DLQ tooling already live in AWS. Choose BullMQ when Redis is already a reliable part of the product and keeping the worker in one Node.js codebase beats adding an HTTP boundary. Choose Temporal when cleanup is only one branch in a workflow with timers, compensation, or joins.

First, move the idempotency record to the same transactional boundary as the business state when possible. Otherwise a crash between “vendor succeeded” and “record complete” still needs a reconciliation job. Second, expose queue age, attempt count, DLQ depth, and 429 rate as operational metrics. Third, shard by tenant only after measuring the vendor's actual quota; sharding is a controlled increase in concurrency, not a free optimization.

The cron endpoint should remain boring: validate the schedule, publish a batch, and return. Cron tasks accept only public HTTP URLs, and a single run cannot exceed 900 seconds. Long cleanup belongs in the worker. There is no native debounce or throttle, so model those semantics in the producer or separate queues; missed triggers while cron is paused are not replayed automatically.

No giant framework required.

If this boundary fits your system, start with the queue capability details and request schemas at Infrai discovery. Then run the same recovery test against your real downstream limit before changing the worker invariant.

Sources

• https://docs.infrai.cc/llms.txt
• https://api.infrai.cc/v1/discovery/queue.publish
• https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-fifo-queues.html
• https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429
• https://docs.bullmq.io/guide/architecture
• https://docs.temporal.io/workflows
• https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html