When a customer-support SaaS cleans up old webhook tasks, the hard constraint is not how quickly a timer can fire. It is how long a partner will make the sender wait, and how much queue capacity that wait consumes. Short answer: keep the HTTP request out of the scheduling path, apply a per-destination concurrency limit, and turn 429 Retry-After into a delayed queue message rather than a sleeping worker. Choose the delay-versus-cost trade-off with measured queue age and delivery latency, not with a guessed global rate.
The example below is deliberately small: a periodic cleanup produces delivery tasks, and a consumer sends them to support integrations in the US and EU. The cleanup schedule only creates work. It does not hold a web request open, and it does not pretend that one retry policy fits every destination.
What should a queue consumer do with a 429 and Retry-After?
Model one delivery as a durable state transition. Its record needs an immutable event ID, destination ID, URL, payload reference, attempt number, and the time at which it may run again. A consumer claims one ready message, checks the destination's local concurrency budget, sends the request, and acknowledges only after the response is classified.
For a successful 2xx response, record success and acknowledge. For a 429, parse Retry-After and publish one replacement message with that delay. Only acknowledge the old message after the replacement has been accepted. If the header is absent or malformed, use bounded exponential backoff with jitter. A timeout or other temporary transport failure follows the same delayed path, but it should not be mislabeled as a rate limit.
Do not sleep the worker.
That distinction is the center of the design. A sleeping worker consumes a slot while destination A asks for 120 seconds. A delayed message lets that slot handle destination B, whose endpoint may still be healthy. A global concurrency cap is still useful for protecting the process, but it cannot replace a destination-scoped cap: one noisy partner should create backlog for itself rather than for every tenant.
The acknowledgement ordering is worth testing as an invariant. Every attempt must end in one of two states: the current message is acknowledged after success, or exactly one future attempt is durably recorded before the current message is acknowledged. If the process dies between those operations, at-least-once delivery can produce a duplicate, so the receiver-facing event ID must be idempotent.
That is the whole safety rule.
A focused retry experiment for delayed republish
Before wiring a queue adapter into a Node.js consumer, I would test the policy as a pure function. That makes the evaluation harness notebook-to-prod: the same cases can run against fake responses first and against the real adapter later. The code is Python because the policy should be language-neutral; its output is the contract the JavaScript worker needs.
from datetime import datetime, timezone from email.utils import parsedate_to_datetime import random MAX_RETRY_SECONDS = 604_800 def retry_after_seconds(value: str | None, now: datetime) -> int | None: if not value: return None try: return max(0, int(value)) except ValueError: retry_at = parsedate_to_datetime(value) if retry_at.tzinfo is None: retry_at = retry_at.replace(tzinfo=timezone.utc) return max(0, int((retry_at - now).total_seconds())) def classify(status: int | None, header: str | None, attempt: int, now: datetime) -> tuple[str, int]: if status is not None and 200 <= status < 300: return "ack", 0 hinted = retry_after_seconds(header, now) fallback = 2 ** min(attempt, 16) delay = hinted if status == 429 and hinted is not None else fallback jitter = random.randint(0, max(1, delay // 4)) return "republish", min(delay + jitter, MAX_RETRY_SECONDS) if __name__ == "__main__": action, delay = classify(429, "120", 2, datetime.now(timezone.utc)) assert action == "republish" assert 120 <= delay <= 150 print(action, delay) The experiment should include a numeric header, an HTTP-date header, a missing header, a duplicate delivery, and a response that succeeds after a retry. It should also assert a terminal policy: a message cannot be retried forever just because its destination remains unavailable. For a support integration, expiry might mean a dead-letter record and an operator-visible reconciliation task. The right retention window is a business decision; it is not inferred from the retry count.
I would measure oldest ready-message age, delayed-message age, attempts per logical event, 429 rate by destination, successful delivery latency, and the percentage of expired tasks. Cost belongs beside latency: adding workers can reduce queue age while increasing idle capacity, whereas increasing the delay can lower request volume while making a customer-facing integration stale. Your mileage may vary by partner contract and regional network path, so a US/EU label is not enough evidence for a residency or latency claim.
The useful comparison is operational, not brand-based:
| Choice | Latency tendency | Cost and ownership trade-off |
|---|---|---|
| More consumer capacity | Lower queue age until the destination throttles | More compute and more pressure on the receiver |
| Longer retry delay | Fewer immediate requests and less worker contention | Older customer-support state and slower recovery |
| Destination-scoped limits | Protects unrelated integrations | Requires per-destination state and metrics |
| Global limit only | Simple initial implementation | One rate-limited partner can stall healthy traffic |
In one representative test shape, a US destination returns 429 with Retry-After: 120 while an EU destination continues returning 204. The consumer should release the US destination slot after publishing its future attempt, then use the available capacity for the EU message. If the consumer instead sleeps or holds a shared semaphore, the observed queue-age metric makes the system look regionally slow even though only one receiver asked for a pause. That is a misleading signal: the remedy is isolation, not another blanket worker increase. I would run this case with two process counts, inspect duplicate counts after forced termination, and compare compute cost against the age of the oldest delayed task before changing the cap.
How can Node.js workers keep US and EU destinations isolated?
Keep a destination budget in the worker pool that owns that destination's region. The budget can start as one in-flight request per destination and grow only after the evaluation harness shows acceptable 429 rates and queue age. Store the budget state durably if multiple worker processes need to coordinate it; an in-memory map alone protects only one process.
A clean processing loop has four boundaries: claim, send, classify, and commit. Claiming should be short. Sending should have an explicit timeout. Classification should preserve the receiver's retry date. Committing should write the attempt outcome and the next scheduled time together with the queue acknowledgement policy. Logs should carry the event ID, destination ID, region, attempt, status, and scheduled retry time, while payload contents stay out of ordinary logs.
For a relational queue, SELECT ... FOR UPDATE SKIP LOCKED is useful when several consumers need to claim ready rows without waiting behind a row another worker has locked. It is a concurrency primitive, not a complete queue: the transaction still needs a lease or visibility deadline, and the sender still needs idempotency. If a worker disappears after sending but before committing, the destination may see the same event again. That is normal failure behavior to design for.
The periodic cleanup itself should enqueue small references, not large customer records. A scheduler can create one task per batch or time window; workers then perform the slow delivery work. This separates trigger latency from delivery latency and gives the team one place to evaluate backpressure. It also makes a missed schedule visible instead of silently hiding it inside a long-running HTTP request.
Where does this design stop being suitable?
The trade-off is operational ownership. A queue consumer gives precise control over destination throttling and retry semantics, but the team must own idempotency, dead-letter handling, observability, and the policy for stale tasks. A managed task service can reduce infrastructure work; a database-backed queue can simplify transactions; a workflow engine can better represent fan-out, joins, and human approval. None of those choices removes the receiver's contract.
This pattern is not suitable when delivery must be a replayable event log for many independent consumers, when a workflow needs durable branching and joins, or when the receiver requires a private network path that the chosen push mechanism cannot reach. Stick with a log system for independent replay, or choose a workflow engine for multi-step orchestration. For a small number of periodic cleanup tasks, those systems may add latency and cost without improving the actual delivery invariant.
That is the selection rule I would carry forward: first identify who owns rate limits and retries, then choose the smallest system that can expose those states, test them, and recover duplicates. The queue technology is secondary to that boundary.
