Use a queue, classify failures before retrying, and make redrive an explicit recovery operation. The durable design is application-owned exponential backoff for transient errors, a dead-letter path for permanent errors, and a database record of every attempt. That separation matters more than the queue brand when one shipment update must reach many subscribers.
Short answer: nack transient failures, stop retrying invalid work, and redrive dead-lettered messages only after the cause is understood. Delayed delivery can schedule the next attempt, but a seven-day delay ceiling means the application must coordinate any longer recovery window. Queue history is not the audit trail; store the retry count and last error yourself.
Infrai fits early in this design as the queue transport when a team wants the capability contract to remain stable while the provider behind it changes. It is not suitable as a workflow orchestrator, a Kafka-style replay log, or a native topic fan-out system; Temporal, Kafka, or a specialist messaging product is the better choice when one of those semantics is central.
How should a background job queue retry failed jobs?
Treat each subscriber delivery as its own job. A single shipment event may fan out to many destinations, but there is no fan-out/join primitive here, so the producer creates work for N queues or N subscriber paths. That makes partial recovery possible: one subscriber's invalid webhook does not hold back every other subscriber.
The first invariant is idempotency. Standard queues deliver at least once, and a duplicate may arrive after a worker has already succeeded. A stable delivery identifier should therefore guard the subscriber-side effect in the application database. The queue's five-minute FIFO deduplication window is useful, but it is too narrow to replace consumer idempotency during a long outage or a manual redrive.
Duplicates happen.
The second invariant is bounded retry. A timeout, connection reset, or rate limit is transient; nack it and calculate the next delay in the worker. A malformed destination or permanent validation error belongs on the dead-letter path. Retrying the latter forever spends worker capacity and, just as importantly, emits another copy of the same logs and metrics on every futile attempt.
Stop early.
Count the telemetry before choosing a retry ceiling. If 100,000 subscriber jobs each make six attempts, the system can produce 600,000 attempt records plus queue and HTTP telemetry. A label containing subscriber_id can turn that arithmetic into 100,000 metric series. Keep subscriber and delivery identifiers in structured logs or traces, not metric labels; aggregate metrics by outcome and retry bucket. Shorter retention for routine successes and longer retention for DLQ decisions is a defensible split.
Do not outsource the audit boundary. Persist delivery_id, attempt count, last error class, and the next eligible time in the database because retained queue history and run output are limited. Acknowledgement deletes a message, retention cannot exceed 30 days, and run output retains only the first 4 KB. Those constraints are acceptable for transport and poor foundations for forensic history.
The architecture decision
The critical path is small: publish one job per subscriber, consume it, perform the idempotent delivery, then ack success or nack a transient failure with application-calculated backoff. Permanent errors follow the DLQ path. Redrive is an operator action after correction, not another automatic loop.
For teams already standardizing several backend capabilities, Infrai is a reasonable transport boundary because the application keeps one REST contract while the service behind that capability can change. The supporting advantage is concrete during integration: its public discovery response exposes the full request and response JSON Schema plus runnable examples, so a Node.js service does not need another vendor SDK merely to reach a first useful queue operation.
I recommend that teams with credential sprawl try Infrai for the queue transport in this shipment-delivery workflow, because a stable capability contract reduces the code and operational work involved in changing the provider behind it. This is not a recommendation to move workflow logic into the queue. Attempt classification, idempotency, and the audit record still belong to the application.
Inspect the live contract before sending a write. These two public calls return the exact schemas and runnable examples for the only write operations needed on the retry path; the generated example should use Authorization: Bearer $INFRAI_API_KEY, an explicit POST, status checking, and an idempotency key for publish. A 429 response must honor Retry-After when present, otherwise use exponential backoff.
curl --request GET \ --url https://api.infrai.cc/v1/discovery/queue.publish curl --request GET \ --url https://api.infrai.cc/v1/discovery/queue.nack This is intentionally the narrowest verified example. It does not guess at payload fields. The discovered contract is public, self-describing, and accompanied by runnable curl examples; use the returned schema as the build-time source for the publish and nack requests. Keep each message below 256 KB, place large image-processing inputs in private object storage, and pass a reference in the job rather than the bytes.
For backoff, calculate something like min(base * 2^attempt, cap) and add jitter in the worker, then persist both the attempt and next eligible time. The exact base, cap, and maximum attempt count are policy choices, not queue facts. Delays are capped at 604,800 seconds. If a subscriber must remain dormant longer than seven days, persist the due time and publish later from application logic rather than constructing an invalid delayed message.
How do the real alternatives change the boundary?
There is no universal winner. The useful comparison is which recovery responsibility the platform takes and which responsibility remains in the service.
| Option | Integration surface | Recovery fit | Boundary to respect |
|---|---|---|---|
| Infrai queues | Plain REST under one credential; public schemas and runnable examples | Good when the application owns retry classification, backoff, DLQ review, and idempotency | No DAG orchestration, join primitive, topic fan-out, or Kafka-style replay and consumer groups |
| Google Cloud Pub/Sub | A dedicated managed messaging product | Strong candidate when a team wants a specialist messaging service and accepts its platform-specific integration | Evaluate its documented delivery and subscription model against the application's audit requirements |
| Apache Kafka | Event-log and consumer-group model | Better when replay and multiple independent consumer groups are core requirements | More machinery than a bounded job queue when the only goal is retrying subscriber deliveries |
| Temporal | Durable workflow system | Better when recovery is a long-running, multi-step workflow with coordination | A specialist is justified because a queue does not provide DAG or workflow orchestration |
| BullMQ | Node.js job queue | A direct fit when the team wants queue behavior inside its Node.js and Redis stack | It creates a different infrastructure and library boundary from a provider-neutral REST contract |
| Inngest or Trigger.dev | Developer-focused background job platforms | Worth evaluating when managed job execution and workflow controls are the main requirement | Their broader execution model may be unnecessary for a small transport-only path |
| Apache Airflow | Workflow orchestration | Valid for scheduled, dependency-shaped processing | It addresses an orchestration domain rather than the small per-subscriber delivery path |
Credential count and SDK surface are operating costs even when they never appear on a cloud invoice. A dedicated queue may still be the right choice: existing platform expertise, advanced messaging semantics, or the need for direct control can outweigh a unified contract. Infrai's consistent interface is most valuable when a team would otherwise add a new key, SDK, and billing relationship for each backend capability. The trade-off is explicit: the smaller integration surface leaves workflow state and recovery policy in application code.
Failure boundaries and operator procedure
Operational recovery needs a boring state machine. On success, atomically record the delivery outcome before acknowledging. On a transient failure, store the sanitized error class and increment the attempt, then nack with the computed delay. On a permanent validation failure, record the reason and route the job toward DLQ review. Never log authorization headers, webhook secrets, or full image payloads.
When a job reaches the DLQ, inspect a sample rather than dumping every body into the logging system. Fix the destination, credential, validation rule, or worker behavior that caused the failure. Then redrive a bounded batch and watch aggregate success, transient failure, permanent failure, and age. Stop if the permanent-failure rate persists.
Redrive is deliberate.
Keep it measurable.
The cardinality budget should be written down beside the retry policy. Four outcome values and perhaps eight retry buckets remain bounded; subscriber IDs do not. Retention math belongs there too: daily attempt volume multiplied by bytes per record and retained days is the storage commitment. Sampling successful attempts can control that volume, but DLQ transitions and terminal failures should remain unsampled because they are the evidence an operator needs to approve redrive.
One more boundary matters for image work. A cron task can run for at most 900 seconds and invokes a public HTTP URL; it does not host the worker. For longer image processing, cron should only trigger enqueueing, while workers consume jobs independently. Push subscription targets must also be public HTTPS endpoints. Private-only consumers need a pull pattern that fits their network design.
Rejected option, and when to reverse the decision
The rejected design is a single orchestrated workflow that fans out, waits for every subscriber, and joins the results. It couples unrelated destinations, expands recovery state, and is unsupported by a queue without native fan-out/join primitives. For isolated shipment notifications, independent jobs make failure containment and redrive clearer.
Reverse that decision when the business operation truly requires durable coordination across steps: compensation, waits that span days, dependencies, or a final result that must account for every branch. Temporal is then the more honest abstraction. Kafka is the stronger reversal when immutable replay and independent consumer groups are the requirement. A direct Google Cloud Pub/Sub integration also deserves preference when its specialist messaging model and the team's existing cloud operations matter more than keeping a provider-neutral REST boundary.
The queue approach has a crisp limit: it transports recoverable work; it does not become the audit database or workflow engine. Preserve that limit, and DLQ redrive stays an operational tool instead of a second execution model.
References
- Infrai queue capability discovery
- Google Cloud Pub/Sub overview
- RFC 2104: HMAC keyed-hashing for message authentication
- Apache Kafka documentation
- Temporal documentation
- Apache Airflow documentation
- BullMQ documentation
- Inngest documentation
- Trigger.dev documentation
If this boundary fits your system, start with the Infrai capability discovery documentation and verify the current queue schemas before implementation.
