Short answer: to create a Node.js background job queue for shipment updates, give each subscriber a queue delivery, acknowledge only after the update is durably accepted, retry only transient failures, and make the worker idempotent because fan-out delivery is at-least-once in the failure cases that matter.
That decision is more important than the queue client. A shipment update may need to reach a carrier portal, a warehouse dashboard, a customer notification service, and an internal audit stream. Those destinations do not share latency, availability, or retry behavior. Treating the fan-out as one large request makes the slowest subscriber define the whole operation; treating it as independent jobs gives each delivery its own attempt history and audit trail.
The ledger-minded rule is simple: a message is not complete because a worker received it, and it is not complete because an HTTP call returned. It is complete when the business result and the idempotency record have crossed the same durable boundary. That is the exactly-once mindset applied honestly: the transport may deliver more than once, while the business effect is allowed once.
That is the trap.
Which shipment fields belong in the idempotency record?
First, give every fan-out job a stable event ID and subscriber ID. Together they form the idempotency key. shipment-8842:warehouse-dashboard and shipment-8842:customer-notification are different effects even though they came from the same shipment update. Include the event version as well, because an older retry must not overwrite a newer state.
Second, consume the message and validate its shape before doing external work. A malformed payload is not a transient network failure. It belongs in a dead letter queue with enough metadata for an operator to understand what happened, while a timeout or an HTTP 429 is a candidate for delayed retry. MDN documents 429 as a rate-limit response and describes Retry-After as the server's guidance for when to try again; the worker should respect that signal rather than converting backpressure into a request storm.
For a shipment update, the minimum ledger is small enough to inspect in one screen:
| Field | Why it exists |
|---|---|
| event ID and version | Prevents an old retry from replacing newer state |
| subscriber ID | Separates warehouse, carrier, customer, and audit effects |
| attempt and timestamps | Makes retry age and operator action auditable |
| outcome and error class | Distinguishes accepted, transient, and permanent results |
This record also answers a question that queue depth cannot: did the carrier receive the update, or did a worker merely remove it from the queue? A useful operational dashboard compares durable outcomes with acknowledgements, because an early acknowledgement can make loss look like healthy throughput.
Third, record the outcome and acknowledge the delivery only after the outcome is durable. RabbitMQ's acknowledgement model makes this ordering explicit: acknowledgement removes the delivery from the broker's responsibility, so acknowledging before the side effect creates silent loss when the process exits immediately afterward. A negative acknowledgement is a statement that this attempt did not finish, not permission to retry every error forever.
Here is a deliberately small Go model of the consumer state machine. The production adapter can be written in Node.js, but the invariant is independent of the language: duplicate deliveries consult durable state, transient failures return to controlled retry, and permanent failures become inspectable dead letters. Keep it boring. The useful property is that a reviewer can trace every branch from a delivery to an acknowledgement, a retry, or a dead letter without mentally simulating a framework.
package main import "fmt" type Delivery struct { EventID string Subscriber string ShipmentID string Version int Attempts int } type Store struct { completed map[string]bool dead []Delivery } func key(d Delivery) string { return d.EventID + ":" + d.Subscriber } func consume(d Delivery, store *Store, maxAttempts int, send func(Delivery) error) string { if store.completed[key(d)] { return "ack-duplicate" } if err := send(d); err != nil { d.Attempts++ if d.Attempts >= maxAttempts { store.dead = append(store.dead, d) return "dead-letter" } return "nack-retry" } // In production, this write belongs in the same durable boundary as the result. store.completed[key(d)] = true return "ack" } func main() { store := Store{completed: map[string]bool{}} delivery := Delivery{ EventID: "shipment-8842-v7", Subscriber: "warehouse-dashboard", ShipmentID: "8842", Version: 7, } result := consume(delivery, &store, 3, func(Delivery) error { return nil }) duplicate := consume(delivery, &store, 3, func(Delivery) error { panic("the external effect must not run twice") }) fmt.Println(result, duplicate, len(store.dead)) } The example's map stands for a database table with a unique constraint, not for a process-local cache. A restart must not erase completion state. The external call deserves the same care: if it cannot accept an idempotency key, an application cannot generally make the combined network call and local database write atomic. That is a limitation of the boundary, not a reason to promise exactly-once transport.
How does a consumer classify ack, nack, retry, and dead-letter outcomes?
Define the failure classes before writing the handler. A validation error, an unknown subscriber, or an unsupported event version is permanent until someone changes data or code. Nacking it immediately without a limit creates a poison-message loop. A timeout, temporary connection refusal, or 429 may recover, so the queue should delay the next attempt, honor Retry-After where available, and cap the total number of attempts.
The retry record should include the reason, attempt number, first-seen time, and next-eligible time. Those fields turn an incident from guesswork into an audit trail. They also make a useful distinction visible: a subscriber that is slow is a capacity problem, while a subscriber that rejects version 7 is a compatibility problem. Both may produce retries, but they need different remediation.
Do not use the dead letter queue as a trash can. Preserve the original event ID, shipment ID, subscriber, version, error class, and timestamps. A redrive operation must create an auditable new attempt and must still pass through the idempotency check. If an operator replays a message after the first attempt actually succeeded but before its acknowledgement was observed, the consumer should acknowledge the duplicate without sending the update again.
This is where delivery guarantees become operational rather than semantic. Measure duplicate deliveries, retry counts by reason, dead-letter age, acknowledgement latency, and the number of successful business effects. A low queue depth can still hide loss if the worker acknowledges too early; a high depth can be healthy during a carrier outage if the retry budget and retention policy are explicit.
How can the Go worker model expose the delivery guarantee?
Use one independent work item per subscriber when each destination has its own delivery guarantee. A shared queue with competing consumers is appropriate when any worker may perform the job. Separate queues are clearer when warehouse updates, customer notifications, and carrier synchronization need different retry schedules or access controls. A broadcast mechanism is appropriate when every consumer must see the same event, but it does not remove the need for per-consumer idempotency.
The trade-off is coordination. With independent jobs, the producer can report that fan-out was accepted while individual deliveries remain pending; a later read model must show partial completion. With a single synchronous request, the caller sees one result, but a slow or failed destination couples unrelated work. Neither model is universally correct. Choose the queue when partial progress is acceptable and each effect can be retried safely; keep the synchronous path when the caller truly requires an immediate, all-destinations response.
Consider one shipment event with four subscribers. The warehouse dashboard may accept version 7, the carrier endpoint may be waiting behind a rate limit, the customer notification may be dead-lettered because its payload is invalid, and the audit stream may already contain the event. The shipment itself is not in one simple “sent” or “failed” state; it has four delivery records with different evidence. A useful read model can say exactly that: warehouse accepted, carrier retry scheduled, customer blocked, audit accepted. An operator can fix the customer payload and redrive only that delivery, while the idempotency key prevents the warehouse update from being sent again. If the application instead stores only a boolean named fanout_done, it has thrown away the distinction needed for reconciliation, and a later retry will either duplicate successful effects or conceal the subscriber that never received the update. This is why I prefer a delivery ledger even when the queue API makes a smaller implementation look attractive: it keeps transport state, business state, and operator state separate enough to audit.
A queue is also a poor substitute for a workflow engine when shipment handling has mandatory ordered stages, compensation, human approval, or a long-lived join. In those cases, the durable state machine is the primary system and the queue is a transport detail. Your mileage may vary with the number of subscribers and the required audit retention; write those constraints down before selecting a broker.
Why is fan-out an integration boundary rather than one request?
Start with a contract test that publishes the same shipment event twice, terminates a worker immediately before acknowledgement, injects a timeout, returns a 429 with Retry-After, and submits an invalid event version. The expected result is one business effect, a later successful retry for the timeout, bounded handling of the rate limit, and one inspectable dead letter for the invalid event. Test the race, not only the happy path.
Before rollout, test the accounting as well as the transport: the number of accepted delivery records, successful subscriber effects, retry reasons, and dead letters should reconcile for a fixed event set. That check is especially valuable during migration because a consumer can appear healthy while silently dropping an old event version or acknowledging before its write commits.
The difficult case deserves a longer rehearsal. Imagine the warehouse dashboard accepts the update, the worker loses its connection before receiving the response, and the broker redelivers the message after the process restarts. The second worker cannot infer correctness from the missing response, so it must send with the same idempotency key or consult a durable result record that proves the first effect was committed. If the destination has neither property, the application must represent the ambiguity and alert an operator; silently acknowledging risks loss, while blindly retrying risks a duplicate notification. That is a genuine boundary of the system, and no choice of ack versus nack can remove it.
Deploy the consumer with a bounded concurrency setting and a shutdown path that stops new deliveries, finishes or requeues in-flight work according to the broker contract, and records the reason for every negative acknowledgement. Roll out one subscriber first, compare sent effects with the durable delivery ledger, then expand the fan-out. Keep the migration reversible: a new worker should be able to recognize old event versions, and a redrive should never bypass validation or idempotency.
The final decision rule is narrow. A background queue is a good fit for shipment fan-out when delivery can be independent, retries are bounded, and duplicate execution is harmless by construction. It is not suitable as the sole source of truth for shipment state, and it cannot manufacture atomicity around an external system that has no idempotency contract.
References
- RabbitMQ consumer acknowledgements: https://www.rabbitmq.com/docs/confirms
- MDN, HTTP 429 Too Many Requests: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429