Your deploy stops all ten workers, ships, and starts them again. While they were down, 8,000 jobs piled up in a classic queue. The first worker back opens a channel, starts consuming, and the broker pushes it the whole backlog. The other nine come up a few seconds later and find nothing to take. That first worker happens to sit on a busy node and runs at 300 ms a job, so it will need 40 minutes to get through what it holds.

The management UI shows 0 ready and 8,000 unacked. Nothing is stuck in the queue; everything is stuck in one process. On RabbitMQ 4.3 or later, no consumer timeout will take those messages back either, because classic queues no longer evaluate consumer timeouts.

The cause is one missing basic.qos call.

Unlimited is the default

RabbitMQ caps how many unacknowledged deliveries a consumer can hold through the prefetch count set by basic.qos. The consumer prefetch guide says: "A value of 0 is treated as infinite, allowing any number of unacknowledged messages." If your client never sends basic.qos, the broker applies default_consumer_prefetch, and in the RabbitMQ server source that default is {false, 0}: per-consumer, zero, unlimited.

Combine that with how dispatch works. Active consumers on a queue receive messages round-robin, but only consumers with spare capacity are eligible, and an unlimited consumer always has spare capacity. When a backlog exists and one consumer attaches, it is the only eligible consumer, so it gets everything. Even when every worker is connected before the jobs arrive, round-robin hands the slow one its full share up front, and the fast ones go idle once theirs is done.

There is a memory cost too. The acknowledgements guide warns that "automatic acknowledgement mode or manual acknowledgement mode with unlimited prefetch should be used with care," because consumers that take in many messages without acking them grow memory on the node they are connected to. Quorum queues cap consumer prefetch at 2,000 to limit Raft log growth, per the same guide, and 2,000 is still a lot of messages to park behind one slow handler.

A single worker silhouette buried under a tall stack of envelopes while a second worker stands beside an empty tray
With no prefetch limit, the consumer that attaches first takes the backlog with it.

The demo: one slow worker, one fast one

You need Docker Compose and nothing else. The broker is the official RabbitMQ 4.4 image and the client is amqplib running on Node.js 24 (amqplib needs Node 18 or newer, per its package metadata).

# docker-compose.yml services: rabbitmq: image: rabbitmq:4.4-management environment: RABBITMQ_DEFAULT_USER: demo RABBITMQ_DEFAULT_PASS: demo ports: - "15672:15672" healthcheck: test: ["CMD", "rabbitmq-diagnostics", "-q", "check_port_connectivity"] interval: 5s retries: 20 demo: image: node:24-alpine working_dir: /app volumes: - ./:/app environment: AMQP_URL: amqp://demo:demo@rabbitmq:5672 depends_on: rabbitmq: condition: service_healthy

The script fills a quorum queue with 1,000 jobs, starts a worker that spends 100 ms per job, waits half a second, then starts a worker that spends 10 ms per job. Each worker processes one job at a time, like a handler doing real work would.

// demo.mjs import amqp from "amqplib"; const URL = process.env.AMQP_URL; const QUEUE = "jobs"; const TOTAL = 1000; const PREFETCH = Number(process.argv[2] ?? 0); const TIMEOUT_MS = Number(process.env.TIMEOUT_MS ?? 0); const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); let t0 = 0; async function fill() { const conn = await amqp.connect(URL); const ch = await conn.createConfirmChannel(); await ch.assertQueue(QUEUE, { durable: true, arguments: { "x-queue-type": "quorum" }, }); await ch.purgeQueue(QUEUE); for (let i = 0; i < TOTAL; i++) { ch.sendToQueue(QUEUE, Buffer.from(String(i)), { messageId: `job-${i}`, }); } await ch.waitForConfirms(); await conn.close(); } async function worker(workMs, s, args = {}) { const conn = await amqp.connect(URL); const ch = await conn.createChannel(); ch.on("error", (e) => console.error(e.message)); await ch.prefetch(PREFETCH); let chain = Promise.resolve(); const onMessage = (msg) => { if (msg === null) { s.cancelled = true; // the broker cancelled us return; } if (msg.fields.redelivered) s.redelivered++; s.held++; s.peak = Math.max(s.peak, s.held); chain = chain.then(async () => { if (s.cancelled) return; await sleep(workMs); if (s.cancelled) return; ch.ack(msg); s.held--; s.done++; s.lastAckAt = (Date.now() - t0) / 1000; }); }; await ch.consume(QUEUE, onMessage, { noAck: false, arguments: args, }); return conn; } function stats() { return { done: 0, held: 0, peak: 0, redelivered: 0, lastAckAt: 0, cancelled: false, }; } await fill(); const slow = stats(); const fast = stats(); const slowArgs = TIMEOUT_MS ? { "x-consumer-timeout": TIMEOUT_MS } : {}; t0 = Date.now(); const conns = [await worker(100, slow, slowArgs)]; await sleep(500); // the fast worker comes up late conns.push(await worker(10, fast)); while (slow.done + fast.done < TOTAL) await sleep(50); const secs = ((Date.now() - t0) / 1000).toFixed(1); console.log(`prefetch=${PREFETCH}, all acked after ${secs}s`); console.table({ slow, fast }); for (const c of conns) await c.close();

Run it once per setting:

docker compose up -d rabbitmq docker compose run --rm demo npm install amqplib@2.2.1 docker compose run --rm demo node demo.mjs 0 docker compose run --rm demo node demo.mjs 1 docker compose run --rm demo node demo.mjs 10 docker compose run --rm demo node demo.mjs 50

ch.prefetch(0) sends the same "no limit" the broker would assume anyway; amqplib documents that a falsey count means no limit. The peak column is the number to watch: it is how many unacked messages the worker held at its worst.

One Node-specific trap is visible in the code. Delete the chain and call the handler directly, and prefetch stops being a buffer size and becomes your concurrency limit. At prefetch 0, that is 1,000 handlers running at once, which in a real service means 1,000 open database queries.

What the numbers should look like

I haven't run this demo, so here is the arithmetic for what it should print, rounded. The slow worker clears 10 jobs a second and the fast one clears 100. The queue empties once the slow worker's acks plus its buffer plus the fast worker's acks reach 1,000. After that, the fast worker sits idle while the slow one works through whatever it still holds.

Prefetch Slow handles Fast handles Slow peak unacked Last ack
0 ~1,000 ~0 ~1,000 ~100 s
1 ~95 ~905 1 ~9.5 s
10 ~105 ~895 10 ~10.5 s
50 ~140 ~860 50 ~14 s

Your run will move around these with connection setup time and timer jitter. At 0, the slow worker takes the whole backlog before its sibling has finished connecting, and the run takes ten times longer than it needs to. At 50, the last five seconds are the fast worker idling while 50 jobs wait behind a 100 ms handler.

Prefetch 1 looks like the winner here, and that is the demo flattering it. Both containers sit on one Docker network, so the round trip between an ack and the next delivery costs almost nothing. Over a real network, a worker at prefetch 1 waits a full round trip after every ack before it gets its next job, and the RabbitMQ docs say prefetch 1 "will significantly reduce throughput" (acknowledgements guide).

Picking the number

The docs give a starting range, "Values in the 100 through 300 range usually offer optimal throughput and do not run significant risk of overwhelming consumers," and add that finding the right value "is a matter of trial and error" (acknowledgements guide). That range fits fast handlers. For slower ones, reason from two sides.

The floor is keeping the worker busy. You want enough messages buffered to cover one network round trip. A rough rule: divide the round trip by the handler time and add one. A 2 ms handler 20 ms away from the broker wants about 11. A 500 ms handler wants 1 or 2, because the round trip is noise next to the work.

The ceiling is how much work you are willing to strand. Multiply prefetch by the handler time. That product is how long a message can wait behind a slow or dying worker while its siblings sit idle, and it is how long that worker needs to drain on a graceful shutdown. If the answer is longer than your latency budget, the number is too high.

Then check that product against your consumer timeout, which the next section covers. The timeout counts how long a consumer holds a delivery, and a message sitting in your client's buffer counts as held, even if no code has touched it yet.

consumer_timeout only guards quorum queues now

RabbitMQ's consumer timeout limits how long a consumer can keep a delivery unacknowledged. The default consumer_timeout is 1,800,000 ms, which is 30 minutes, and you can override it per queue with the consumer-timeout policy key or the x-consumer-timeout queue argument.

RabbitMQ 4.3, released April 23, 2026, changed where this lives. Timeout handling moved into quorum queues (and the new JMS queues), and classic queues and streams stopped evaluating consumer timeouts. On a 4.3+ broker, a classic-queue consumer with unlimited prefetch and a stuck handler can hold its messages until its connection dies. Before 4.3, a timeout closed the whole AMQP 0-9-1 channel, taking down every consumer on it. Now, if the client supports consumer cancel notifications, the broker cancels only the timed-out consumer and returns its messages to the queue; clients without that support still get the channel closed with PRECONDITION_FAILED (consumers guide).

amqplib advertises consumer_cancel_notify in its connection capabilities, and its docs say a server-side cancel invokes your consume callback with null. That is the msg === null branch in the demo. To try it, give the slow worker a per-consumer timeout, which the quorum queue guide documents as the x-consumer-timeout consumer argument:

docker compose run --rm -e TIMEOUT_MS=10000 demo \ node demo.mjs 0

By the docs, the slow worker should get cancelled around the 10-second mark, with roughly 100 jobs done, and the fast worker should pick up the rest. Any job the slow worker had started is now in someone else's hands too. Your handler keeps running after the cancel. The broker has just stopped waiting for its ack. The 4.3 release post adds one more detail: a consumer timeout increments a message's acquired-count, not its delivery-count, so timeouts do not eat into the quorum queue delivery limit, which defaults to 20 since RabbitMQ 4.0.

A large hourglass tipping messages back into a queue while a worker silhouette keeps working on an envelope
After a timeout, the queue gets the messages back while the old handler is still working.

Redelivery means the handler must be idempotent

When a channel or connection closes with manual acks, "any delivery (message) that was not acked is automatically requeued," and those redeliveries have the redeliver flag set to true (acknowledgements guide), which amqplib exposes as msg.fields.redelivered. The same guide spells out the consequence: "consumers must be prepared to handle redeliveries and otherwise be implemented with idempotence in mind," and "a consumer can receive a message that was previously delivered to another consumer."

Prefetch decides how many messages a crash or a timeout throws back at once. It does not decide whether duplicates happen. A worker can finish the side effect and die before its ack reaches the broker, and the next worker will run the same job again. So key every job and record the key in the same transaction as the side effect. With node-postgres and a processed (id text primary key) table, the insert's rowCount tells you whether this is the first time:

async function handle(msg, client, applyJob) { const id = msg.properties.messageId; await client.query("BEGIN"); try { const { rowCount } = await client.query( "INSERT INTO processed (id) VALUES ($1) " + "ON CONFLICT DO NOTHING", [id], ); if (rowCount === 1) await applyJob(client, msg); await client.query("COMMIT"); } catch (err) { await client.query("ROLLBACK"); throw err; } }

Ack only after COMMIT returns. A duplicate then inserts nothing, skips the side effect, commits, and gets acked. Don't use the redelivered flag as your dedupe check. It tells you the broker has handed this message out before, not whether your side effect ran.

Before the next deploy, find every consume( call in your codebase and check that a prefetch( call runs on the same channel before it.

If this was useful

Prefetch, acks and redelivery are where a queue-based design meets the real world, and most of the pain shows up as duplicates, stranded work or one consumer doing everything. The Event-Driven Architecture Pocket Guide covers the patterns around that boundary (sagas, CQRS, the outbox) and the traps that come with them, in a format you can read before the next incident review.

Event-Driven Architecture Pocket Guide: Saga, CQRS, Outbox, and the Traps Nobody Warns You About