send() is async: the seven stages a record crosses before your callback hears back.
send() returns immediately. That single fact explains most producer confusion: the call returning means the record was accepted into the client, not that it reached a broker, and definitely not that it is durable. The gap between what the returning call seems to promise and what it actually promises is where producer bugs live, because application code tends to treat a returned send() as a delivered record and moves on.
Between the return of send() and the firing of the callback, a record crosses seven stages. Every producer failure — the lost message, the duplicate, the mysterious latency spike — lives in one of them, and each stage fails in its own recognizable way.
In this post, I walk through all seven stages in the order a record crosses them, name the failure that belongs to each one, and close with the client internals that implement the stages, the defaults that matter, and the rule of thumb that turns a producer bug report into a stage number.
The core tension
The producer API is asynchronous by design, and the design is the right one: a client that blocked on every record would couple application latency to broker latency and throw away batching, which is where producer throughput comes from. The cost of that design is that delivery becomes a process with its own timeline, spread across a metadata cache, a partitioner, an in-memory accumulator, a background network thread, and a replicated broker log. A record can be healthy at five of those stages and dead at the sixth, and from the application's side all six outcomes look identical until the callback fires — or does not fire.
That framing changes how producer problems get debugged. The useful unit of analysis is not the producer as a whole but the stage: which stage held the record last, and what that stage does with records it cannot move forward. The seven stages below are that map.
Stage 1: Create producer and send()
Application code builds a KafkaProducer — in a sane setup, acks=all and enable.idempotence=true — and calls send(record, callback).
send() returns a Future immediately. The record has not left the JVM yet. The only blocking producer calls are flush() and close(), which is why shutdown code that skips them quietly throws away whatever is still in flight. The failure that belongs to this stage is an assumption: code that treats the return of send() as delivery has no delivery guarantee at all, only an acceptance guarantee, and every later stage becomes a place where a record the application believes is stored can still disappear.
Stage 2: Metadata
Before anything can be sent, the producer needs to know the topic's partitions and which broker leads each one. It fetches that metadata from any broker and caches it.
The cache refreshes on a timer — metadata.max.age.ms — or when a send fails against a stale view. A producer that "cannot find the leader" is usually working from metadata that was true a minute ago, not metadata that is true now. This stage fails quietly and heals on its own schedule: the view is eventually consistent by design, and the errors it produces point at the broker while the stale copy sits in the client.
Stage 3: Serialize and partition
Key and value are serialized to bytes. Then the partitioner decides where the record goes.
A keyed record hashes to a partition: the same key always lands on the same partition, which is what makes per-key ordering possible. A record with no key is handled by sticky partitioning — the producer fills one partition's batch before moving on, because full batches are cheaper than scattered ones. Partitioning is a one-way decision; once a record is assigned, nothing downstream reassigns it. Placement and ordering are therefore decided in the same step, at the same moment, and a partitioning mistake — a key that spreads one entity's records across partitions, or a key so hot it overloads one — cannot be corrected by any amount of broker-side health.
Stage 4: Accumulate and batch
Records wait in the RecordAccumulator, batched per partition. A batch ships when it fills — batch.size, 16 KB by default — or when linger.ms expires, whichever comes first.
buffer.memory is the ceiling on all of it. When brokers slow down and batches stop draining, the accumulator fills; once it is full, send() blocks up to max.block.ms and then throws. This stage is where lost messages are actually born: a record that never left the accumulator — because the process was killed, or the buffer exception was swallowed — was never lost by Kafka. It never arrived. The accumulator is also where throughput is made: batching, linger, and compression all happen here, so the same stage that hides silent loss is the stage that makes the producer fast, and tuning it trades directly between the two.
Stage 5: Sender and ProduceRequest
A background Sender thread drains ready batches and builds one ProduceRequest per partition leader. Application threads never touch the network; this thread is the only way out.
With idempotence enabled, up to five in-flight requests per connection are allowed while preserving order, because each batch carries a sequence number the broker can check. Retries also live here, bounded not by a retry count but by delivery.timeout.ms — 120 seconds by default — after which the producer gives up on the batch. The failure that belongs to this stage is the ambiguous retry: a write that succeeded on the broker while its acknowledgement was lost leaves the producer unable to distinguish a delivered record from an undelivered one, and idempotence — the sequence numbers, the in-flight limit — exists precisely to make that retry safe.
Stage 6: Broker append and replication
The leader broker appends the batch to its log segment and assigns offsets. Follower replicas fetch and copy it.
With acks=all, the write is not complete when the leader has it. It completes when every in-sync replica has it, and only then does the high watermark advance. That distinction is the whole durability story: an acknowledged record is a replicated record, and an unacknowledged one is a hope. This is the first stage where the record exists outside the producing process, and it is the stage the earlier stages are all machinery for reaching — bytes in a replicated log, with an offset, under a high watermark other consumers can trust.
Stage 7: Response and callback
The leader replies with the base offset of the batch. The Future completes and the callback fires with the record's partition and offset.
Failure takes the other path: retries run inside delivery.timeout.ms, and when that budget is spent, the callback receives the exception instead. A callback that ignores its error argument converts every delivery failure into silent loss, one record at a time. The callback is the first honest signal in the entire flow — the first point at which the application knows, rather than assumes, what happened to the record — and it only stays honest if the error half of it is handled as seriously as the success half.
Under the hood
The client internals line up with the stages: Partitioner, then RecordAccumulator, then Sender, then NetworkClient. On the broker side, KafkaApis hands the request to ReplicaManager, which writes to the Log — a segment file and its index on disk, named by partition and base offset. Offsets live in those files. Everything above them is machinery for getting bytes into them exactly once per acknowledged write.
Read that way, the seven stages stop being trivia about one client library. They are the shape any buffered, batched, replicated write path takes: decide placement, hold records until sending them is cheap, send them through a single accountable channel, make retries recognizable, and call a write durable only when the replicas agree.
Defaults that matter
acks=all enable.idempotence=true retries=Integer.MAX_VALUE delivery.timeout.ms=120000 linger.ms=5-20 compression.type=snappy Retries set to Integer.MAX_VALUE looks reckless and is not: delivery.timeout.ms is the real budget, and the count is just told not to give up first. Each line in that block maps to a stage. Acknowledgement strength and idempotence govern stages 5 and 6, where retries and replication decide between loss and duplication. The delivery timeout bounds stage 5's persistence. Linger and compression shape stage 4, where batches form. A baseline is not a ritual; it is the stage map with the dangerous defaults already moved.
What breaks, and the tradeoffs that stay
The design's tensions do not tune away. A longer linger makes bigger, cheaper batches and adds latency to every record that waits for one. A larger buffer absorbs longer broker slowdowns and holds more records in the exact place a killed process loses them. acks=all makes an acknowledgement mean something and makes every write wait on the slowest in-sync replica. Idempotence makes retries safe and caps in-flight requests to preserve the order that makes them checkable. The producer is a set of chosen points on those curves, and the stage map is what makes the choosing deliberate instead of accidental.
Rule of thumb
A "lost" message usually never left stage 4 — or died unflushed at shutdown. A "duplicate" is a stage 5–6 retry without idempotence: the write succeeded, the acknowledgement was lost, and the producer did the only thing it could do, which was send it again.
Name the stage and the bug names itself.
Takeaways
- send() returning means the record entered the client, nothing more; the callback is the first honest signal.
- Partitioning decides ordering and placement in one step, and it never gets revisited downstream.
- The accumulator is where throughput is made (batching, linger, compression) and where silent loss hides (full buffer, unflushed shutdown).
- acks=all plus replication is what turns a write into a durable write; the high watermark is the proof.
- Producer debugging is stage debugging: metadata, partition, accumulator, sender, broker, callback.
What's next
The producer is one half of the flow. The same stage-by-stage reading applies on the other side of the log, where a consumer's poll loop, group membership, and offset commits decide whether a durable record becomes processed work — and to the platforms that host both clients, where restarts and throttling reach into these stages from outside the process. The stage map travels because the shape repeats: asynchronous acceptance first, honest acknowledgement last, and a small number of named places in between where records can wait, fail, or quietly disappear.
A returned send() is a record entering a pipeline, not a record delivered. Debug the pipeline stage by stage, and producer bugs stop being mysteries.


