Apache Kafka Interview Questions: explaining the log that moves your data

An Apache Kafka interview is not a quiz on config flags. It tests three things: whether you understand how Kafka works as a distributed commit log, whether you can reason about ordering and parallelism through partitions, and whether you can pick the right delivery guarantee for a pipeline and defend it. For senior data roles there is a fourth — the scenario, "messages are being processed twice after a consumer crash; what is happening and how do you fix it?" In every case the answer that lands is a short, confident sentence you say out loud, with the deeper detail ready for the follow-up.

This guide covers the Kafka interview questions that come up most in 2026: the topic–partition–offset model, why ordering is only per-partition and how the partition key controls it, the delivery semantics (at-most-once, at-least-once, exactly-once), and when Kafka is the right tool. It is data engineering prep for a topic where explaining why Kafka behaves the way it does matters as much as naming the component.

Title card for a 2026 Apache Kafka interview questions guide for data engineers, showing a partitioned append-only log with producers writing and consumer groups reading by offset, on a dark citrus-and-lavender background.

How Kafka works: the commit log

Everything in a Kafka interview builds on one mental model: Kafka is a distributed, append-only commit log, not a traditional queue. Producers append records to the end of a log; consumers read forward from a position they control. The vocabulary carries most of the questions:

  • Topic — a named stream of records (for example orders). It is split into partitions.
  • Partition — an ordered, append-only sequence of records. Each record has an offset, its position in the partition. Partitions are the unit of parallelism and of ordering.
  • Producer — appends records to a topic; a partition key decides which partition each record lands in.
  • Consumer / consumer group — consumers read partitions and track their offset; a consumer group shares the work of a topic, with each partition assigned to exactly one consumer in the group.
  • Broker — a Kafka server holding partitions; a cluster of brokers replicates partitions for durability, with one leader per partition handling reads and writes.

The sentence that frames everything: Kafka is a durable, replayable log partitioned for parallelism, and consumers — not the broker — track their own position, which is why many independent consumers can read the same data at their own pace. Say that and you have framed the whole system before anyone asks about a component.

A diagram of Kafka's architecture showing a topic split into several partitions, each an ordered append-only log of records with increasing offsets, producers appending to partitions by key, brokers replicating partitions with a leader and followers, and a consumer group where each partition is assigned to one consumer that tracks its own offset.

Partitions, keys and ordering: the make-or-break round

This is where Kafka interviews are won and lost, because the single most misunderstood fact is that Kafka only guarantees ordering within a partition, never across a topic. If a topic has four partitions, records in different partitions have no ordering relationship at all.

That makes the partition key the most important design decision. The producer hashes the key to pick a partition, so all records with the same key land in the same partition and are therefore ordered relative to each other. Choose the key to match the entity whose order matters:

# key = entity whose order must be preserved -> same partition -> ordered producer.send("orders", key=order_id, value=event) # all events for one order_id keep their relative order; # events for different orders may interleave across partitions 

Two consequences to say out loud. First, parallelism is bounded by partitions: a consumer group can have at most as many active consumers as there are partitions, because each partition goes to exactly one consumer in the group — add more consumers than partitions and the extras sit idle. Second, a hot key creates a hot partition: if one key dominates, its partition does far more work than the others, the classic skew problem. The fixes mirror any skew: a better key, more partitions, or splitting the hot key. When a consumer joins or leaves, the group rebalances — partitions are reassigned across the consumers — which you want to be aware of because it briefly pauses consumption.

A diagram showing that Kafka ordering holds only within a partition, with the partition key hashing records for the same key into the same ordered partition while different keys spread across partitions, and a consumer group where the number of partitions caps the number of active consumers, with a hot key overloading one partition.

Data Engineering
DE · Kafka / streaming
Explain partitions and ordering out loud

Practise on Zynter →

Delivery semantics: at-most-once, at-least-once, exactly-once

The deepest Kafka follow-up is about guarantees, because it is where a pipeline silently loses or duplicates data. Name the three and the trade-off each makes:

Semantic What happens Cost / how
At-most-once commit the offset before processing — if the consumer crashes mid-process, the record is skipped no duplicates, but data loss on failure; rarely what you want
At-least-once commit the offset after processing — a crash before commit means the record is re-read and reprocessed no loss, but duplicates on retry; the common default
Exactly-once idempotent producer + transactions tie the write and the offset commit into one atomic unit no loss, no duplicates; more overhead and only within Kafka's transactional boundary

The key insight to say: the offset commit timing is what decides the semantic. Commit before you process and you risk loss; commit after and you risk duplicates. Most pipelines run at-least-once and make the consumer idempotent — deduplicate on a key, or use an upsert — so a reprocessed record is harmless. Exactly-once is real in Kafka via the idempotent producer (dedupes producer retries with a sequence number) and transactions (atomic write-plus-offset-commit), but it adds overhead and only holds within Kafka-to-Kafka boundaries, so you reach for it deliberately, not by default.

# at-least-once + idempotent consumer is the pragmatic default # producer side: durability + no duplicate retries # acks=all, enable.idempotence=true for record in consumer: upsert(record.key, record.value) # idempotent: reprocessing is harmless consumer.commit() # commit AFTER processing 

A diagram of Kafka delivery semantics comparing at-most-once where the offset is committed before processing so a crash loses the record, at-least-once where the offset is committed after processing so a crash causes reprocessing and duplicates, and exactly-once using the idempotent producer and transactions to make the write and offset commit atomic, with a note that at-least-once plus an idempotent consumer is the common default.

Data Engineering
DE · Delivery semantics
Defend your delivery guarantee out loud

Practise on Zynter →

When to use Kafka, and the ecosystem

"Use Kafka" is not always the answer, and knowing when it is reads as experience. Kafka shines when you need a durable, replayable, high-throughput log that many independent consumers read at their own pace — event streaming, decoupling producers from consumers, a buffer in front of a slow sink, change-data-capture, or a source of truth you can replay to rebuild state. It is not a drop-in for a task queue that needs per-message acknowledgement and priority, and it is overkill for low-volume request/response.

Name a few ecosystem pieces so the answer is grounded:

  • Retention and log compaction. Kafka keeps records for a configured retention (time or size), and log compaction keeps only the latest record per key — which turns a topic into a replayable snapshot of current state, ideal for CDC and keyed state.
  • Consumer offset tracking. Offsets are themselves stored in Kafka, so a consumer group resumes exactly where it left off after a restart.
  • Kafka Connect for source/sink integrations without custom code, Kafka Streams / ksqlDB for stream processing, and a schema registry for enforcing record schemas across producers and consumers.
  • Replication. Each partition is replicated across brokers with a leader and in-sync replicas, so a broker failure does not lose data — the durability story behind acks=all.

The framing to land: Kafka is the durable log at the centre of a streaming platform; the ecosystem (Connect, Streams, schema registry, compaction) is what turns the log into pipelines.

How to prepare for a Kafka interview

You do not need to memorise every config. You need the mental model automatic and the trade-offs reflexive, so a pipeline question triggers the right answer under follow-ups.

  • Day 1: Learn the commit-log model cold — topic, partition, offset, producer, consumer group, broker, replication — and be able to draw it and say why consumers track their own offset.
  • Day 2: Drill partitions and ordering: per-partition ordering only, the partition key, parallelism bounded by partitions, hot-key skew, and rebalancing. Say each in one sentence.
  • Day 3: Own delivery semantics — at-most/at-least/exactly-once, the offset-commit timing, idempotent consumers, and the idempotent producer plus transactions — plus when to use Kafka at all. Then do it out loud, timed, with someone pushing on "why are messages processed twice?".

If you want a deep bank of Kafka and data-engineering questions to drill the mechanics, the sister platform PipeCode has dedicated data-engineering guides; use it for the drills, then come back here to rehearse explaining them out loud under follow-ups.

The follow-ups that decide a Kafka round

Scan the probes and they rhyme. Master these five and you are ready for pipeline questions that are not even on the list.

  • "Does Kafka guarantee ordering?" → Only within a partition, never across a topic; the partition key puts same-key records in the same partition to keep them ordered. (problem solving)
  • "How many consumers can read a topic in parallel?" → At most one per partition in a consumer group, so parallelism is capped by the partition count. (technical execution)
  • "Messages are processed twice after a crash — why?" → At-least-once with the offset committed after processing: a crash before commit re-reads the record; make the consumer idempotent. (communication)
  • "How do you get exactly-once?" → The idempotent producer plus transactions to make the write and offset commit atomic, accepting the overhead and the Kafka-boundary limit. (technical execution)
  • "Kafka or a traditional queue here?" → Kafka for a durable, replayable, multi-consumer log; a queue for per-message ack, priority, or low-volume request/response. (problem solving)

Each maps to a debrief dimension. Reading this list teaches you the answers; it does not teach you to deliver them while someone asks "why are messages processed twice?" — that only comes from saying them out loud.

Frequently asked questions

What is Apache Kafka and how does it work?

Apache Kafka is a distributed, append-only commit log used for event streaming. Producers append records to a topic, which is split into partitions — each an ordered sequence of records identified by an offset — and brokers hold and replicate those partitions across a cluster. Consumers read forward through partitions and track their own offset, so many independent consumer groups can read the same data at their own pace and replay it from any point. The key idea that separates Kafka from a traditional queue is that it is a durable, replayable log where the consumer, not the broker, owns its position.

Does Kafka guarantee message ordering?

Kafka guarantees ordering only within a single partition, never across a whole topic. Records in the same partition are read in the exact order they were appended, by offset, but records in different partitions have no ordering relationship. That is why the partition key matters: the producer hashes the key to choose a partition, so all records sharing a key land in the same partition and stay ordered relative to each other. If you need events for one entity — say one order or one user — to be processed in order, you key by that entity so they share a partition.

What are Kafka's delivery semantics?

There are three. At-most-once commits the offset before processing, so a crash mid-process skips the record — no duplicates but possible data loss. At-least-once commits the offset after processing, so a crash before the commit causes the record to be re-read and reprocessed — no loss but possible duplicates, and it is the common default. Exactly-once uses the idempotent producer and transactions to make the write and the offset commit atomic, giving no loss and no duplicates at the cost of extra overhead and only within Kafka's transactional boundary. Most pipelines run at-least-once and make the consumer idempotent so reprocessing is harmless.

How does a Kafka consumer group work?

A consumer group is a set of consumers that share the work of reading a topic. Kafka assigns each partition to exactly one consumer in the group, so the consumers divide the partitions between them and process in parallel without reading the same record twice. Because each partition goes to one consumer, the number of partitions caps the useful parallelism — more consumers than partitions leaves the extras idle. When a consumer joins or leaves, the group rebalances and reassigns partitions, which briefly pauses consumption. The group's committed offsets let it resume exactly where it left off after a restart.

When should you use Kafka instead of a message queue?

Use Kafka when you need a durable, replayable, high-throughput log that multiple independent consumers read at their own pace — event streaming, decoupling services, buffering in front of a slow sink, change-data-capture, or a replayable source of truth. A traditional message queue is a better fit when you need per-message acknowledgement, priority ordering, or a simple task queue, and for low-volume request/response Kafka is usually overkill. The deciding question is whether you want a log you can replay and fan out to many consumers, or a queue where a message is consumed once and acknowledged.

How do I practise data engineering interviews out loud?

Reading about Kafka is not the same as defending a delivery-semantics choice while someone asks why messages are processed twice. On Zynter, the Data Engineering track runs live voice rounds modelled on how real data-engineering loops run — a voice interviewer pushes with the follow-ups above one at a time, then gives you a debrief on problem solving, communication, technical execution and time management. The score is a practice signal, not a hiring decision.

Practise it on Zynter

You can read every guarantee. The round is won on defending your delivery semantics out loud.

Pick a Data Engineering round, reason through a Kafka pipeline, and defend it out loud to a voice interviewer that keeps asking "does Kafka guarantee ordering?" and "why are messages being processed twice?" — then read the debrief. Zynter.ai is on-demand mock interviews modelled on real company loops, with 150+ rounds across 20 companies including the Data Engineering track (Sep 2026).

Practise a Data Engineering round →
Browse all interview tracks