Kafka has a reputation for being complicated, and its vocabulary does not help: topics, partitions, offsets, segments, consumer groups, rebalances, generations. I used these words for a long time before I could explain why each one has to exist.
The quickest way I know to understand a design is to rebuild a small version of it. mq is a small Kafka-style message queue in Go, using only the standard library. It has topics split into partitions, an append-only segmented log per partition, consumer groups with committed offsets, retention, and a TCP protocol with a Go client. It is a single broker and a learning project. What it gave me is a clear picture of why a log sits at the centre of a message queue.
What it does
The code is four packages:
-
wal: the storage engine. Segment files with CRC-checked records, a sparse offset index, crash recovery that cuts off a torn tail, and retention by size and age. -
broker: topics and partitions on top ofwal, key-hash or round-robin partitioning, long-polling fetches, consumer groups (broker-side assignment, heartbeats, session expiry, generation-fenced commits) and the TCP server. -
protocol: length-prefixed JSON frames. Each request has an ID, so a client can pipeline requests on one connection, and a parked long-poll does not block anything else. -
client: a Go client with a producer API and a groupConsumerthat handles joining, heartbeats, rebalances and commits.
On disk, a topic looks like this:
data/ topics/events/topic.json {"Partitions": 3} topics/events/0/00000000000000000000.log [len][crc32c][offset|ts|key|value] ... topics/events/0/00000000000000000000.index [rel offset u32][file pos u32] every 4 KiB topics/events/0/00000000000000004211.log next segment, named by its base offset offsets.json {"group/topic/partition": next offset} What I built from scratch, and what it teaches
Why a log is the core. In this design a message is never "taken" from the queue. It is appended to a partition and gets an offset, which is just a position. Consumers keep their own position, and "acknowledging" a message is committing a number. Once you see that, most of the rest follows: several groups can read the same topic independently, and a consumer can go back and read again.
Storage details. Each record is framed with a length and a CRC-32C checksum. The index is sparse: one entry every 4 KiB, mapping a relative offset to a file position. That is enough, because a read does a binary search on the segments' base offsets, finds the index entry at or below the wanted offset, then scans a few KiB. Segments roll when they are full, and each new one is named by its base offset. Retention deletes whole segments instead of rewriting files, which is why it is cheap.
Crash consistency. A crash in the middle of a write leaves a torn record at the end of the active segment. On open, recovery scans the active segment, truncates at the first bad record and carries on with the right offsets. Sealed segments load their index and only check from the last index entry to the end. The important distinction: a torn tail in the active segment is expected and repaired, but corruption in a sealed segment is an error, not silent data loss.
Consumer group coordination. When a member joins, the group's generation number goes up and partitions are assigned round robin. A member whose heartbeat carries a stale generation is told to rebalance and rejoins. A member that misses heartbeats for the session timeout is expired, which triggers a rebalance. The generation works as a fencing token: a commit is refused unless the member, the generation and the partition ownership all match. So a consumer that missed a rebalance cannot commit over the new owner. This is also where you see why delivery is at-least-once.
A protocol that does not block. Fetches are long polls: the broker parks the request until data arrives, and a produce wakes the waiting fetches. Because every request carries an ID, a parked long poll and other requests can share one connection.
How to use it
Install the broker binary with Go:
go install github.com/AbdeldjalilChougui/mq/cmd/mq@latest Add the Go client to your module:
go get github.com/AbdeldjalilChougui/mq/client@latest From a clone of the repository, the command line shows the whole flow:
go build -o mq ./cmd/mq ./mq serve -addr 127.0.0.1:9092 -data data & ./mq create -topic events -partitions 3 seq 1 10 | ./mq produce -topic events ./mq consume -topic events -group printers # Ctrl-C to stop ./mq topics And from Go:
c, _ := client.Dial(ctx, "127.0.0.1:9092") c.Produce(ctx, "events", -1, []byte("user-42"), []byte("signed up")) k := c.NewConsumer("mailer", "events") for { msgs, _ := k.Poll(ctx, time.Second) for _, m := range msgs { /* handle */ } k.Commit(ctx) } The loop is the usual one for this kind of system: poll, handle, commit. If the process dies after handling and before committing, the messages are delivered again. Your handler has to tolerate that.
How it is tested
go test -race ./... The wal tests cover appends and reads, segment rolling with reads across many segments, reopening, a lost index file being rebuilt, recovery from a torn write (the file is cut in the middle of the last record), trailing junk, a flipped byte in a middle record, refusing to open a damaged sealed segment, and retention by size and by age on a fake clock.
The broker tests cover key-to-partition stability, round robin, reloading topics after a restart, long polling, group assignment, rebalance on heartbeat, fenced commits from a stale generation or for another member's partition, session expiry, and offsets surviving a restart.
The client tests run over TCP on 127.0.0.1: two consumers sharing a 4-partition topic with every message delivered exactly once in steady state, a clean handover when one leaves, and an end-to-end crash-recovery test that stops the broker, truncates the last record on disk, restarts it and checks the data and the next offset.
Things to know
This is a learning project, and the README says what that means:
- Single broker. No replication, no leader election, no cluster metadata. If the disk dies, the data is gone.
- Durability is configurable but coarse. By default writes are flushed to the OS on each append and fsynced on segment roll and close.
wal.Options.SyncEveryWritefsyncs every record. A power loss can lose the last unsynced records, and recovery then drops any torn tail. - At-least-once delivery. During a rebalance, two consumers can briefly read the same partition; only the current owner's commit is accepted. There are no transactions or idempotent producers.
- Assignment is done by the broker, not by a group leader as in Kafka, and a group consumes exactly one topic.
- JSON framing is easy to debug but slow, and values are base64 on the wire.
- No authentication or TLS. Bind it to localhost or a trusted network.
- Offsets are stored by rewriting one JSON file per commit.
What I learned
The biggest change is in how I think about acknowledgements. I used to picture a queue as a list where messages are removed when handled. A log with offsets is simpler and stronger: the broker stores bytes in order, and each group stores one number per partition.
The second lesson is about fencing. Without the generation check, a slow consumer that wakes up after a rebalance would commit an old offset over the new owner's progress. A single integer that goes up on every rebalance closes that hole.
The third is that recovery code needs a rule about what damage is acceptable. A torn tail is normal after a crash. A bad record in the middle of a sealed file is not, and refusing to open is the honest answer.
Get the code
The source, the client and the tests are on GitHub: https://github.com/AbdeldjalilChougui/mq. It is MIT licensed. Start with the wal package: it is the part everything else stands on.
Originally published at abdeldjalilchougui.biz.


