System design was confusing to me until I did this

I'm on my way to become a senior dev, and for a long time system design just didn't click for me.

I could tell you what replication or a load balancer is, but if you asked me what I'd actually do when one of them breaks in production, I'd mostly guess. I learned every piece on its own and never really seen how they affect each other.

What finally changed for me was reading case studies. A postmortem shows you the whole system failing at once, in order, with REAL people making REAL calls. There you see why the "obvious" fix sometimes makes the issue worse.

Besides that, I also got to know cool stories from real companies :o

At first I just read them start to end like a blog post, and honestly I forgot most of it a week later. So now I read them a bit differently.

How it started to click for me

I only read the timeline first, and I stop when the team starts figuring out the cause.
I write down what I'd check next. Usually I'm wrong, but that's kinda the point.
I look at what changed right before things broke. A deploy, a config change, something like that. It's surprising how often that's it.
I try to guess the cause before reading on, even if it's a bad guess.
Then I read what they actually did and think about whether I'd have done the same. Sometimes the fast fix and the safe fix are different, and that's the interesting part.
At the end I try to take away one thing about how the system should've been built.

It's wayy way slower, but I remember much more this way.

(I'll give you a pretty popular example now, skip it for one more extra advice)

Example: Discord's messages database

By 2022 Discord stored trillions of messages in Cassandra, on 177 nodes. On-call kept getting paged, latency jumped around for no clear reason, and sometimes someone had to manually reboot a node and babysit it. Their blog literally calls it "a high-toil system".

Honestly my first thought was "the cluster is too small, add more nodes".

Well, the slowdowns came with traffic. Messages are grouped by channel and time window, and each group lives on the same few nodes. Now imagine a server with hundreds of thousands of people, and someone pings @everyone. Everyone opens the same channel at once, and all those reads hit the same few nodes.

I guessed "too much traffic overall", but it was one busy channel. Those few nodes slow down, and because Discord's queries waited for a majority of the copies to answer, everything touching those nodes slowed down too. One channel could make Discord slow for people who had nothing to do with it. Java garbage collection pauses made it even worse.

To fix it, they didn't just add nodes. First they put a small Rust service in front of the database that does "request coalescing". That means, if 1000 people ask for the same message at the same moment, it asks the database ONLY ONCE and gives the answer to all 1000. Every request for the same channel goes to the same instance, so it actually catches them.

Then they moved everything to ScyllaDB, which is like Cassandra but in C++, so no garbage collector. The first migration tool estimated 3 months, so they wrote their own in Rust "in an afternoon", and the estimate dropped to 9 days.

It got stuck at 99.9999% because of old deleted-message markers Cassandra never cleaned up.

They cleaned that range and it finished seconds later.

Result: 177 nodes became 72, reading old messages went from 40-125 ms to 15 ms, and sending from 5-70 ms to a steady 5 ms (p99).

Then the World Cup final happened, every Messi and Mbappe goal showed up as a spike on their graphs, and the database barely broke a sweat.

Would I have done the same? No, I'd have bought more nodes and been confused why it didn't help.

So the lesson for me is that one busy key can slow down everyone, not just itself. Before making the database bigger, check if you're asking it the same question over and over.

One more extra advice

Read old case studies too. This is from 2023, but "one hot key hurts everyone" shows up everywhere, from caches to queues.

How to make this MORE INTERESTING

If you like this kind of stuff, I turn real outages into short cases you can play, where you make the calls before you see what the real team did: briefcasestudy.com