This is the third in my series on control plane architecture and design patterns. See the first article on a high-level overview of control planes here, and the second on the scaling component here.

I ended my last piece on scaling with a promise to come back to hot partitions, i.e., the situation where a single piece of your system is hit with disproportionate traffic no matter how well the rest of your infrastructure is provisioned. That problem lives squarely in load balancing and routing, which is what I want to dig into here.

The thing that surprised me most when I actually started building these systems is how little "load balancing" has to do with load balancers, once you get past the simplest cases. A load balancer is one tool for solving a routing problem. It's not the only one, and for a lot of distributed systems, it's not even the primary one.

The Simple Case: Fronting Stateless Services

For a basic web application, load balancing is close to what most people picture:

  • DNS-based load balancing -  multiple A records pointing to different IPs, with clients or resolvers distributing requests across them. It's cheap and simple, but coarse: you get little control over how traffic is actually distributed, and DNS on its own has no real-time awareness of backend health. Many DNS providers let you associate health checks with your records so that unhealthy IPs get pulled out of rotation, which closes part of that gap. But those checks aren't free: each one is a probe you pay for and have to tune, and the cost grows with the number of endpoints and the frequency you check them. They also have no global view of the system. A check can tell you whether an endpoint responds, but not how loaded it is, how it compares to its peers, or whether pulling it out of rotation will overload the ones that remain.
  • A dedicated load balancer or proxy (an ALB, an NLB, nginx, Envoy) sitting in front of your fleet, actively health-checking backends and routing each connection or request according to an algorithm, e.g., round robin, least connections, weighted, or latency-based.


For stateless, interchangeable backends, this gets you most of the way there. Any healthy instance can serve any request, so the algorithm mostly just needs to avoid overloading any single one.


This is also where most teams' mental model of "load balancing" stops. It's worth pushing past that, because the moment your backend isn't stateless and interchangeable, this model stops being sufficient.

The Harder Case: Routing With State

A distributed database doesn't have interchangeable backends. Each host owns specific partitions of data, and a request for a given key has to go to the host that actually holds it (or a replica of it). You can't round-robin your way out of that.  The routing decision is now coupled to where the data lives, which is a placement problem, not a traffic-distribution problem.

This is where load balancing stops being about the request path and starts being about the partitioning and placement strategy decided when data is written, long before any request shows up.

Placement Strategy Is Load Balancing in Advance

When a new table or dataset is created in a system like Cassandra, the control plane decides how its partitions get distributed and replicated across the fleet. Get that placement strategy wrong, and no amount of clever request routing fixes it later. You've already baked an imbalance into where the data lives.

Good placement strategies generally try to:

  • Spread partitions evenly across available hosts, so no single host owns a disproportionate share of the dataset
  • Respect replication and fault-domain constraints simultaneously - replicas of the same partition shouldn't land on the same rack or availability zone, even while you're optimizing for even distribution
  • Account for heterogeneous hardware; if your fleet isn't uniform, a host with double the capacity should reasonably get more partitions than one with less

None of this touches the request path directly. It's load balancing that happens at write time, and it's arguably the more important half of the problem.

The Problem Placement Alone Can't Solve: Hot Partitions

Even a well-designed placement strategy runs into a specific failure mode: hot partitions. This happens when traffic to specific keys is wildly uneven, usually because of a poorly chosen partition key (a timestamp, a low-cardinality value, a celebrity user ID) that concentrates requests onto one partition regardless of how evenly the data was distributed.

The uncomfortable part is that this isn't something you can fully prevent through upfront design. Traffic patterns shift, applications evolve, and a key that was fine at launch can become a hotspot a year later. Detecting and correcting for it has to be an ongoing, automated function of the control plane, not a one-time decision.

Detecting Hot Partitions

This generally requires:

  • Per-partition metrics, not just aggregate fleet metrics such as throughput, request count, and latency broken down at the partition level, since fleet-wide averages will hide a hot partition sitting next to a dozen idle ones
  • A threshold model that distinguishes a genuinely overloaded partition from normal variance. Naive fixed thresholds tend to either fire constantly on noisy workloads or miss slow-building hotspots
  • Attribution, ideally, knowing which keys within a partition are driving the load helps decide whether splitting the partition will actually help, or whether the hot key will just follow you into the new split

Correcting for It: Re-Partitioning Live

Once a hot partition is identified, the fix is usually to split it and redistribute the pieces, but doing this against a partition serving live production traffic is one of the more delicate operations a control plane performs. A few things matter here:

  • The split has to be incremental and reversible. Splitting a partition in one atomic step against live traffic is asking for an outage; the operation needs to be able to pause, resume, or roll back mid-flight.
  • Routing has to update consistently with the split, so requests during the transition land on the correct new partition rather than the stale one. This usually means a routing/metadata layer that's updated as part of the same operation, not after it.
  • The new partitions need to actually solve the problem. If the hot key itself is the issue rather than uneven key distribution, splitting the partition just relocates the hotspot rather than resolving it. Sometimes the real fix is upstream, in how the application chooses partition keys in the first place.

Systems like DynamoDB build this kind of adaptive re-partitioning directly into their control plane, specifically so that a single misbehaving key doesn't require manual intervention every time it happens.

The Pattern Shows Up Everywhere Stateful Systems Do

Databases are the clearest illustration of placement-time routing, but the same problem, and largely the same solutions, show up anywhere a system shards state across a fleet. Two examples worth knowing well, because they each stress the pattern in a different direction.

Sharded Caches

A distributed cache like Redis Cluster or Memcached with consistent hashing looks superficially like the database case: keys are hashed to determine which shard owns them, and requests get routed accordingly. But caches diverge from databases in one important way that reshapes the whole routing problem - cache misses are cheap enough to tolerate, but hot keys are far more common and far more punishing.

Because caches sit in front of a slower system of record, a single viral key (a trending post, a flash-sale product, a piece of session data for a suddenly-popular event) can spike orders of magnitude above normal traffic in seconds, not the gradual drift you'd see with a hot database partition. A few things follow from that:

  • Consistent hashing helps with rebalancing, not hot keys. It minimizes how many keys move when you add or remove a shard, but it does nothing to stop one key from being disproportionately hot. The hash function doesn't know or care about request volume.
  • Client-side or edge-level request coalescing often matters more here than server-side re-sharding. If ten thousand requests for the same key arrive in the same second, the fix usually isn't "move this key to its own shard" (by the time you've detected and acted on it, the spike may already be over),  it's collapsing concurrent requests for the same key into one upstream fetch.
  • Local caching layers in front of the shard tier (an in-process cache on each application host, for instance) can absorb a hot-key spike before it ever reaches the sharded layer at all, which is often a simpler fix than trying to make the sharded routing layer reactive enough to keep up.

The routing decision for a cache miss is still a placement problem, i.e., determination of which shard owns this key. The operational fix for a hot key, however, usually lives one layer up from the routing logic itself, in how aggressively you can avoid sending duplicate requests downstream in the first place.

Stream Processors

Systems like Kafka consumer groups or Flink jobs route by partition, much like a sharded database. A message with a given key is deterministically assigned to a partition, and a specific consumer or task instance owns that partition for processing.

But the load balancing constraint here is different in a way that matters: ordering guarantees typically require that all messages for a given key are processed by the same partition, in order, which means you can't rebalance a hot partition by simply splitting it the way you might with a database.

This creates a distinct set of routing trade-offs:

  • Partition count is usually a fixed, upfront decision. Unlike a database that can dynamically split an overloaded partition, most stream processing systems set partition count at topic-creation time, because increasing it later changes the key-to-partition mapping and can break ordering guarantees for existing consumers mid-stream. Getting this number right up front matters more here than almost anywhere else in the stack. Under-provision partitions and you have a hot-partition problem you architecturally cannot fix without a disruptive migration. Over-provisioning is the safer mistake, but it isn't free either: every partition carries its own log segments, file handles, and replication state on the brokers, so a very high count slows recovery after a broker failure (more partitions need new leaders), makes consumer group rebalances more disruptive, and shrinks producer batches, which hurts efficiency and latency. It also won't rescue you from a single hot key, since a given key always maps to one partition no matter how many exist. The practical target is enough headroom for foreseeable growth in consumer parallelism, not an arbitrarily large number "to be safe."

  • Consumer-side scaling is bounded by partition count, not by consumer group size. Adding more consumers than partitions doesn't help. The extra consumers just sit idle, so "scaling" a stream processing system often means capacity planning for partition count well ahead of actual load, rather than reactively adding resources the way you would with a stateless service.

  • Skewed key distribution is a design-time problem, not a runtime one. Because you can't cheaply reshuffle partitions after the fact, uneven key distribution (one customer ID generating 100x the events of any other) has to be addressed either in how keys are chosen, or by explicitly over-partitioning for the skew you expect by anticipating the hot path rather than reacting to it.

Stream processors push placement-time routing to its logical extreme: nearly all of the load balancing work happens at design time, because the ordering guarantees that make the system useful are exactly what make runtime rebalancing so constrained.

Bringing It Together

Load balancing in a distributed system operates on two timelines that are easy to conflate:

  • Request-time routing - the classic load balancer problem, distributing incoming requests across available, healthy backends
  • Placement-time routing - deciding, before any request arrives, where data or resources live in the first place, and continuously correcting that placement as traffic patterns change

For stateless services, the first one is basically the whole problem. For anything stateful and partitioned, the second one dominates, and it's a control plane responsibility that runs continuously in the background rather than a configuration you set once and forget.

Coming Up Next

Next in this series, I'll get into reliability guarantees - how control planes enforce promises (replication, TTL-based deletion, eventual consistency) that are genuinely hard to guarantee synchronously in the request path, and the auditing systems teams build to catch the gap between what a system promises and what it actually did.