From Monitoring to Diagnosis: Redis Hot Key Troubleshooting and Mitigation

On load-test day, the P99 for the order details endpoint jumped from 40ms to 900ms. The database was idle, the application reported no errors, and the slow query log was empty. Eventually, monitoring showed one Redis node pegged at 100% CPU while using only 20% memory—a single product-detail key was taking 80k QPS on a single instance.

1. A Hot Key Is Not a Big Key

People often conflate the two, but the solutions are completely opposite.

A big key means the value is too large: a Hash stuffed with hundreds of thousands of fields, and a single HGETALL saturates the network and memory. The fix is to split the data structure.

A hot key means access is too concentrated: the value may be only a few hundred bytes, but it gets read tens of thousands of times per second. The fix is to spread out the access.

The trickier part is this: Redis Cluster uses CRC16 of the key modulo 16384 to assign 16,384 slots, and a key always lands in the same slot on the same node.

So:

Adding nodes won't solve a hot key. Even if you scale to ten nodes, that key still hits only one core.

And Redis executes commands single-threaded (single-threaded refers to command execution, not network I/O). When one core is maxed out, all keys on the same node slow down together, even if they are otherwise idle.

This explains a common confusion: why the slow query log is empty. It's because individual commands aren't slow—the slowness is in the queueing.

2. First Determine Whether It's a Hot Key

Don't rush to optimize; confirm first. If three signals appear at the same time, it's basically confirmed:

1. One Redis node's CPU is significantly higher than other nodes in the same cluster, while memory usage is not high
2. Response times for keys on that node rise across the board, not just for individual keys
3. From the business-side monitoring, one key accounts for an abnormally high share of QPS (e.g., a single key exceeds 20% of the instance's total QPS)

The third is the most decisive, but often the hardest to obtain, because most teams only monitor at the instance level.

3. Three Ways to Locate It

Method 1: redis-cli --hotkeys (Easiest, but with a prerequisite)

This option has been built in since Redis 4.0; it works by reading LFU counters. The prerequisite is that the eviction policy must be one of the LFU variants:

redis-cli -h 10.0.0.11 -p 6379 config set maxmemory-policy allkeys-lfu redis-cli -h 10.0.0.11 -p 6379 --hotkeys 

Two pitfalls:

• If the current policy is allkeys-lru, --hotkeys throws an error directly. After switching to LFU, you have to wait for the counters to accumulate for a while before the results mean anything; running it right after switching returns nothing
• LFU counters themselves decay, so the heat level drops after the hotspot passes. It reflects recent conditions, not historical peaks

This method is suitable for scenarios where the problem is already happening and you want to catch it in the act quickly. Be careful when switching eviction policies in production; it's best to do it during off-peak hours.

Method 2: Client-Side Instrumentation for TopN

This is the most reliable approach because it isn't limited by server-side policies and you can see where calls come from. The idea is simple: wrap it, record keys, and output periodically.

package com.example.hotkey; ​ import java.util.List; import java.util.Map; import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.atomic.LongAdder; import java.util.stream.Collectors; ​ import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.scheduling.annotation.Scheduled; import org.springframework.stereotype.Component; ​ @Component public class HotKeySampler { ​ private static final Logger log = LoggerFactory.getLogger(HotKeySampler.class); ​ private final ConcurrentHashMap<String, LongAdder> counters = new ConcurrentHashMap<>(); ​ public void record(String key) { counters.computeIfAbsent(key, k -> new LongAdder()).increment(); } ​ public List<Map.Entry<String, Long>> top(int n) { return counters.entrySet().stream() .sorted((a, b) -> Long.compare(b.getValue().sum(), a.getValue().sum())) .limit(n) .map(e -> Map.entry(e.getKey(), e.getValue().sum())) .collect(Collectors.toList()); } ​ @Scheduled(fixedDelay = 60000) public void flushAndReset() { List<Map.Entry<String, Long>> top = top(10); if (!top.isEmpty()) { log.warn("近 60 秒访问最多的 key: {}", top); } counters.clear(); } } 

Usage is to call sampler.record(key) wherever you read from the cache. The statistics themselves have memory overhead, so you need to do two things: only count read operations (writes are not the hotspot problem), and set an upper limit on the number of keys (e.g., discard new ones after 50,000 entries, so instrumentation doesn't blow up the heap).

Don't forget to enable the scheduled task on the startup class:

@SpringBootApplication @EnableScheduling public class HotKeyDemoApplication { public static void main(String[] args) { SpringApplication.run(HotKeyDemoApplication.class, args); } } 

Method 3: MONITOR (Avoid It If You Can)

redis-cli -h 10.0.0.11 -p 6379 monitor | awk -F '"' '{print $2}' | sort | uniq -c | sort -rn | head 

It lets you see all commands directly, but never use it in production: MONITOR copies every command and sends it to that connection, effectively doubling the output pressure on Redis. The hotspot is already suffocating it; turning this on could finish it off.

Only use it in a test environment or when there's truly no other option, and limit it to a few seconds.

4. Split: Two-Level Cache Plus Key Sharding

There's more than one solution, but the following two are enough for 90% of cases.

Tactic 1: Put a Local Cache in Front

Read requests for a hot key are highly repetitive—for a viral product detail, ten thousand requests read the same data. Letting them all hit Redis is wasteful; a local cache can eliminate over 95% of them.

package com.example.hotkey; ​ import java.time.Duration; import java.util.concurrent.ThreadLocalRandom; import java.util.function.Supplier; ​ import org.springframework.data.redis.core.StringRedisTemplate; import org.springframework.stereotype.Component; ​ import com.github.benmanes.caffeine.cache.Cache; import com.github.benmanes.caffeine.cache.Caffeine; ​ @Component public class TwoLevelCache { ​ private final Cache<String, String> local = Caffeine.newBuilder() .maximumSize(10_000) .expireAfterWrite(Duration.ofSeconds(30)) .build(); ​ private final StringRedisTemplate redis; private final HotKeySampler sampler; ​ public TwoLevelCache(StringRedisTemplate redis, HotKeySampler sampler) { this.redis = redis; this.sampler = sampler; } ​ public String get(String key, Supplier<String> loader) { String hit = local.getIfPresent(key); if (hit != null) { return hit; } sampler.record(key); String value = redis.opsForValue().get(key); if (value == null) { value = loader.get(); if (value != null) { long ttl = 300 + ThreadLocalRandom.current().nextLong(60); redis.opsForValue().set(key, value, Duration.ofSeconds(ttl)); } } if (value != null) { local.put(key, value); } return value; } } 

Don't skip three details:

• The local TTL must be significantly shorter than the Redis TTL (here, 30 seconds vs. 5 minutes). A local cache trades consistency for performance, and a short TTL is the least costly compromise
• Add random jitter when writing back. If a batch of keys expires at the same time, they'll all go back to the source at once—that's how a hot key turns into a cache avalanche
• The local cache must have an upper bound. If maximumSize is not set, you're burying a memory leak for yourself

Tactic 2: Key Sharding

A local cache solves reads, but if writes are also concentrated (e.g., inventory deduction), or a local cache isn't suitable (multiple languages, too much data), you have to split the key.

package com.example.hotkey; ​ import java.util.concurrent.ThreadLocalRandom; ​ public final class KeySharding { ​ private KeySharding() { } ​ public static String[] shardKeys(String key, int shards) { String[] keys = new String[shards]; for (int i = 0; i < shards; i++) { keys[i] = key + ":s" + i; } return keys; } ​ public static String pick(String key, int shards) { int index = ThreadLocalRandom.current().nextInt(shards); return key + ":s" + index; } } 

When writing, write the same value into N copies (key:s0 through key:sN-1), and pick one at random when reading. This way the N keys spread across different slots and nodes, and the pressure on a single core is diluted.

The cost is straightforward: N× storage amplification and N× write amplification. So N is typically 4 to 8—don't overdo it.

5. Four Pitfalls I've Hit

1. Local cache and Redis falling out of sync caused users to see stale data. The admin changed a product price, and Redis was updated, but the local cache on the user's machine still had the old value. A 30-second TTL means up to 30 seconds of inconsistency. If your business can't tolerate that, don't use local caching—or use Redis Pub/Sub to broadcast invalidation messages (make sure the complexity and payoff are worth it).

2. Batch read costs explode after key sharding. Some code used to fetch 10 keys with a single MGET. After sharding, it becomes 40 keys across multiple nodes, and MGET degrades into N individual single-node requests—making network overhead even worse. Key sharding only works for high-frequency single-key scenarios, so don’t apply it to batch scenarios.

3. After the hot key disappears, the code remains. Once a campaign ends, the hotspot is gone, but the sharding logic and local cache stay in the code. From then on, every read goes through an extra local cache layer and uses N times more storage. Recommendation: make the sharding switch configurable, and turn it off when the campaign ends.

4. Optimizing only the read side while writes continue to collapse. In many write scenarios—flash-sale inventory deduction, like counters—the hotspot is inherent, and sharding cannot resolve conflicts in business semantics. These cases should be handled with queues or sharded counters; don’t count on the cache layer.

6. Solution Comparison

A suggested order for deciding: first confirm it's a hot key, then see whether local caching works; only if local caching doesn't work should you consider sharding, and only as a last resort change the structure.

Doing it in the reverse order often turns a simple problem into a complicated one.

References

• Redis key eviction (LFU counters and eviction policies)
• redis-cli tool documentation (including --hotkeys and --bigkeys)
• Spring Data Redis reference documentation
• Spring Boot reference documentation
• Caffeine version directory on Maven Central