By Prajakta Shende — Database Customer Engineer, Google Cloud, with close to a decade at AWS in the data space before that.
Nobody's database fails during the demo. They fail on Black Friday, on quarter-end close, or at 2 a.m. when a node loses its disk and the cluster has to decide — in seconds, with real customer traffic — whether it still has a quorum. If you haven't rehearsed that moment on purpose, the first rehearsal will be a production incident.
Pressure testing isn't a benchmark. Benchmarks are marketing. Pressure testing is a dress rehearsal for the worst day of the year: it tells you where the cliff is, how the system behaves at the edge, and what recovery actually looks like — before your customers find out for you. Here's the checklist I wish someone had handed me a decade ago.
1. Test the workload you actually have, not the one in the benchmark
Every database vendor ships numbers run on pristine schemas and uniform key distributions. Your production workload has neither. The two things that make or break a pressure test are the read/write ratio and the key skew — and both have to come from your real traffic.
A 95/5 read-heavy workload behaves nothing like a 50/50 one. Skewed access (a handful of hot keys taking most of the traffic) stresses very different machinery than uniform access. Capture a representative slice of production queries — anonymized — and replay that. If you can't replay production traffic, build a synthetic mix with the same ratios and skew, and document your assumptions so the next person knows what's a guess. The benchmark number you quote in a design doc should be your workload's number, not the vendor's.
2. Warm it up like production
Cold caches lie. Every test that starts against an empty buffer pool, an unwarmed OS page cache, or a freshly restarted node will report performance that no production system will ever see again. Run your workload until the metrics stabilize — the same p99 you measured at minute 5 should still be there at minute 50. If it isn't, you're measuring warmup, not capacity.
This sounds obvious. Nearly every "surprising" post-launch regression I've investigated skipped it.
3. Measure the tail, not the average
Averages are how databases pass tests and fail customers. At scale, what your users feel is the tail: p99, p999, and the shape of the tail under stress. Two systems with identical average latency can have wildly different p99 — and the one with the fatter tail is the one that pages you at 2 a.m.
Log full latency histograms, not summaries. And measure the tail during the ugly parts of the test: during a node failure, during a rebalance, during a rolling restart. The steady-state p99 is a vanity metric; the failure-mode p99 is the one your incident reviews will be about.
4. Kill things on purpose
This is the part most teams postpone until production does it for them. A pressure test that never loses a node is a benchmark with extra steps. Deliberately kill processes, kill whole nodes, partition the network, stall a disk — and measure two things each time:
- Detection time: how long before the cluster notices and reacts.
- Recovery behavior: what happens to write availability, read latency, and consistency during the window.
Write down what you expected before you run it. The gap between the expected recovery and the actual one is the most valuable output of the entire exercise — that gap is your incident playbook's first draft. I've seen clusters that were theoretically highly available take minutes to elect a new leader while the application timed out everything in flight. Theory doesn't pay the pager duty bill; measured recovery time does.
5. Find the cliff, don't just admire the slope
Load testing usually stops at "it handled our target." That's half the job. Keep going past the target until the system degrades — and watch how it degrades. Does latency rise linearly (you can plan around that), or does it fall off a cliff at some threshold (you cannot)?
The cliff's location is the number your capacity planning actually needs. Run the test at 50%, 80%, 100%, 120% of expected peak and map the curve. The difference between "graceful degradation" and "catastrophic collapse" decides whether your on-call gets a warning or a wake-up call.
6. Soak it
Some failures only appear with time: memory leaks, compaction debt, disk fragmentation, connection churn. A four-hour test won't find them. A 48-hour soak at realistic load will. It's boring, it's expensive to run, and it's the test that catches the bugs that show up in week three of production.
Schedule it at least once before every major launch. If you can't afford the 48 hours, you definitely can't afford the week-three incident.
7. Validate consistency while things break, not just after
It's not enough that the database comes back up. Verify that the data is right: run a consistency checker against a known-good baseline while failures are being injected, not just once everything is calm. The scariest bugs I've seen weren't outages — they were systems that stayed up and served slightly wrong answers. Checksums, row counts, and aggregate comparisons during chaos testing catch the corruption that uptime metrics hide.
The pre-production checklist
Before any distributed database takes production traffic, I want written answers to five questions:
- What is our workload's real read/write ratio and key skew, and what did we replay?
- What is the measured p99 under failure (node loss, partition), not just steady state?
- What is the detection time and recovery time for a lost node, in numbers?
- Where is the cliff — at what multiple of expected peak does the system stop degrading gracefully?
- Did we run a 48-hour soak, and what did the tail look like at hour 48 vs. hour 4?
Pressure testing doesn't prevent incidents. It converts surprise incidents into rehearsed ones — and rehearsed incidents are the kind you survive without a postmortem titled "we didn't know."
