Slow etcd, API Server on Its Knees: Measure, Understand and Fix Datastore Latency

kubectl get pods -A takes eleven seconds. The web console spins. Controllers start logging client rate limiter Wait returned an error: context deadline exceeded. Someone restarts the API server pods, it gets slightly better for ten minutes, then it comes back.

Nine times out of ten this is not an API server problem. The API server is a stateless request translator; its p99 is almost entirely a function of how fast etcd can answer. And etcd's answer time is, in turn, almost entirely a function of two things: how long fsync takes on the underlying disk, and how much data has to be walked through to serve a range request. This article is about measuring both precisely, then fixing them.

Why etcd latency becomes everyone's problem

etcd is a Raft-replicated, write-ahead-logged key-value store. Every write must be:

1. appended to the WAL and fsynced to stable storage on the leader,
2. replicated to a quorum of peers, each of which fsyncs too,
3. applied to the boltdb backend, which is itself committed with another fsync.

That means a single kubectl apply of a ConfigMap costs you at least two disk flushes on a majority of control plane nodes. If fsync p99 goes from 3 ms to 60 ms, your write throughput collapses by a factor of twenty — and since the API server serializes a lot of work behind leader election leases, node heartbeats, and admission webhooks, the whole control plane degrades at once.

Worse, slow disks cause leader elections. The default --heartbeat-interval is 100 ms and --election-timeout 1000 ms. If the leader cannot flush its WAL in time, followers stop receiving heartbeats, a new election starts, and every in-flight write fails with etcdserver: request timed out. You then see kube-controller-manager and kube-scheduler lose their leader leases, which triggers a reconciliation storm — more writes, into an already saturated store.

The second failure mode is size. etcd keeps every revision until compaction, and boltdb never returns freed pages to the filesystem until you defragment. A cluster that churns 50k Events per hour, or stores 40 MB Helm release Secrets, ends up with a multi-gigabyte database where range requests have to page through far more disk than necessary — and eventually hits the backend quota and goes read-only with a NOSPACE alarm.

Measuring: from the symptom down to the disk

Layer 1 — the API server

Start at the top so you know which resources hurt:

histogram_quantile( 0.99, sum by (le, verb, resource) ( rate(apiserver_request_duration_seconds_bucket{verb!~"WATCH|CONNECT"}[5m]) ) ) > 1 

Then isolate the share spent in etcd. The API server exposes its own client-side view of the datastore:

histogram_quantile(0.99, sum by (le, operation, type) (rate(etcd_request_duration_seconds_bucket[5m])) ) 

If etcd_request_duration_seconds is flat and apiserver_request_duration_seconds is high, your problem is admission webhooks, authorization, or conversion webhooks — not etcd. Check apiserver_admission_webhook_admission_duration_seconds before blaming the datastore. If both move together, continue down.

Layer 2 — etcd internals

These four queries decide the diagnosis. Run them against the etcd ServiceMonitor targets (on OpenShift, etcd metrics are scraped out of the box into the openshift-monitoring Prometheus):

# 1. WAL fsync latency — target p99 < 10ms histogram_quantile(0.99, sum by (le, pod) ( rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m]))) # 2. Backend (boltdb) commit latency — target p99 < 25ms histogram_quantile(0.99, sum by (le, pod) ( rate(etcd_disk_backend_commit_duration_seconds_bucket[5m]))) # 3. Peer round-trip time — target p99 < 50ms, ideally < 10ms histogram_quantile(0.99, sum by (le, To) ( rate(etcd_network_peer_round_trip_time_seconds_bucket[5m]))) # 4. Instability signals increase(etcd_server_leader_changes_seen_total[1h]) rate(etcd_server_slow_apply_total[5m]) rate(etcd_server_slow_read_indexes_total[5m]) 

Interpretation grid:

Layer 3 — size and fragmentation

etcd_mvcc_db_total_size_in_bytes # physical file size etcd_mvcc_db_total_size_in_use_in_bytes # logically used etcd_server_quota_backend_bytes # hard limit (etcd_mvcc_db_total_size_in_bytes - etcd_mvcc_db_total_size_in_use_in_bytes) / etcd_mvcc_db_total_size_in_bytes # fragmentation ratio 

Default quota is 2 GiB; OpenShift raises it to 8 GiB. A fragmentation ratio above 0.45 with a DB over a few hundred MB is your cue to defragment.

From the cluster:

oc get pods -n openshift-etcd -l app=etcd -o wide oc rsh -n openshift-etcd etcd-ip-10-0-137-12.ec2.internal # inside the pod, the etcdctl wrapper already exports certs and endpoints etcdctl endpoint status -w table --cluster etcdctl endpoint health --cluster etcdctl alarm list 

endpoint status gives you DB size, raft term, and leader per member in one shot.

Fixing the disk: fsync is physics, not configuration

No etcd tuning compensates for a slow disk. Validate the hardware with the same access pattern etcd uses — sequential small writes with fdatasync after each:

# Run on a control plane node, on the same filesystem as /var/lib/etcd sudo mkdir -p /var/lib/etcd/fiotest sudo podman run --rm -v /var/lib/etcd/fiotest:/test:Z docker.io/library/alpine \ sh -c "apk add --no-cache fio >/dev/null && \ fio --rw=write --ioengine=sync --fdatasync=1 --directory=/test \ --size=22m --bs=2300 --name=etcd-fsync-bench" 

Read the fsync/fdatasync/sync_file_range percentile block. 99th percentile must be under 10 ms; under 2 ms is what healthy NVMe-backed control planes look like. If you are above that:

• Use local NVMe or a high-IOPS block volume (AWS io2/gp3 with provisioned IOPS, Azure Premium SSD v2). Never network filesystems, never burst-credit volumes.
• Move the WAL to a dedicated device with --wal-dir so WAL fsyncs never queue behind backend commits.
• Verify no other workload shares the device. On OpenShift, control plane nodes should not run user workloads; confirm with oc describe node | grep -A5 "Non-terminated Pods".
• Check for I/O throttling from cgroups and for iowait spikes: iostat -x 1 and look at w_await on the etcd device.

Also give etcd CPU and scheduling priority. On OpenShift this is handled by the cluster-etcd-operator (system-node-critical priority class, static pod). On self-managed clusters, make sure the etcd unit has CPUWeight/IOWeight favouring it and that the pod uses priorityClassName: system-node-critical.

Compaction and defragmentation

Compaction discards old MVCC revisions. The kube-apiserver drives it by default every 5 minutes (--etcd-compaction-interval=5m), so you rarely need to do it manually. Verify it is actually happening:

oc logs -n openshift-etcd etcd-ip-10-0-137-12.ec2.internal -c etcd | grep -i "compact" 

Manual compaction, if you must:

rev=$(etcdctl endpoint status -w json | jq -r '.[0].Status.header.revision') etcdctl compact "$rev" --physical 

Defragmentation is what actually returns freed pages to the filesystem. It blocks the member it runs on — expect seconds to a minute on a multi-GB database — so never defragment all members at once.

On OpenShift 4.9+, the cluster-etcd-operator defragments automatically when fragmentation exceeds 45% and the DB is larger than 100 MB. Check before doing it by hand:

oc logs -n openshift-etcd -l app=etcd -c etcd --tail=-1 | grep -i defrag 

Manual, member by member, leader last:

# Identify the leader (IS LEADER column) oc rsh -n openshift-etcd etcd-ip-10-0-137-12.ec2.internal etcdctl endpoint status -w table --cluster # Defrag one follower at a time oc rsh -n openshift-etcd etcd-ip-10-0-137-12.ec2.internal \ etcdctl --endpoints=https://10.0.137.12:2379 defrag --command-timeout=90s # Wait for the member to be healthy again before the next one oc rsh -n openshift-etcd etcd-ip-10-0-137-12.ec2.internal etcdctl endpoint health --cluster # Move leadership away before defragmenting the leader etcdctl move-leader <target-member-hex-id> 

If a NOSPACE alarm fired, defragmentation alone is not enough — you must disarm it:

etcdctl alarm list etcdctl alarm disarm 

Finding the pathogenic objects

Defragmenting a 7 GB database buys you a week. Deleting what put 7 GB in there buys you a year.

Count keys per resource type directly in etcd:

oc rsh -n openshift-etcd etcd-ip-10-0-137-12.ec2.internal etcdctl get /kubernetes.io --prefix --keys-only \ | awk -F'/' 'NF>2 {print $3}' | sort | uniq -c | sort -rn | head -30 

For a count on a specific prefix without dumping keys:

etcdctl get /kubernetes.io/events --prefix --count-only -w fields | grep Count etcdctl get /kubernetes.io/secrets --prefix --count-only -w fields | grep Count 

Then look at size, not just count, from the API side:

for r in secrets configmaps events pods customresourcedefinitions; do printf "%-28s %s bytes\n" "$r" \ "$(oc get "$r" -A -o json | jq '[.items[] | tostring | length] | add // 0')" done # Top 10 heaviest individual objects of a kind oc get secrets -A -o json \ | jq -r '.items | map({ns: .metadata.namespace, n: .metadata.name, s: (tostring|length)}) | sort_by(-.s) | .[:10][] | "\(.s)\t\(.ns)/\(.n)"' 

The usual offenders, in order of frequency:

• Events. Default TTL is 1 hour, but a CrashLoopBackOff fleet can still generate hundreds of thousands. They also drive huge write rates — far more damaging than their size.
• Helm release Secrets. sh.helm.release.v1.* are gzip+base64 blobs; charts with big CRDs easily reach tens of MB each, and Helm keeps 10 revisions by default. Set --history-max 3.
• Oversized ConfigMaps/Secrets used as poor-man's object storage.
• CRs with runaway status, especially operators that append to a condition list without truncating.
• metadata.managedFields on objects patched by many controllers — server-side apply bloat that can double an object's size.
• Orphaned resources from deleted tenants: completed Jobs, bound-but-unused PVCs, stale ReplicaSets.

Cleanups that pay off immediately:

# Prune completed jobs older than a day across all namespaces oc get jobs -A -o json | jq -r '.items[] | select(.status.succeeded==1) | select((.status.completionTime | fromdateiso8601) < (now - 86400)) | "\(.metadata.namespace) \(.metadata.name)"' \ | xargs -r -n2 sh -c 'oc delete job -n "$0" "$1"' # Stop the bleeding on events (last resort, they will regenerate) oc delete events -A --all 

Prevent regrowth with quotas and TTLs. A per-namespace object quota is the cheapest guardrail:

apiVersion: v1 kind: ResourceQuota metadata: name: object-count-guardrail namespace: team-payments spec: hard: count/configmaps: "250" count/secrets: "250" count/jobs.batch: "100" count/cronjobs.batch: "20" pods: "150" 

For Jobs, let the TTL controller do the work instead of a cron script:

apiVersion: batch/v1 kind: Job metadata: name: nightly-reconcile namespace: team-payments spec: ttlSecondsAfterFinished: 1800 backoffLimit: 2 template: spec: restartPolicy: Never containers: - name: job image: registry.access.redhat.com/ubi9/ubi-minimal:latest command: ["/bin/sh", "-c", "echo reconcile"] 

Throttling the noisy clients

Size is half the story; write rate is the other half. Find who is hammering the API:

topk(15, sum by (resource, verb, client) ( rate(apiserver_request_total{verb=~"POST|PUT|PATCH|DELETE"}[5m]))) 

A controller doing PATCH on the same object every second is a quorum write every second, multiplied by your replica count. Fix the controller if you own it; isolate it with API Priority and Fairness if you do not:

apiVersion: flowcontrol.apiserver.k8s.io/v1 kind: PriorityLevelConfiguration metadata: name: noisy-operators spec: type: Limited limited: nominalConcurrencyShares: 5 lendablePercent: 50 limitResponse: type: Queue queuing: queues: 32 handSize: 4 queueLengthLimit: 50 --- apiVersion: flowcontrol.apiserver.k8s.io/v1 kind: FlowSchema metadata: name: noisy-operators spec: matchingPrecedence: 500 priorityLevelConfiguration: name: noisy-operators distinguisherMethod: type: ByUser rules: - subjects: - kind: ServiceAccount serviceAccount: name: legacy-sync-operator namespace: legacy-tools resourceRules: - verbs: ["create", "update", "patch", "delete"] apiGroups: ["*"] resources: ["*"] clusterScope: true namespaces: ["*"] 

On clusters older than 1.29, use flowcontrol.apiserver.k8s.io/v1beta3. Verify the effect with apiserver_flowcontrol_rejected_requests_total and apiserver_flowcontrol_current_inqueue_requests.

Alerting before users notice

Ship these rules rather than waiting for a ticket:

apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: etcd-datastore-health namespace: openshift-monitoring labels: role: alert-rules spec: groups: - name: etcd-latency rules: - alert: EtcdWalFsyncSlow expr: | histogram_quantile(0.99, sum by (le, pod) ( rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m]))) > 0.01 for: 10m labels: severity: warning annotations: summary: "etcd WAL fsync p99 above 10ms on {{ $labels.pod }}" description: "Disk cannot sustain etcd write latency. Check device IOPS and co-tenancy." - alert: EtcdBackendCommitSlow expr: | histogram_quantile(0.99, sum by (le, pod) ( rate(etcd_disk_backend_commit_duration_seconds_bucket[5m]))) > 0.025 for: 10m labels: severity: warning - alert: EtcdDatabaseFragmented expr: | (etcd_mvcc_db_total_size_in_bytes - etcd_mvcc_db_total_size_in_use_in_bytes) / etcd_mvcc_db_total_size_in_bytes > 0.45 and etcd_mvcc_db_total_size_in_bytes > 200e6 for: 30m labels: severity: warning - alert: EtcdQuotaNearlyExhausted expr: | etcd_mvcc_db_total_size_in_bytes / etcd_server_quota_backend_bytes > 0.80 for: 10m labels: severity: critical - alert: EtcdLeaderChurn expr: increase(etcd_server_leader_changes_seen_total[1h]) > 3 labels: severity: critical - alert: ApiServerEtcdLatencyHigh expr: | histogram_quantile(0.99, sum by (le, operation) ( rate(etcd_request_duration_seconds_bucket[5m]))) > 1 for: 10m labels: severity: warning 

Security best practices

Performance work on etcd touches the most sensitive component of the cluster. A few non-negotiables:

• Never expose 2379/2380. Client and peer ports must be reachable only from control plane nodes. On OpenShift this is enforced by the node firewall rules; on self-managed clusters, verify with ss -ltnp | grep 238 and your security groups.
• Always use mTLS, with distinct CAs for client and peer traffic. --client-cert-auth=true and --peer-client-cert-auth=true are mandatory, not optional. On OpenShift, certificates are rotated by the cluster-etcd-operator — do not hand-manage them.
• Enable encryption at rest. Secrets live in etcd in plaintext by default; anyone with a snapshot has your credentials. On OpenShift:

apiVersion: config.openshift.io/v1 kind: APIServer metadata: name: cluster spec: encryption: type: aescbc # or 'aesgcm' on recent releases 

Then watch the rollout with oc get openshiftapiserver -o jsonpath='{.items[0].status.conditions[?(@.type=="Encrypted")]}'.

• Treat snapshots as secrets. cluster-backup.sh output contains every Secret in the cluster. Encrypt at rest, restrict ACLs to root, store off-cluster, and set a retention policy:

sudo /usr/local/bin/cluster-backup.sh /home/core/etcd-backup sudo chmod 600 /home/core/etcd-backup/* 

• Audit who reads bulk data. A LIST of all Secrets cluster-wide is both an etcd performance event and a credential-exfiltration signal. Keep an audit policy at Metadata level minimum for secrets, and alert on apiserver_request_total{resource="secrets",verb="LIST"} from unexpected service accounts.
• Defragment in a maintenance window with a fresh backup. Defrag rewrites the database file; a crash mid-defrag on a member means restoring that member from quorum. Always confirm endpoint health --cluster returns all members healthy before moving to the next one.
• Be careful with unsupportedConfigOverrides on OpenShift's kubeapiserver/etcd CRs. Tuning --event-ttl or heartbeat intervals that way puts you outside support and is silently reverted across upgrades.

Key takeaways

• The API server's p99 is etcd's p99 plus webhooks. Split the two with apiserver_request_duration_seconds versus etcd_request_duration_seconds before touching anything.
• Two metrics decide everything: etcd_disk_wal_fsync_duration_seconds (p99 < 10 ms) and etcd_disk_backend_commit_duration_seconds (p99 < 25 ms). Validate the hardware with fio --fdatasync=1 — no configuration fixes a slow disk.
• Leader changes are a symptom, never a root cause. Chase the fsync latency or the peer RTT behind them.
• Compaction reclaims revisions; only defragmentation reclaims disk. Run it one member at a time, leader last, after a backup. On OpenShift, let the cluster-etcd-operator do it unless you have a reason not to.
• Hunt the pathogenic objects: Events, Helm release Secrets, bloated CR statuses, managedFields. Counting keys per prefix in etcd takes two minutes and usually names the culprit immediately.
• Prevent regrowth with ResourceQuota object counts, ttlSecondsAfterFinished, Helm --history-max, and API Priority and Fairness for clients you cannot fix.
• Everything you store in etcd is plaintext unless you enable encryption at rest — and every snapshot you take is a complete copy of your cluster's secrets.