Prometheus data lost after pod restart? How to set kube-prometheus-stack persistent storage (a storageSpec PVC) without losing the emptyDir data

Where kube-prometheus-stack kept 30 days of metrics before and after the move. Measured on our cluster on 2026-11-15.

If Prometheus data was lost after a pod restart, or you are setting up kube-prometheus-stack persistent storage, the cause to check first is this: unless prometheus.prometheusSpec.storageSpec is set, prometheus-operator stores the Prometheus TSDB on an emptyDir, and an emptyDir is deleted together with the pod. The same cause is behind the operator log line recreating StatefulSet because the update operation wasn't possible right after you add storageSpec: if the old pod had an emptyDir, its TSDB was deleted with that pod. Our Prometheus pod had run that way for 349 days, and at the end its emptyDir held 22.7G covering the last 31.5 days of metrics. To check yours: if kubectl get prometheus -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name} storage={.spec.storage}{"\n"}{end}' prints a line that ends in storage= with nothing after it, that Prometheus keeps its TSDB on an emptyDir. To add persistent volume to Prometheus without losing data, create the PVC prometheus-<name>-db-prometheus-<name>-0 and copy the TSDB into its prometheus-db/ folder before adding prometheus.prometheusSpec.storageSpec.volumeClaimTemplate, because the operator deletes the StatefulSet when you add it. In our move, the 30 days of blocks and WAL that we copied were kept, and a deliberate pod delete afterwards lost nothing stored. The samples the old pod wrote after the second copy (00:38:05) were never copied, and Prometheus's scrape of itself has a 2 min 6 s hole around the switch (the cause is our inference from the times, see "What it cost"). The cluster is a 2-node k3s cluster (server-1, agent-1) running kube-prometheus-stack 79.5.0 from Argo CD, prometheus-operator v0.86.2 and Prometheus v3.7.3; all times are UTC, 2026-11-15 unless a date is given.

What we saw: the Prometheus data volume was an emptyDir (prometheus-kube-prometheus-stack-prometheus-db {})

The finding

A full run of kubegap v0.3.0 at 2026-11-14 16:25:29 (the tool is described in the check section below) listed as its problem 2, translated from Korean: "Prometheus data is on an emptyDir." The reason it gave: the moment the Prometheus pod is deleted and created again, 30 days of metrics are gone. Two findings carried it, both fail:

prometheus.storage-risk storage emptyDir (PVC None), PVC usage None, TSDB 22.3GiB, retention 30d (size cap 0B), actual oldest sample 31.2 days, failed compactions over 7 days 0, WAL corruptions 1, corruptions at startup 1 observability.stack-storage-emptydir emptyDir volumes: config-out, prometheus-kube-prometheus-stack-prometheus-db, replicas 1 

The StatefulSet and the Prometheus object

At 00:31:25 we read the objects ourselves. The StatefulSet has no volumeClaimTemplates (the first, empty line of the output), and the data volume is an emptyDir ({}):

$ kubectl -n monitoring get sts prometheus-kube-prometheus-stack-prometheus -o jsonpath='{.spec.volumeClaimTemplates}{"\n"}{range .spec.template.spec.volumes[*]}{.name}{" "}{.emptyDir}{.persistentVolumeClaim}{"\n"}{end}' config tls-assets config-out {"medium":"Memory"} prometheus-kube-prometheus-stack-prometheus-rulefiles-0 web-config prometheus-kube-prometheus-stack-prometheus-db {} $ kubectl -n monitoring get prometheus kube-prometheus-stack-prometheus -o jsonpath='retention={.spec.retention} retentionSize={.spec.retentionSize} storage={.spec.storage} replicas={.spec.replicas}' retention=30d retentionSize= storage= replicas=1 

The container mounted it at /prometheus with no subPath. The pod was 349 days old, the RESTARTS column showed 6 for the pod (it has two containers, 2/2, while step 4 below prints the count of the prometheus container alone), and the pod itself had never been replaced (pod IP removed, node name masked):

$ kubectl -n monitoring get pod prometheus-kube-prometheus-stack-prometheus-0 -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES prometheus-kube-prometheus-stack-prometheus-0 2/2 Running 6 (23h ago) 349d <pod IP> server-1 <none> <none> created=2025-11-30T06:09:35Z 

How much was in it, and on which disk

$ kubectl -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- sh -c 'du -sh /prometheus; df -h /prometheus; ls /prometheus | wc -l' 22.7G /prometheus Filesystem Size Used Available Use% Mounted on <node root LVM volume> 454.9G 249.7G 185.6G 57% /prometheus 23 

The 23 entries are 19 block directories plus chunks_head, lock, queries.active and wal. df shows the node's root filesystem, because an emptyDir lives under the kubelet's directory on the node's disk. Prometheus shared that disk with everything else on server-1.

Prometheus's own metrics at 00:31:36 (labels removed):

prometheus_tsdb_lowest_timestamp_seconds 1791979200 # 2026-10-14T12:00:00Z (time() - prometheus_tsdb_lowest_timestamp_seconds)/86400 31.521971956019602 prometheus_tsdb_head_series 218919 prometheus_tsdb_storage_blocks_bytes 23835478583 prometheus_tsdb_wal_storage_size_bytes 393179565 prometheus_tsdb_retention_limit_bytes 0 prometheus_tsdb_compactions_failed_total 0 prometheus_tsdb_wal_corruptions_total 1 prometheus_tsdb_blocks_loaded 19 

prometheus_tsdb_retention_limit_bytes 0 means no size cap: only the 30-day time limit deleted data.

The restart the day before

On 2026-11-14 the node rebooted; the Prometheus containers restarted at 01:12:30 (reason Unknown). Two minutes later Prometheus logged, as collected by the kubegap log scan of that pod:

2026-11-14T01:14:32.965Z level=WARN source=db.go:1019 msg="Encountered WAL read error, attempting repair" component=tsdb err="read records: c… 2026-11-14T01:14:32.965Z level=WARN source=wlog.go:413 msg="Starting corruption repair" component=tsdb segment=8483 offset=19763238 

(The err text is cut in the scan; the pod and its full log no longer exist.) This is the WAL corruptions 1 in the finding.

Why: the prometheus operator emptyDir default puts the TSDB in the pod

No storage means emptyDir, in the operator code

kube-prometheus-stack 79.5.0 ships storageSpec: {} in its values.yaml, and our values file never set it (git log -S storageSpec on that file is empty back to its first commit, 2025-11-11). With nothing there, the Prometheus object has no spec.storage, and prometheus-operator v0.86.2 (pkg/prometheus/server/statefulset.go, lines 91–130) does this:

storageSpec := cpf.Storage switch { case storageSpec == nil: statefulset.Spec.Template.Spec.Volumes = append(statefulset.Spec.Template.Spec.Volumes, v1.Volume{ Name: prompkg.VolumeName(p), VolumeSource: v1.VolumeSource{ EmptyDir: &v1.EmptyDirVolumeSource{}, }, }) ... default: // storageSpec.VolumeClaimTemplate pvcTemplate := operator.MakeVolumeClaimTemplate(storageSpec.VolumeClaimTemplate) if pvcTemplate.Name == "" { pvcTemplate.Name = prompkg.VolumeName(p) } ... statefulset.Spec.VolumeClaimTemplates = append(statefulset.Spec.VolumeClaimTemplates, *pvcTemplate) } 

VolumeName is prometheus-<name>-db (pkg/prometheus/common.go, lines 177–183), which is our prometheus-kube-prometheus-stack-prometheus-db. The operator documentation says the same in one sentence: "By default, the operator configures Pods to store data on emptyDir volumes which aren't persisted when the Pods are redeployed."

What an emptyDir survives, and when "Prometheus data lost after pod restart" is true

The Kubernetes documentation for emptyDir: "When a Pod is removed from a node for any reason, the data in the emptyDir is deleted permanently." and "A container crashing does not remove a Pod from a node. The data in an emptyDir volume is safe across container crashes." The data is deleted when a new pod is created; a restart of the Prometheus container inside the same pod keeps the emptyDir.

Which everyday events keep an emptyDir and which delete it. Both columns happened on this Prometheus, one day apart. The node reboot is in the left column because on 2026-11-14 ours restarted the containers in the same pod and the emptyDir stayed.

Prometheus does not replicate its data. Its storage documentation (v3.7.3) says local storage "is not clustered or replicated" and "should be managed like any other single node database". With one replica and an emptyDir, one pod replacement removes all of it.

Adding storageSpec: "recreating StatefulSet because the update operation wasn't possible"

Kubernetes does not allow volumeClaimTemplates to change on an existing StatefulSet, so when storage appears on a running Prometheus the update is rejected with 422, and the operator (pkg/prometheus/server/operator.go, lines 984–1003) deletes the StatefulSet and creates it again:

if ok && sErr.ErrStatus.Code == 422 && sErr.ErrStatus.Reason == metav1.StatusReasonInvalid { ... logger.Info("recreating StatefulSet because the update operation wasn't possible", "reason", strings.Join(failMsg, ", ")) propagationPolicy := metav1.DeletePropagationForeground if err := ssetClient.Delete(ctx, sset.GetName(), metav1.DeleteOptions{PropagationPolicy: &propagationPolicy}); err != nil { return fmt.Errorf("failed to delete StatefulSet to avoid forbidden action: %w", err) } continue } 

Foreground propagation deletes the pod before the StatefulSet is gone, and the emptyDir goes with the pod. We saw exactly this log line during the move (next sections). So adding storageSpec and syncing, with nothing else, starts Prometheus on an empty PVC and deletes the old TSDB.

A Red Hat blog post, as quoted in a search result summary (the page itself answered 404 when we fetched it), says "it is not possible to retain data that was written on the default EmptyDir-based installation". Our move contradicts that, and the reason is in how a StatefulSet names its PVCs: the PVC a StatefulSet pod uses is named <volumeClaimTemplate name>-<pod name>, and a PVC that already exists under that name is used as it is. If you create it first and put the TSDB in it, the recreated pod opens the old data.

Where the files have to sit: prometheus-db/

The mount path is /prometheus in both cases, but the subPath is not (pkg/prometheus/common.go, lines 197–204):

func SubPathForStorage(s *monitoringv1.StorageSpec) string { if s == nil || s.DisableMountSubPath { return "" } return "prometheus-db" } 

Without storage the emptyDir is mounted at its root; with a volumeClaimTemplate the pod mounts the prometheus-db directory of the PVC. A copy into the root of the PVC would be invisible to Prometheus.

What "Encountered WAL read error, attempting repair" dropped on 11-14

In Prometheus v3.7.3 the repair that follows Encountered WAL read error, attempting repair discards all data after the corruption point: it deletes every WAL segment newer than the corrupted one and rewrites the corrupted one only up to the offset (tsdb/wlog/wlog.go line 399; the log lines at 413, 421 and 443). So the samples after segment 8483, offset 19763238, were lost, while the 19 blocks on disk were not touched. We did not measure how many samples that was.

No size cap on a shared disk

With no size cap (the prometheus_tsdb_retention_limit_bytes 0 above) on a filesystem shared with the rest of server-1, nothing would have stopped the TSDB from filling the node disk; that is why the same change sets retentionSize.

What to check in your cluster now

Steps 1 to 4 are read-only and need kubectl and jq. The names are the kube-prometheus-stack defaults for release name kube-prometheus-stack in namespace monitoring; change them to yours.

1. Where does the TSDB live?

kubectl -n monitoring get prometheus -o jsonpath='{range .items[*]}{.metadata.name}{" storage="}{.spec.storage}{" retentionSize="}{.spec.retentionSize}{"\n"}{end}' kubectl -n monitoring get sts -l app.kubernetes.io/name=prometheus -o json | jq -r '.items[] | .metadata.name, " volumeClaimTemplates: \([.spec.volumeClaimTemplates[]?.metadata.name])", (.spec.template.spec.volumes[] | select(.name|test("-db$")) | " \(.name): \(if .emptyDir then "emptyDir" else "not emptyDir" end)")' 

Same problem if: storage= is empty, the -db volume says emptyDir, and volumeClaimTemplates: []. Ours: storage= retentionSize= and prometheus-kube-prometheus-stack-prometheus-db: emptyDir.

2. How much is in it?

kubectl -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- sh -c 'du -sh /prometheus; df -h /prometheus' 

Same problem if: step 1 said emptyDir; then everything du prints is lost the next time the pod is deleted. df shows which disk it shares, but it cannot tell an emptyDir from a local-path PVC: on k3s the local-path PVC we moved to sits on the same node disk as the emptyDir did. Ours: 22.7G, on a 454.9G root filesystem 57% used.

3. How far back does it go, and is there a size cap?

In the Prometheus UI or with promtool query instant:

(time() - prometheus_tsdb_lowest_timestamp_seconds) / 86400 prometheus_tsdb_retention_limit_bytes 

Same problem if: the second value is 0. That means no retentionSize: only time deletes data, and nothing stops the TSDB from filling the disk it shares with the node. Ours: 31.5 days and 0.

4. Has the pod been replaced, and did a restart drop WAL samples?

kubectl -n monitoring get pod prometheus-kube-prometheus-stack-prometheus-0 -o jsonpath='{.metadata.creationTimestamp}{" restarts: "}{.status.containerStatuses[?(@.name=="prometheus")].restartCount}{"\n"}' kubectl -n monitoring logs prometheus-kube-prometheus-stack-prometheus-0 -c prometheus | grep -E "attempting repair|Starting corruption repair" 

A container restart (an OOMKill, for example) keeps the emptyDir; ours on 2026-11-14 was a node reboot that restarted the containers in the same pod, and the emptyDir stayed. Same problem if: step 1 said emptyDir, whatever the values in this step are; they only show how close you have already been. With an emptyDir, the oldest sample cannot be older than the pod. If the creation time is close to your oldest sample, the pod was replaced then and everything before it was deleted with the old pod. If the creation time is older than your retention, every restart so far kept the data, and the next pod replacement deletes all of it. A repair line in the log means one of those restarts has already dropped the newest WAL samples. Ours: pod created 2025-11-30, last container restart at a node reboot on 2026-11-14, followed by Encountered WAL read error, attempting repair.

Doing this on every run

This problem was found by kubegap, an open-source audit that reads the cluster your kubectl points at and lists operational risks with the evidence for each. kubegap repeats step 1, step 3 and the log check of step 4 on each run: its summary reads the storage kind (emptyDir, or the PVC template and its size), retentionSize, the oldest sample in days, the WAL corruption count and the PVC usage ratio, and reports them as the findings prometheus.storage-risk, observability.stack-storage-emptydir and logs.signatures.prometheus.wal-corrupt. It does not read the pod creation time or restart count of step 4. After the move, a full run at 00:44:32 no longer reported any of the three for Prometheus:

kubegap full runs collected 2026-11-14 16:25 UTC (the report, v0.3.0) and 2026-11-15 00:44 UTC (after the move, one commit after v0.3.0: v0.3.0-1-g8c36a92).

One gap remains in that run: its PVC usage value (pvc_used_ratio) is null, because the kubelet exports no kubelet_volume_stats_* series for this PVC. At 00:50:02 only 2 of the cluster's 39 PVs had one. We did not investigate why the kubelet reports nothing for these local volumes.

What we changed, and what we have not

Changed: kube-prometheus-stack persistent storage, or how to add persistent volume to Prometheus without losing data

Prometheus is installed by the Argo CD Application monitoring from the chart values in the manifests repository, with automated prune: true and selfHeal: true, so the change had to go through Git. We filled the PVC before the commit, because the sync would delete the pod (see "Adding storageSpec" under Why) and with it the only copy of 31 days of metrics.

Where the files went so the recreated pod found them. One copy pod on the same node mounted the old emptyDir from the host, read-only, and the new PVC.

1. The PVC and a copy pod (00:32:52). Applied with kubectl for the move only, not kept in Git (server-1 and the pod uid are masked):

apiVersion: v1 kind: PersistentVolumeClaim metadata: name: prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0 namespace: monitoring spec: storageClassName: local-path-server-1 accessModes: [ReadWriteOnce] resources: requests: storage: 50Gi --- apiVersion: v1 kind: Pod metadata: name: prometheus-data-copy namespace: monitoring spec: nodeSelector: kubernetes.io/hostname: server-1 restartPolicy: Never containers: - name: copy image: alpine:3.20 command: ["sh", "-c", "apk add --no-cache rsync >/dev/null && mkdir -p /dst/prometheus-db && sleep infinity"] volumeMounts: - { name: src, mountPath: /src, readOnly: true } - { name: dst, mountPath: /dst } volumes: - name: src hostPath: path: /var/lib/kubelet/pods/<pod uid>/volumes/kubernetes.io~empty-dir/prometheus-kube-prometheus-stack-prometheus-db type: Directory - name: dst persistentVolumeClaim: claimName: prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0 

The pod uid comes from kubectl -n monitoring get pod prometheus-kube-prometheus-stack-prometheus-0 -o jsonpath='{.metadata.uid}'. The StorageClass local-path-server-1 is a k3s local-path StorageClass: it uses the rancher.io/local-path provisioner, keeps its volumes on the same node disk as the emptyDir, and has binding mode WaitForFirstConsumer, so the PVC was bound only once a pod used it (here the copy pod); the volume was provisioned at 00:33:01.

2. First copy while Prometheus kept running (00:33:20–00:37:53).

$ kubectl -n monitoring exec prometheus-data-copy -- rsync -a --delete --stats --exclude=lock /src/ /dst/prometheus-db/ Number of files: 158 (reg: 116, dir: 42) Number of regular files transferred: 116 Total transferred file size: 24,396,762,944 bytes sent 24,414,426,021 bytes received 2,562 bytes 89,266,649.30 bytes/sec 

4 min 33 s at 89 MB/s. The second run used the same command.

3. Second copy right before the switch (00:38:04–00:38:05). Only the WAL and head chunks had changed:

Number of files: 158 (reg: 116, dir: 42) Total transferred file size: 108,133,626 bytes 22.7G /dst/prometheus-db 

4. The values change (pushed 00:38:05, Argo CD refresh requested 00:38:08). Comment lines omitted:

 prometheus: prometheusSpec: retention: 30d + retentionSize: 40GB + storageSpec: + volumeClaimTemplate: + spec: + storageClassName: local-path-server-1 + accessModes: ["ReadWriteOnce"] + resources: + requests: + storage: 50Gi 

5. The operator recreated the StatefulSet (00:38:26), as the source says it would:

ts=2026-11-15T00:38:26.821426997Z level=info caller=/workspace/pkg/prometheus/server/operator.go:996 msg="recreating StatefulSet because the update operation wasn't possible" component=prometheus-controller key=monitoring/kube-prometheus-stack-prometheus statefulset=prometheus-kube-prometheus-stack-prometheus shard=0 reason="Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'revisionHistoryLimit', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden" 

The old pod's containers were stopped (Killing) the same second, and the new pod was created on server-1 at 00:38:30. Its data mount and the new argument:

"dbMount": { "mountPath": "/prometheus", "name": "prometheus-kube-prometheus-stack-prometheus-db", "subPath": "prometheus-db" } args: ... "--storage.tsdb.retention.time=30d","--storage.tsdb.retention.size=40GB","--storage.tsdb.path=/prometheus" ... 

6. Prometheus opened the copied TSDB. From the new pod's startup log (19 Found healthy block lines, the first shown):

00:38:32.659Z msg="Found healthy block" component=tsdb mint=1791979200038 maxt=1792173600000 ulid=01M53F5PAQ7C40DZXPAKF6RB13 00:38:33.786Z msg="Replaying WAL, this may take a while" component=tsdb 00:39:21.785Z msg="WAL replay completed" component=tsdb checkpoint_replay_duration=2.889331339s wal_replay_duration=45.109577526s ... total_replay_duration=48.613000449s 00:39:22.360Z msg="TSDB started" 00:39:22.867Z msg="Server is ready to receive web requests." 

The mint of the first block is the old lowest timestamp, 1791979200. The startup probe failed once with 503 during the replay (00:39:15); the pod was Ready at 00:39:32. At 00:39:55: du -sh /prometheus 22.7G, lowest timestamp 1791979200 (2026-10-14T12:00:00Z), 19 blocks, prometheus_tsdb_retention_limit_bytes 42,949,672,960, prometheus_tsdb_wal_corruptions_total 0. The copy pod was deleted at 00:40:09.

The proof: deleting the pod on purpose

To check that a new pod keeps the data, we deleted the pod at 00:41:41 and compared the same values before and after. Before, at 00:41:40: lowest timestamp 1791979200, 19 blocks, 22.7G.

$ kubectl -n monitoring delete pod prometheus-kube-prometheus-stack-prometheus-0 # 00:41:41 pod "prometheus-kube-prometheus-stack-prometheus-0" deleted from monitoring namespace 

The new pod was created at 00:41:43 on the same PVC (Bound, 11 min old), replayed the WAL in 54.4 s, logged TSDB started at 00:42:44 and was ready by 00:43:03. At 00:43:57 all five values were the same as at 00:39:55.

Prometheus at 00:44:17, after both pod replacements: the 5-week graph still starts around Oct 14. The colour changes at the right end because the instance label (the pod IP) changed with each new pod.

What it cost

What the move cost, measured from Prometheus scraping itself every 30 s: two holes in new samples; the stored 30 days stayed.

The self-scrape series up{job="kube-prometheus-stack-prometheus"} has no samples from 00:37:45 to 00:39:51 (2 min 6 s) and from 00:41:21 to 00:42:51 (1 min 30 s). The second hole is the deliberate restart: nothing was scraped while the pod was down, and nothing stored was lost. For the first hole, the inference from the times above: the last sample before the second copy was scraped at 00:37:45 and is in the copy; the next one, due about 30 s later, was written after the copy finished at 00:38:05, while the old pod ran until 00:38:26, so it stayed in the old emptyDir and was deleted with it; after that nothing scraped until the new pod was up. A third rsync between the push and the recreate would have narrowed it; we did not try.

Why 50Gi and retentionSize: 40GB

40GB in retentionSize is 40 GiB: Prometheus documents the units as "Based on powers-of-2, so 1KB is 1024B", the operator's ByteSize type says the same, and the metric shows 42,949,672,960 bytes. That is 80% of the 50Gi request, within the Prometheus documentation's advice "setting the retention size to, at most, 80-85% of your allocated Prometheus disk space". With both limits set, whichever triggers first is used; the current 22.7G is well below either.

On this StorageClass the 50Gi is not a limit. The local-path-provisioner README: "No support for the volume capacity limit currently. The capacity limit will be ignored for now." So retentionSize is the only cap actually enforced, and beyond it only the free space on the node disk limits the TSDB. One more limit from the Prometheus docs: "Only the persistent blocks are deleted to honor this retention although WAL and m-mapped chunks are counted in the total size", so a growing WAL is counted but not shrunk by the cap.

Not changed, and why

• Alertmanager and Grafana are still on emptyDir. kubegap reports observability.stack-storage-emptydir for both. They were not part of problem 2 in the kubegap report; we expect the same copy-then-switch method to work for them but have not tried it.
• The PVC is on the same disk of the same node as the emptyDir was. The data now survives a pod replacement, not the loss of the disk or of server-1. Copies outside the node and the cluster are problem 1 of the same kubegap report and are not covered here. The StorageClass also has reclaim policy Delete; the StatefulSet's persistentVolumeClaimRetentionPolicy is Retain for both deletion and scale-down, so deleting the StatefulSet keeps the PVC, but deleting the PVC by hand deletes the data.
• No volume-filling alert for the new PVC. There is no usage metric for it (the kubelet exports none), so neither a KubePersistentVolumeFillingUp alert nor kubegap sees it fill. retentionSize is the cap that is enforced; the existing node disk alert rules were not changed.
• Prometheus memory and CPU. Memory was 1081Mi of the 1536Mi limit before the move, and the startup probe allows 60 × 15 s = 15 minutes; WAL replays here took 45 s and 54 s. CPU throttling and the memory limit are other items of the same kubegap report.

Where else this happened

The Incident Library, a searchable index of public postmortems on benchlet.com (links below), has no case of a Prometheus TSDB deleted because it ran on an emptyDir. The closest cases are Prometheus restarts that went wrong at the WAL replay or the probes. This move restarts Prometheus twice and replays the WAL each time (45 s and 54 s here, under a startup probe that allows 15 minutes), so if your WAL is larger, check the startup probe before the switch:

• UK Ministry of Justice, 2024-04-15: a planned change restarted Prometheus, the WAL reload took longer than the startup probe allowed, and the platform had no monitoring for 3 h 21 min until the probe was raised to 30 minutes.
• Saifeddine Rajhi, 2024-06-14 (personal blog): after a kube-prometheus-stack upgrade Prometheus was OOMKilled in a loop while replaying its WAL, which needed more than 50Gi of memory against about 30Gi in steady state.
• UK Ministry of Justice, 2023-11-01: Prometheus on the live cluster restarted several times after readiness probe failures, leaving gaps in the metrics.

More: Prometheus + monitoring down, 8 cases. Cases where data on a pod or node was gone after a change, the outcome this move avoided, are in data loss on Kubernetes, 5 cases.

Timeline

FAQ

prometheus data lost after pod restart

It depends on what restarted, because with no storageSpec the prometheus operator emptyDir default stores the TSDB in a volume that is deleted with the pod, not when a container restarts. A container restart (OOMKill, liveness failure) keeps the emptyDir, so the blocks stay; ours was a node reboot that restarted the containers in the same pod, and the emptyDir stayed. Even then, a WAL repair after an unclean stop can still drop the newest samples. A new pod gets an empty emptyDir: drain, eviction, kubectl d...