Kubernetes CronJob pg_dump backup: does the dump end up on the same node as the database?

Every backup Job completed. Five of the seven dumps were on the same disk as the databases they copy.

If your Kubernetes CronJob pg_dump backup writes to a PVC of the local-path StorageClass (the k3s default) and kubectl get jobs -A shows every backup Job as Complete 1/1, that output does not tell you which node's disk the dump is on. On our cluster all seven backup Jobs completed on 2026-11-11, but five of the dumps were on the same single disk as the databases they copy, and every k3s etcd snapshot and the server token needed to restore them were on that disk too. A local-path volume is created on the node where the first pod that uses it is scheduled, and in five of our seven backup CronJobs nothing placed that pod on a node other than the database's. To see yours, list each PV with its node (check 1 below): if a backup claim and the database's data claim show the same NODE, the dump is on the database's node, and on the same disk if that node has only one. The cluster runs k3s v1.33.5+k3s1 on two computers: one server, which holds etcd and all seven backed-up databases on a single SSD, and one agent. All times are UTC.

What we saw

server-1 is the only server and has one 500 GB SATA SSD, no RAID; agent-1 is the agent, on the same network. Storage is the k3s built-in local-path provisioner; applications are deployed by Argo CD. We captured this state on 2026-11-12 from 01:25 UTC; names are masked and columns are trimmed.

Seven backup CronJobs (six pg_dump, one mysqldump) run daily between 02:50 and 03:50. The latest run of each (the output had three Jobs per CronJob and one old Job created by hand; only the newest Job of each CronJob is kept here, and the CONTAINERS, IMAGES and SELECTOR columns are cut):

$ kubectl get jobs -A NAMESPACE NAME STATUS COMPLETIONS DURATION AGE prod-a app-a-postgres-backup-29906115 Complete 1/1 9m7s 22h prod-b app-b-postgres-backup-29906130 Complete 1/1 4s 21h prod-b app-b-postgres-3-backup-29906130 Complete 1/1 4s 21h prod-b app-b-postgres-2-backup-29906130 Complete 1/1 4s 21h prod-c app-c-postgres-backup-29906150 Complete 1/1 4s 21h prod-d app-d-mysql-backup-29906090 Complete 1/1 11s 22h prod-e app-e-postgres-backup-29906100 Complete 1/1 4s 22h 

Each job writes a gzip dump (<name>_<timestamp>.sql.gz; the mysqldump job <name>-<date>-<time>.sql.gz) to a local-path PVC mounted at /backup and deletes older files. No command in any of the seven sends anything outside the cluster: there is no s3, rclone, restic, scp or rsync in them.

kubectl get pv at 2026-11-12 01:25 UTC. Red: a dump volume on the same node as its database, 5 of 7.

The same volumes, next to the node of each source database pod:

All seven source databases are on server-1, whose one disk holds a single filesystem at / with the etcd database, the etcd snapshots and the local-path volumes. Every PV had reclaim policy Delete.

All five k3s etcd snapshots, file:// on the server's own disk.

The five snapshots are etcd-snapshot-server-1-<unix time>, 41,775,136 to 45,232,160 bytes each. The k3s unit has no --etcd-s3* argument and there is no /etc/rancher/k3s/config.yaml. The server token, /var/lib/rancher/k3s/server/token (mode 600, root), is on the same disk. A cleanup CronJob keeps only the 5 newest snapshot files, daily at 03:00.

One disk failure on server-1 would have removed the etcd database, all 5 snapshots and the token, every source database, and 5 of the 7 dumps (inference from the facts above). Only the app-a and app-b-postgres-3 dumps on agent-1 would have been left.

Why local-path created the dump volumes on the database's node

local-path creates the volume where the first pod runs. Both StorageClasses use WaitForFirstConsumer: the volume is created on the node where the first pod that uses the claim is scheduled. The second class is named after the server, but nothing in it pins a volume to that node: no allowedTopologies, only a path.

# kubectl get storageclass local-path-server-1 -o yaml (trimmed) allowVolumeExpansion: false apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: local-path-server-1 parameters: path: /var/lib/rancher/k3s/storage provisioner: rancher.io/local-path reclaimPolicy: Delete volumeBindingMode: WaitForFirstConsumer 

After that the PV is fixed to the node. From the local-path-provisioner README: PersistentVolumes created by the provisioner "use the kubernetes.io/hostname label for node affinity. This ensures the volume is only accessible on the node where it was provisioned."

Only one of the seven jobs was placed on another node on purpose.

• app-a-postgres-backup has nodeSelector: kubernetes.io/hostname: agent-1. It was created on 2026-07-02 with that selector on purpose, and it was the only backup placed away from its database by design.
• app-b-postgres-3-backup had no nodeSelector and used local-path-server-1. Its volume is on agent-1 only because its first pod was scheduled there (inference from the binding rule above).
• app-c-postgres-backup had nodeSelector: storage: server-1, which pinned its dump to the database's node.
• app-d-mysql-backup had nodeSelector: kubernetes.io/hostname: server-1 and wrote to a claim from the database StatefulSet's own volumeClaimTemplate, so its dump volume was created with the database pod, on its node.
• app-b-postgres-backup, app-b-postgres-2-backup and app-e-postgres-backup had no nodeSelector, and their dump volumes were created on server-1, where their databases run.

A Complete Job status does not record which disk the dump was written to.

k3s keeps etcd snapshots on the server unless told otherwise. From the k3s etcd-snapshot docs: "Scheduled snapshots are enabled by default, at 00:00 and 12:00 system time, with 5 snapshots retained per server node on local disk" and "K3s etcd snapshots are stored on the node file system, and may optionally be uploaded to an S3-compatible object store for disaster recovery scenarios." We had not set --etcd-s3, and server-1 is the only server, so all five snapshots were on one disk.

The snapshot is unusable without the token. From the k3s backup and restore docs: "you must also back up the server token file at /var/lib/rancher/k3s/server/token" and "If you do not use the same token value when restoring, the snapshot will be unusable". A copy of the snapshots alone would not have been enough.

The snapshot does not hold volume contents. It is "a complete copy of the etcd datastore, along with cluster certificate authority (CA) certificates and private keys" (k3s docs), so it has the API objects but not the data in volumes (inference from what etcd stores): the dumps and the snapshots have to be moved separately.

Check where your Kubernetes CronJob pg_dump backup is stored

These are read-only, except checks 5 and 6, which create throwaway objects. kubegap, an audit tool that reads the cluster of your current kube context, found this problem here on 2026-11-11 by reading the same three things as checks 1 to 3. It reports all three in one run, so you can run it instead of pasting those three commands. Checks 4 to 6 are run by hand.

1. Which node holds each dump, and which node holds the database it copies?

kubectl get pv --sort-by=.spec.claimRef.namespace -o custom-columns='NS:.spec.claimRef.namespace,PVC:.spec.claimRef.name,NODE:.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0],CLASS:.spec.storageClassName,RECLAIM:.spec.persistentVolumeReclaimPolicy' 

Same problem if: the row of a backup PVC and the row of the database's data PVC show the same NODE, and that node has one disk (lsblk -d -o NAME,TYPE,SIZE on the node lists one row of TYPE disk; on server-1 it was sda disk 465.8G). Read NODE, not CLASS: with local-path a StorageClass name does not choose the node; the first pod that uses the claim does (WaitForFirstConsumer). RECLAIM Delete means deleting the claim deletes the directory on the node.

2. Does any backup job send its result to another machine?

kubectl get cronjob -A -o json | jq -r '.items[] | select(.metadata.name | test("backup|dump|snapshot")) | [.metadata.namespace, .metadata.name, (.spec.jobTemplate.spec.template.spec.nodeSelector // {} | tostring), ([.spec.jobTemplate.spec.template.spec.containers[] | ((.command // []) + (.args // [])) | join(" ")] | join(" ") | test("s3|rclone|restic|kopia|scp|rsync") | tostring)] | @tsv' 

On our cluster, before the change:

prod-a app-a-postgres-backup {"kubernetes.io/hostname":"agent-1"} false prod-b app-b-postgres-backup {} false prod-b app-b-postgres-3-backup {} false prod-b app-b-postgres-2-backup {} false prod-c app-c-postgres-backup {"storage":"server-1"} false prod-d app-d-mysql-backup {"kubernetes.io/hostname":"server-1"} false prod-e app-e-postgres-backup {} false 

Same problem if: the last column is false and the nodeSelector column is empty or names the database's node. Ours had all seven false, four with {} and two naming server-1. The match reads only the command and args, so true means "look closer", and a script mounted from a ConfigMap shows false even if it uploads: open the script before you conclude.

3. Where are the etcd snapshots? (k3s with embedded etcd)

kubectl get etcdsnapshotfile 

Same problem if: every LOCATION starts with file:// and nothing copies that directory to another machine (check 2). Those snapshots are on the server's own disk, next to the etcd database they copy.

4. Did you back up the server token anywhere but the server?

On the server: sudo ls -l /var/lib/rancher/k3s/server/token. Then look for the same file where your snapshot copies go.

Same problem if: the token exists only on the server. Without the same token value, a restored snapshot is unusable (k3s docs, backup and restore).

5. Does the newest dump restore from the other machine?

A Job in the dump's namespace that mounts the dump PVC read-only (it is scheduled on the node that holds the PVC), starts an empty server in an emptyDir, restores with ON_ERROR_STOP, and counts tables. The Job below is for PostgreSQL; use the image your backup uses (uid 70 is postgres in the alpine image):

apiVersion: batch/v1 kind: Job metadata: {name: restore-test, namespace: <ns>} spec: backoffLimit: 0 ttlSecondsAfterFinished: 3600 activeDeadlineSeconds: 900 template: spec: restartPolicy: Never securityContext: {runAsUser: 70, runAsGroup: 70, fsGroup: 70} containers: - name: restore image: postgres:16-alpine command: ["/bin/sh", "-c"] args: - | set -eu F=$(ls -t /backup/*.sql.gz | head -1); echo "latest $F" gzip -t "$F" initdb -D /work/pg -U postgres >/dev/null pg_ctl -D /work/pg -o "-c listen_addresses='' -c unix_socket_directories=/work" -w start >/dev/null createdb -h /work -U postgres restored zcat "$F" | psql -h /work -U postgres -d restored -v ON_ERROR_STOP=1 -q psql -h /work -U postgres -d restored -Atc "select count(*) from pg_tables where schemaname not in ('pg_catalog','information_schema')" volumeMounts: - {name: backup, mountPath: /backup, readOnly: true} - {name: work, mountPath: /work} volumes: - {name: backup, persistentVolumeClaim: {claimName: <dump-pvc>, readOnly: true}} - {name: work, emptyDir: {}} 

Then kubectl get pod -n <ns> -l job-name=restore-test -o wide shows the node it ran on, and the table count should equal the source's (select count(*) from pg_tables where schemaname not in ('pg_catalog','information_schema') in the source database).

Same problem if: the Job ran on the database's node, fails, or the count differs.

6. Does the etcd copy restore?

Copy the newest snapshot and the token from the other machine to a third one with Docker, check the sha256 against the copy, then restore with the same k3s version and no agent (no kubelet, so no workload from the snapshot can start):

docker run --rm --privileged -e K3S_TOKEN="$(cat token)" -v k3s-test:/var/lib/rancher/k3s -v "$PWD":/snap:ro \ rancher/k3s:<your version, e.g. v1.33.5-k3s1> server --disable-agent \ --cluster-reset --cluster-reset-restore-path=/snap/<snapshot file> docker run -d --name k3s-test --privileged -e K3S_TOKEN="$(cat token)" -v k3s-test:/var/lib/rancher/k3s \ rancher/k3s:<same version> server --disable-agent docker exec k3s-test kubectl get ns,deploy,sts,cronjob -A --no-headers | wc -l docker rm -f k3s-test; docker volume rm k3s-test 

The first command prints "Managed etcd cluster membership has been reset, restart without --cluster-reset flag now" and exits. Same problem if: it exits with an error, or the object counts are far from the live cluster's (objects created after the snapshot account for small differences). Delete the copied snapshot afterwards: it contains every Secret of the cluster.

What we changed and what we have not

What was on each machine before and after. Red: lost together with the database if server-1's disk fails.

Changes 2 and 3 went through the manifests repository and Argo CD. Change 1 was made with kubectl patch and recorded in the repository afterwards. The node settings on server-1 are recorded in the README next to the CronJob.

1. Retain on 30 of the 32 PVs (2026-11-12 01:35). We set every PV outside the logging and monitoring namespaces to Retain with kubectl patch pv <name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}', and no pod restarted. Retain keeps the node directory when a claim is deleted and does nothing when the disk fails, so it is not the fix for this problem.

2. Every dump Job runs on agent-1. All seven CronJobs now have:

spec: jobTemplate: spec: template: spec: nodeSelector: kubernetes.io/hostname: agent-1 

The five that were on server-1 write to a new PVC of class local-path (app-b-postgres-backup-agent-1, app-b-postgres-2-backup-agent-1, app-c-postgres-backup-agent-1, app-d-mysql-backup-agent-1, app-e-postgres-backup-agent-1). Because of WaitForFirstConsumer, each is created on agent-1, where its first pod runs. Changing only the nodeSelector would not have moved the old volumes: an existing local-path PV keeps its node affinity, so a pod pinned elsewhere could not mount it (inference from the provisioner's node affinity rule). Retention is now 7 days for all seven (find -mtime +6 -delete).

3. The etcd snapshots and the token are pulled to agent-1. A CronJob kube-system/etcd-snapshot-backup on agent-1 runs at 30 0,12 * * *, 30 minutes after each k3s snapshot, and pulls /var/lib/rancher/k3s/server/db/snapshots/ and /var/lib/rancher/k3s/server/token from server-1 with rsync over SSH into the PVC etcd-snapshot-backup-agent-1, kept 7 days. Its name contains "backup", so the existing alert BackupCronJobStale (no success for 26 hours) covers it.

On server-1, root may log in over SSH only with a forced-command key and only from agent-1 and the pod network:

# /etc/ssh/sshd_config.d/10-etcd-snapshot-backup.conf Match Address 10.0.0.11,10.42.0.0/16 PermitRootLogin forced-commands-only 

# /root/.ssh/authorized_keys (one line; key not shown) restrict,from="10.0.0.11,10.42.0.0/16",command="/usr/bin/rrsync -ro /var/lib/rancher/k3s/server/" ssh-ed25519 <public key> etcd-snapshot-backup@agent-1 

rrsync -ro limits the key to read-only rsync under /var/lib/rancher/k3s/server/. sshd -t passed. sshd -T -C user=root,host=x,addr=<address> was compared before and after for two addresses: 203.0.113.9, outside the Match, was unchanged, and 10.0.0.11 changed only in permitrootlogin (no to forced-commands-only). Addresses in 10.42.0.0/16 are inside the Match too. The private key is kept in the secret manager the cluster already uses and reaches the pod as a Secret through External Secrets.

The first copy failed. All three attempts of the Job, on agent-1, ended the same way:

ssh: connect to host 10.0.0.12 port 22: Operation timed out rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] rsync error: unexplained error (code 255) at io.c(232) [Receiver=3.4.1] 

The agent-1 node itself could not reach server-1:22 either. The ufw rows on server-1 before the change:

22/tcp ALLOW IN 100.64.0.0/10 Anywhere ALLOW IN 10.42.0.0/16 Default: deny (incoming), allow (outgoing), allow (routed) 

That allows 22/tcp from the Tailscale range and every port from the pod network, but nothing from agent-1's own address, 10.0.0.11. The pod's connection timed out although the pod network was allowed, so it reached server-1 from 10.0.0.11 (inference). One rule fixed it:

ufw allow from 10.0.0.11 to any port 22 proto tcp comment "etcd-snapshot-backup from agent-1" 

The next run copied 5 snapshots and the token to agent-1.

Verification (2026-11-12 17:24 to 17:55). Each backup CronJob was run once by hand after Argo CD applied the change: all seven ran on agent-1 and completed (app-a-postgres-backup, the largest, in 9 minutes). Every backup PVC in use is now on agent-1.

The newest dump of each of the five moved databases was then restored from its PVC on agent-1 into an empty server (psql with ON_ERROR_STOP=1, or mysql), and compared with counts taken inside the source pod at the same minute:

Five dumps restored from agent-1 at 17:42 UTC. Table and row counts equal the source; the MySQL source rows were not counted.

The etcd copy was restored the same way as check 6. The newest snapshot (12:00) and the token were copied from agent-1 with a matching sha256 and restored with the same k3s version (--cluster-reset --cluster-reset-restore-path, then a normal start) in a container on a third machine, with the agent disabled so nothing from the snapshot could run:

The 12:00 snapshot copy restored by k3s v1.33.5 with no agent.

Restored versus live: namespaces 28/28, deployments 68/68, statefulsets 22/22, cronjobs 29/30, secrets 145/147, configmaps 153/154, persistentvolumes 32/38. Every difference is an object created after the 12:00 snapshot: the new CronJob, two Secrets (the copy's SSH key and one from an unrelated change that day), the copy's known_hosts ConfigMap and the six new PVCs. The only processes in the container were k3s and its supervisor.

Three earlier attempts failed on the test script, not the copy; you may hit them with check 6:

• kubectl exec ... cat of the snapshot returned 43,974,656 bytes instead of 45,989,920; kubectl cp with a sha256 check worked.
• With --network none or an internal Docker network, k3s exited with no default routes found; the default bridge network worked.
• With --token-file, k3s exited with /var/lib/rancher/k3s/server/token does not exist, please pass --token; K3S_TOKEN worked.

What we have not changed:

• No off-site copy. Both copies are on a second computer in the same house: a disk failure on server-1 no longer takes the backups with it, but a fire, theft or power event that takes both machines would. This is a practice cluster, and I chose to stop at the second machine.
• The old dump PVCs on server-1 are kept. They hold the dumps taken before the change and are no longer written to.
• Two dumps were not restore-tested in this change. app-a and app-b-postgres-3 were already on agent-1; the restore test covered the five that moved.
• Eight PVs are Delete. Read at 18:21 after the change, 30 PVs are Retain and 8 are Delete: the six new backup claims, which got Delete from the StorageClass default, and one volume each in the logging and monitoring namespaces.
• The snapshot copy holds every Secret and the cluster CA keys, so access to agent-1's disk now matters as much as access to server-1's (inference).

Where else this happened

The Incident Library, a searchable collection of public incident reports, has three that show what happened elsewhere when a control plane or a data store depended on a single disk, node, zone or pod; the first is the disk failure that server-1 is exposed to:

• An individual engineer's blog, 2023-05-07 (IL-0185): a disk failure made the first control plane node of a Kubespray cluster read-only, and recovery needed the inventory reordered and a new control plane node joined.
• Devsisters, 2021-02-19 (IL-0172): an AWS Tokyo cooling failure took down 6 of 60 CockroachDB nodes on Kubernetes in 22 minutes; ranges lost quorum because no availability zone locality was configured, and a table backup taken from the surviving nodes did not work, which is the failure that checks 5 and 6 are there to catch before you need the backup.
• Palark, 2020-03-13 (IL-0196): one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the data; as with the five dumps on server-1, the data was in one place.

The library lists 5 reports where etcd was a single point of failure, IL-0185 among them, and 2 where data was lost, the Devsisters and Palark cases above.

Timeline (UTC)

FAQ

kubernetes cron job for postgres db backup

A Kubernetes CronJob pg_dump backup, like a Kubernetes CronJob mysqldump backup, writes its dump to a PVC. With local-path, the claim is created on the first Job pod's node, possibly the database's. Pin the Job with nodeSelector to another node with a new claim: an existing local-path PV stays on its node. Or upload it off the cluster.

kubernetes rancher local path specify node

The StorageClass does not choose the node. With volumeBindingMode: WaitForFirstConsumer, local-path creates the volume on the node where the first pod using the claim is scheduled, and then pins the PV there with kubernetes.io/hostname node affinity. To choose the node, put nodeSelector or node affinity on that first pod before the claim is bound.

where does k3s local-path store data

In /var/lib/rancher/k3s/storage on the node that holds the volume (${HOME}/.rancher/k3s/storage in rootless mode). The node is recorded in the PV's node affinity. Read it with the kubectl get pv command in check 1 above. With reclaim policy Delete, deleting the claim deletes that directory on the node.

How do you verify that PostgreSQL backups on Kubernetes can actually be restored?

Run a Job that mounts the dump's claim read-only, starts an empty PostgreSQL in an emptyDir, restores with psql -v ON_ERROR_STOP=1, and counts tables. Compare with the same count in the source. Check which node the Job ran on: for a backup meant to survive a disk failure, it must not be the database's node.

restore k3s etcd snapshot new machine

Copy the snapshot and the server token file /var/lib/rancher/k3s/server/token to the new machine. Run k3s server --cluster-reset --cluster-reset-restore-path=<file> with the token in K3S_TOKEN (with --token-file, k3s exited with /var/lib/rancher/k3s/server/token does not exist, please pass --token), then start k3s without --cluster-reset. We used the same k3s version. Without the same token the snapshot is unusable.

Originally published at benchlet.com.