Kanister restore drills that end in PROVED or UNPROVED

This is a CI log from 11 October 2026. The backup step finishes normally:

actionset.cr.kanister.io/backup-postgres-20261011073901 created backup-postgres-20261011073901: running backup-postgres-20261011073901: complete -- protect: backup=backup-postgres-20261011073901 loc=pg_backups/demo-postgres/postgres/20261011T073903Z-8979a792-cc74-4761-b9a5-f2c700173fdd/backup.sql.gz sha256=68bd312d6458… (PASS) 

Less than a second later the drill script overwrites that object in the bucket with the 19 bytes NOT-A-GZIP-ARTIFACT. Kanister has no reason to notice. The ActionSet stays complete, and a dashboard that counts successful backup jobs would show this one as green.

A few seconds after that, the validate phase pulls the object back and compares hashes:

sha256 actual=2a107527d87dc4dd1aab241f22e6f0e67f3450b623a60ebe2503fcd1317a1525 expected=68bd312d6458f79e78cee2182814ed9bc51130878a66683a25adbfeb201a9641 sha256 mismatch 

The drill stops there, never starts a restore, and prints UNPROVED with the reason artifact_integrity. That run is the corrupt-artifact failure drill in kanister-backup-restore, a repo I built to run restore drills on a kind cluster and reduce the outcome to one binary verdict. Below is a walk through v1.0.1: what each stage checks, what the corrupt-artifact run looked like, where the measured 8 second RTO comes from, and what that number does not mean.

The contract

The verdict is computed against a short YAML file, config/recovery-contract.yaml. It has no scores or percentages. Every clause either holds or fails.

version: 2 name: recovery-contract lifecycle: PROTECT → VALIDATE → RESTORE → PROVE contract: artifact_integrity: required isolated_restore: required application_ready: required data_verification: required cross_component_consistency: required rto_seconds_max: 60 evidence_freshness_hours_max: 24 recovery_point_skew_seconds_max: 60 components: - postgres - s3 future: - solr 

The demo application has two parts. Postgres holds rows in app_objects(object_key, sha256, generation), and each row points at an object in an S3 bucket. Metadata in a database and blobs in object storage is a common shape, and it can fail in a way a database-only drill never sees: every row comes back, and some of them point at objects that are gone or different. So the contract treats the recovery point as one logical unit across both stores.

The stack

Versions are pinned in one file, config/versions.env:

KANISTER_VERSION=0.118.0 KIND_VERSION=v0.33.0 K8S_VERSION_LATEST=v1.37.0 K8S_VERSION_PREVIOUS=v1.36.4 SEAWEEDFS_VERSION=3.85 POSTGRES_VERSION=17-alpine SHELLCHECK_VERSION=v0.10.0 SHFMT_VERSION=v3.10.0 

The whole happy path is one target:

make full-drill 

It creates the kind cluster, installs the Kanister operator with Helm, deploys SeaweedFS as the S3 endpoint with a Kanister Profile, applies the Blueprints and the RecoveryContract / RecoveryDrill CRDs, seeds the demo app and runs the four stages.

Object storage used to be MinIO. The switch to SeaweedFS 3.85 came in v1.0.0, because the MinIO images the drill pulled were no longer publicly available. S3 credentials reach Blueprint phases only through the Kanister Profile (toJson .Profile), and the Blueprint phases don't run with set -x, so credentials stay out of job logs.

PROTECT: a marker, a unique key and a hash

There are two drill paths. The Postgres-only one (make recovery-drill, also used by corrupt-artifact) writes a proof marker into the source database before each backup: a fresh UUID and a timestamp in a recovery_markers table. In the corrupt-artifact run it was drill-3-cff71037-1b0c-4d9d-b434-d2bb26fcdb0f-2026-10-11T07:39:01Z. The marker pays off later. If the restored database contains this exact row, it came from this exact backup, and a stale or pre-seeded database can't fake that.

The Postgres Blueprint runs pg_dump --clean --if-exists, gzips the output, hashes it and pushes it with kando:

pg_dump --clean --if-exists -U "${PGUSER}" "${PGDATABASE}" | gzip -c > /tmp/backup.sql.gz SHA256="$(sha256sum /tmp/backup.sql.gz | awk '{print $1}')" kando location push /tmp/backup.sql.gz --profile '{{ toJson .Profile }}' --path "${BACKUP_LOCATION}" 

The object path and the hash become Kanister output artifacts (backupLocation, backupSha256). Every backup gets its own key with a timestamp and a UUID, so a new backup can't quietly replace an older one. Pruning is controlled by BACKUP_RETAIN_N, which keeps 5 by default.

The multi-service path, which make full-drill runs, relies on linked state instead of a marker. Its PROTECT stage copies every S3 object the rows reference under recovery-points/<rp_id>/objects/ and writes a recovery-point manifest with the object keys, their sha256 values and a generation number.

VALIDATE: hash, gzip and a scratch restore

Validation is a separate Blueprint action. It pulls the artifact back and stops on any doubt. Trimmed from blueprints/postgres/blueprint.yaml:

ACTUAL_SHA="$(sha256sum /tmp/backup.sql.gz | awk '{print $1}')" if [[ -z "${EXPECTED_SHA}" || "${EXPECTED_SHA}" == "<no value>" ]]; then echo "missing backupSha256 artifact metadata" >&2 exit 1 fi if [[ "${ACTUAL_SHA}" != "${EXPECTED_SHA}" ]]; then echo "sha256 mismatch" >&2 exit 1 fi gzip -t /tmp/backup.sql.gz 

After the hash and gzip -t, it creates a throwaway database named validate_scratch_$$, pipes the dump into psql -v ON_ERROR_STOP=1, and fails if the scratch database ends up with no public tables. A missing hash fails too. An empty backupSha256 counts as a broken backup. Without the metadata needed to check an artifact, there is no basis for calling it good.

The scratch restore adds time to every drill. It is also the only check in this stage that shows Postgres will accept the dump. A hash only proves the bytes didn't change since backup, and a dump can be broken from the start.

RESTORE: a fresh namespace with a quota

The restore never targets the source namespace. Each drill creates a namespace named restore-drill-<timestamp>, labels it justrunme.com/role=restore-drill and applies deploy/drill-quota.yaml:

spec: hard: requests.cpu: "2" requests.memory: 2Gi limits.cpu: "4" limits.memory: 4Gi requests.storage: 10Gi persistentvolumeclaims: "4" pods: "20" 

A LimitRange in the same file sets container defaults, so a drill can't eat the cluster. The Postgres in that namespace starts unseeded. If a restore ActionSet completes without applying anything, the marker check has nothing to find, and that is exactly what the noop-restore scenario tests with a Blueprint whose restore does nothing.

S3 objects are pulled from the recovery point and written into a fresh prefix, restored/<drill-namespace>/<rp_id>, which is where the restored app reads from. The RTO clock covers both halves. From scripts/app-recovery-drill.sh:

# RTO includes Postgres ActionSet restore + S3 restore into the app prefix. END=$(now_epoch) RTO_SECONDS=$((END - START)) # restore_completed requires BOTH Postgres and S3 restore success. if [[ "$RESTORE_OK" == true && "$S3_RESTORE_OK" == true ]]; then checks_restore=true 

The Kanister operator gets scoped RBAC: a namespaced Role in kanister, a RoleBinding per drill namespace, and no cluster-wide access to Secrets. CI checks this with kubectl auth can-i get secrets --namespace default --as=system:serviceaccount:kanister:kanister-operator and fails the job if the answer is yes.

PROVE: the restored app, both stores, one recovery point

PROVE asks three things about the restored copy. Is Postgres up (pg_isready)? Is the data there? Do the two stores agree?

Data verification counts the rows in app_objects, then sends a HEAD request for every object_key under the restore prefix. Zero rows fails, and so does a single missing object.

Consistency lives in scripts/prove_consistency.py. Every object in the recovery-point manifest has to exist with the recorded sha256. The hash in each Postgres row has to match the restored object. The generation has to be the same in app_manifest, app_objects and the manifest. And the time skew between the Postgres and S3 halves of the recovery point has to stay under 60 seconds.

That skew used to be labeled RPO. v1.0.1 renamed rpo_seconds to component_skew_seconds, because without a last-write timestamp the drill can't honestly report an RPO. What it measures is how far apart in time the two halves of one recovery point were captured.

Finally, scripts/evaluate_contract.py turns the evidence JSON into a verdict. It walks the clauses in a fixed order and reports the first one that fails:

clauses = [ ("backup", backup_ok, "required"), ("artifact_integrity", artifact_ok, "required"), ("isolated_restore", restore_ok, "required"), ("application_ready", app_ok, "required"), ("data_verification", data_ok, "required"), ] if consistency_required: clauses.append( ("cross_component_consistency", consistency_ok, "required") ) clauses += [ ("recovery_point_skew", skew_ok, f"< {skew_max}s"), ("rto", rto_ok, f"< {rto_max}s"), ("evidence_freshness", fresh_ok, f"< {age_max}h"), ] 

The evaluator fails closed. Evidence is validated against a schema first. A missing skew on a two-component recovery fails, an RTO that isn't an integer fails, and evidence older than 24 hours fails even if every other check passed. Exit code 0 means PROVED, and 2 (or 3 for a skew failure) means UNPROVED, so a CI step can gate on the exit code without parsing anything.

The corrupt-artifact drill, end to end

Failure drills reuse the same pipeline. make failure-drill SCENARIO=corrupt-artifact calls scripts/break.sh to arm the scenario in a state file, then runs the drill:

cat >"$STATE_FILE" <<EOF BREAK_SCENARIO=corrupt-artifact EXPECTED_BLOCKED_AT=validate EOF 

After a normal backup, the drill sees the armed scenario and replaces the object:

if [[ "$SCENARIO" == "corrupt-artifact" ]]; then echo "-- protect: corrupting artifact at drills/${BACKUP_LOC}" CORRUPT="$(mktemp)" printf 'NOT-A-GZIP-ARTIFACT' >"$CORRUPT" "$ROOT/scripts/s3-object.sh" put "drills/${BACKUP_LOC}" "$CORRUPT" rm -f "$CORRUPT" fi 

The log at the top of this post is from Recovery Drill run 38121907545 on commit 5910732, job k8s v1.37.0. The PUT returned 200, validate failed on the hash, and Kanister recorded ActionSetFailed Action: validate. The evaluator then printed this card (box padding tidied):

┌────────────────────────────────────────────┐ │ RECOVERY DRILL #0003 │ ├────────────────────────────────────────────┤ │ Backup PASS │ │ Artifact integrity FAIL │ │ Isolated restore BLOCKED │ │ Application ready — │ │ Recovery marker — │ ├────────────────────────────────────────────┤ │ RPO 2s │ │ RTO 0s │ │ Evidence age 0m │ ├────────────────────────────────────────────┤ │ RECOVERY UNPROVED │ │ Reason artifact_integrity │ └────────────────────────────────────────────┘ 

A card that shows Backup PASS and RECOVERY UNPROVED together is the case the repo was built to catch. Restore shows BLOCKED because the drill never started it, and RTO is 0s for the same reason. The RPO row is still there because the run predates the rename. The v1.36.4 job in the same run ended the same way, with a different expected hash and the same actual hash, since both jobs wrote the same 19 bytes.

Other ways a backup goes bad

scripts/failure-drill.sh accepts eight scenarios. Five run on the Postgres-only path (corrupt-artifact, wrong-secret, schema-drift, slow-restore, noop-restore) and three on the Postgres plus S3 path. The documented run 38131463362 recorded the three application scenarios in docs/results.md:

The reasons follow from the clause order. Deleting one object from the recovery point makes the S3 half of the restore fail, so isolated_restore is the first false clause even though the Postgres ActionSet completed. The job log says isolated_restore FAIL (postgres=true s3=false). Restoring Postgres from one recovery point and S3 from another leaves rows that point at keys the app can't read, which fails data verification before the consistency check is reached.

corrupt-s3-object is the subtle one. Postgres restores cleanly and every row is there. The only failing check is that one object's sha256 no longer matches what its row says. A drill that only looks at the database would report success.

Where the 8 seconds come from

The README results table and docs/results.json cite run 38131463362, measured at 2026-10-11T10:23:50Z:

The matching line in the v1.37.0 job log is isolated_restore PASS (rto=8s prefix=restored/restore-drill-20261011102033/rp-20261011T101959Z-878b148bde), and that restore covered 3 objects. A month later, the scheduled run on main on 10 November 2026 reported rto PASS (8s / 60s) on v1.37.0 and 7 s on v1.36.4.

The scope matters more than the value. This is a single-node kind cluster on a GitHub runner, a tiny demo database and three small objects. docs/EVIDENCE.md says the numbers "do not represent large production datasets or enterprise backup fabrics", and it lists "Treating ~8 s kind RTO as a contractual enterprise target" under not proven. What the 8 seconds do show is that the RTO clause is wired to a real clock around a real restore. If a Blueprint change made restores slow, it would show up as a failed clause against the 60 second ceiling.

Metrics and alerts

scripts/export-metrics.sh writes the latest verdict in Prometheus text format, to a file for the textfile collector, and pushes it to a Pushgateway when PUSHGATEWAY_URL is set. From the same CI run:

recovery_rto_seconds{scenario="happy-path"} 8 

Next to it are recovery_contract_verdict (1 for PROVED, 0 for UNPROVED), recovery_evidence_age_hours and recovery_last_success_timestamp. deploy/prometheus-rules.yaml defines three alerts on them, shown here without labels and annotations:

- alert: RecoveryEvidenceStale expr: recovery_evidence_age_hours > 24 for: 1h - alert: RecoveryContractUnproved expr: recovery_contract_verdict == 0 for: 15m - alert: RecoveryRTOBreach expr: recovery_rto_seconds > 60 for: 5m 

The stale-evidence alert covers the quietest failure. When drills stop running, no drill fails, and the last PROVED verdict keeps looking fine long after it stopped being current.

How CI runs it

.github/workflows/recovery-drill.yml runs on pull requests, on pushes to main and nightly (cron: '17 4 * * *'). Next to shellcheck, shfmt and the evaluator's pytest suite, it runs a matrix of two kind clusters, v1.37.0 and v1.36.4. Each job runs make full-drill and then asserts more than the verdict string:

assert ev.get("components", {}).get("postgres") is True assert ev.get("components", {}).get("s3") is True assert ev["checks"].get("cross_component_consistency") is True proof = ev.get("proof") or {} assert proof.get("rp_id"), "missing rp_id" assert (proof.get("recovery_point") or {}).get("generation") assert proof.get("backup_sha256") 

Those flags have to be earned. One of the v1.0.1 fixes reads "Compute components.postgres / components.s3 from real restore+verification results (never hardcode true)", and CI now asserts s3=false on delete-s3-object.

The workflow also scans Kanister job logs for credential material and fails if it found no pods or log snapshots to check, since a leak check that looks at nothing would pass forever. Evidence from .evidence/ and .metrics/ is uploaded as recovery-evidence-k8s-<version>, and the verdict markdown goes into the job summary.

Limits

Most of these are written down in the repo itself.

• The numbers come from single-node kind with a small demo dataset. They say nothing about multi-terabyte restores, and the repo has no evidence for those.
• Cross-region DR, immutability or compliance audits and vendor SLAs are out of scope.
• RPO is not claimed. Component skew is the distance between the two halves of a recovery point, which is a different quantity.
• Solr is listed under future in the contract and isn't covered.
• The marker proves that a specific restored row came from a specific backup. It says nothing about replication lag across regions.
• At v1.0.1 the kind job in CI runs delete-s3-object, corrupt-s3-object and mismatched-recovery-points. corrupt-artifact ran in that job until the multi-service change in #19, and the run quoted above is from before that change. The scenario is still runnable with make failure-drill, and the offline evaluator fixtures in make ci still cover it.

Checks worth copying

Hash the artifact at backup time, check the hash at validate time, and treat a missing hash as a failure. That one comparison caught the corrupt-artifact case, and it costs almost nothing.

Restore into a namespace that starts empty and write a unique marker before each backup. Without both, a restore that did nothing can look like a restore that worked.

If the app spans more than one store, define one recovery point across all of them and verify the links between them: the object exists, its hash matches the row, the generation agrees. A database restored with rows pointing at missing blobs is still an outage.

Run the failure drills on the same schedule as the happy path. A green verdict means more once the same pipeline has been seen going red when it should.

Alert on evidence age as well as on failed drills.

Code, contract, Blueprints and CI evidence: github.com/justrunme/kanister-backup-restore.