My nightly backup job had been green for months. I never restored it once.
Last week I looked at a cron job of mine that had printed "backup ok" every night for months, and I asked myself a plain question: when did I last load one of those files into a database? I could not remember a date. A green job tells you a file was written. It does not tell you the file works.
I built a small test and ran it end to end. I created a Postgres database, backed it up on a schedule, changed the live data afterwards, restored the backup into a separate database, and compared row counts. I used Levelrail, the self-hosted deploy platform I work on, built from the v1.0.1 release branch (commit d510e8488). Everything below ran on my Mac with Docker, and I mark the parts I only read in the docs.
TL;DR
| Step | Command | What it proves |
|---|---|---|
| Schedule | backups schedule set | A backup runs without you |
| Restore as new | backups restore-as-new | The backup loads into a fresh database |
| Compare | databases query | Counts match the moment of the backup |
| Drill | backups drill run | Checksum, restore and content check pass |
| Copy | backups destinations copy set | A second bucket holds the same object (docs only) |
The test setup
I made a Postgres 16 database with two tables: 500 customers and 2000 orders. Then I recorded three numbers I could check later.
levelrail-cli databases create --name main --engine postgres --version 16 levelrail-cli databases query main --sql "select (select count(*) from customers) as customers, (select count(*) from orders) as orders, (select sum(total_cents) from orders) as total_cents" customers orders total_cents 500 2000 8805000 I pick a sum on top of the counts because two tables with the right row count can still hold the wrong values.
Schedule it and take one now
A backup target is the place the files go. Levelrail has a local type that needs no bucket, which is fine for a first run but lives on the same disk as the database, so it is no protection against losing the machine. I used it first, then switched to an S3-compatible target, which I get to below.
levelrail-cli backup-targets create --name nightly --provider local --path nightly levelrail-cli backups schedule set main --target TARGET_ID --cron "0 3 * * *" --retain 7 --retain-days 30 levelrail-cli backups trigger main --target TARGET_ID levelrail-cli backups list main The list printed a succeeded row of about 11.8 KB, finished in the same second it started. Retention combines two limits: keep at most the newest 7 backups, and nothing older than 30 days. A backup goes when either rule would remove it (docs/guides/self-host-postgres-with-backups.md).
Break the live data, then restore the old state
This is the step that makes the test mean something. After the backup, I deleted 500 orders from the live database, the way a bad migration or a wrong DELETE would.
levelrail-cli databases query main --write --confirm main --sql "delete from orders where id > 1500" levelrail-cli backups restore-as-new main --backup BACKUP_ID --new-name main-restore Restoring as new builds a separate database next to the original and never touches it. The new container was up in under a minute. Then I ran the same count query against both.
live customers orders total_cents 500 1500 6597750 restored customers orders total_cents 500 2000 8805000 The restored database has the numbers from before the delete, to the cent, and the live one has the damage. That is the output I wanted from a backup, and I had never produced it from my cron job.
An in-place restore also exists. It overwrites the live database and asks you to type the database name, so I would use restore-as-new first and only then decide.
What a restore drill adds
A manual rehearsal is good once. A drill is the same idea on a timer. For Postgres, MySQL and MariaDB, Levelrail downloads the stored backup, compares its SHA-256 with the checksum recorded at backup time, replays it into a throwaway container with no network, checks the result, and deletes the container whether it passed or failed (docs/prove-your-backups-work.md).
My first drill attempt failed, and the reason is worth knowing:
start drill of backup "bkh_...": server returned 409: restore drills are not applicable to local targets Drills need an object-store target. So I started a throwaway S3-compatible server (SeaweedFS, in a container) and connected it as a custom target. The docs say a storage destination on the same host is refused unless you set APP_NOTIFY_ALLOW_PRIVATE_NETWORKS=true on the control plane. Despite the name, that one switch also gates storage destinations, so I set it only on my laptop test.
levelrail-cli backup-targets create --name s3test --provider custom --region us-east-1 \ --endpoint S3_ENDPOINT --bucket backups \ --access-key-id KEY --secret-access-key SECRET levelrail-cli backups trigger main --target S3_TARGET_ID levelrail-cli backups drill run --backup BACKUP_ID drill bkd...: passed (stage done, 5513 ms) object present and complete: true checksum matches: true restored into scratch: true restored content matches: true After that, levelrail-cli backups health moved the database from unverified (no restore has been proven to work yet) to healthy with a "last verified restore" timestamp. By default every resource with a succeeded backup is drilled once a week, and a failed drill fires a restore_drill_failed alert rule, which you create under Alerts. I did not wait a week for the automatic run, so that part is from the docs.
Off-box copies
One bucket is still one failure domain. Levelrail can copy each finished backup to a second destination, check the copy against the primary's checksum, and alert when the copy falls behind (docs/backup-copies.md). You turn it on with one command:
levelrail-cli backups destinations copy set SECOND_TARGET_ID I only ran copy show, which printed backup copies: off. I did not test the copy path, so treat that section as documentation, not as my result. Two details from the docs matter. Only backups that finish after you enable it are copied, and restores always read from the primary destination.
Three things I would not skip
Back up the master key. Secrets stored by the platform are encrypted with it, and the control plane's own backups do not contain it (docs/control-plane-backup.md). Keep a copy somewhere that is not the server.
Back up the control plane too, not just your databases. Its snapshots live on the same disk by default, so read the disaster recovery page for the off-box path.
Bind mounts are not backed up. Only Docker-managed named volumes are (docs/guides/self-host-postgres-with-backups.md).
How this compares
| Approach | Proves the file exists | Proves it loads | Runs on a schedule by itself |
|---|---|---|---|
pg_dump in cron | Only if you add a check | Only if you write the restore step | Cron runs the dump, not a restore |
| Manual restore rehearsal | Yes | Yes | No, someone has to remember |
| Levelrail drill | Yes (size and checksum) | Yes (Postgres, MySQL, MariaDB) | Yes, weekly by default |
The drill is not magic. For engines other than those three it checks the object, not a replay. Object lock and versioning detection depends on the storage provider answering those calls. And my row-count check is a stronger test of my own data than any generic drill, so I would still run it by hand after big schema changes.
FAQ
How often should I test a restore?
Weekly is the default drill interval in Levelrail and it is a sensible floor. Do a manual restore-as-new after any major version upgrade or schema change.
Does restore-as-new touch my live database?
No. It creates a separate database from the backup. I confirmed this above: the live database kept its 1500 orders while the copy showed 2000.
Can I use Cloudflare R2 or AWS S3?
The CLI accepts aws, r2 and custom providers (docs/backups-and-storage.md). I ran my test against a local S3-compatible server, not against R2 or S3.
Where this leaves me
My cron job now has a replacement that I have actually watched restore a database. The number I keep thinking about is 6597750 against 8805000: the cost of finding out too late.
If this saved you time, a star on the repo helps others find it.
glincker / levelrail
Self-hosted alternative to Vercel and Coolify. Git push, HTTPS, logs, rollback. No Kubernetes.
Levelrail
Your own Vercel, on your own servers.
Push to deploy with HTTPS, logs, metrics and one-step rollback. One Go binary, no Kubernetes. Apache 2.0.
Website · Docs · Install · Compare · FAQ · Roadmap · Discussions
English · 简体中文 · Español · Português (BR) · Deutsch · Français · 日本語
Turn any Linux server into a private cloud: Git deploys, HTTPS, databases with verified backups, metrics, logs and alerts, without Kubernetes and without a monitoring stack to bolt on. Pre-1.0: single-server is production ready, multi-node is beta (Feature status).
200 MBidle RAM, 1.5% of one CPU | 1 binaryGo control plane, Docker runtime | 360one-click service templates | 240+MCP tools for coding agents |
Install
curl -fsSL https://levelrail.com/install.sh | sudo sh Needs Linux (amd64 or arm64) with systemd and root; the installer adds Docker. Open ports…
Sources checked
Levelrail docs read from the v1.0.1 release branch: docs/prove-your-backups-work.md, docs/guides/self-host-postgres-with-backups.md, docs/backup-copies.md, docs/backups-and-storage.md, docs/control-plane-backup.md.
