What to back up (and watch) when you self-host a crypto payment gateway

Most self-hosting advice about backups boils down to "dump the database somewhere". For a crypto payment gateway that's necessary, but it isn't the whole story, and the parts people skip are the ones that hurt.

Here's the thing that took me a while to see clearly: on a non-custodial gateway, a dead server doesn't lose you money. The money was never on it. What a dead server loses you is the ability to prove what happened: which order a deposit belonged to, which webhooks fired, whether that $499 payment from Tuesday ever confirmed. And if you lose the wrong file, you also lose every two-factor secret and webhook secret your install ever issued.

This post is the backup and monitoring setup I'd put on any Coldtill server before it takes a real payment. Paths and commands are Coldtill's, because that's what we build, but the categories apply to any self-hosted gateway.

Four things, four different failure modes

A Coldtill install has four pieces of state worth thinking about, and they fail differently:

The row people miss is .env. It holds COLDTILL_SECRET, which seals the TOTP and webhook secrets stored in the database. Restore a database dump without the matching .env and you get a database full of encrypted blobs you can't open: every user re-enrols two-factor and every webhook endpoint gets recreated. That's not a disaster, but it's a bad afternoon you can avoid with one cp.

And the cold wallet seed isn't really a server backup at all. Coldtill never sees those keys; it only needs the address. It's on the list because it's the only one of the four that actually holds money.

Where things live

If you installed with the one-line script from the Coldtill GitHub repo, the deploy directory is /opt/coldtill. The installer makes .env readable only by root and the Coldtill user, so run Compose with sudo:

cd /opt/coldtill sudo docker compose ps 

You should see PostgreSQL, Caddy and the four Coldtill processes (api, watcher, sweeper, payouts).

Nightly database dumps

pg_dump through the running container is all you need:

#!/usr/bin/env bash # /usr/local/bin/coldtill-backup set -euo pipefail cd /opt/coldtill mkdir -p /var/backups/coldtill docker compose exec -T postgres pg_dump -U coldtill coldtill \ | gzip > /var/backups/coldtill/coldtill-$(date +%F).sql.gz 

Run it from root's crontab (sudo crontab -e) at a quiet hour:

15 3 * * * /usr/local/bin/coldtill-backup 

One thing worth knowing: running the install script again is also how you upgrade, and it saves a database dump to /opt/coldtill/backups/ before it restarts on the new image. Nice, but don't treat it as your backup strategy. It only runs when you upgrade, and it sits on the same disk.

Get the dumps off the box

A backup on the same disk as the database doesn't survive the thing you're most likely to lose: the whole server. Ship them somewhere else. The Coldtill docs use restic to an S3-compatible bucket, which I like because it's encrypted, deduplicated and has sane retention:

sudo apt install restic export RESTIC_REPOSITORY=s3:https://s3.example.com/coldtill-backups export RESTIC_PASSWORD_FILE=/root/.restic-pass restic init 

Then add two lines to the end of the backup script:

restic backup /var/backups/coldtill restic forget --keep-daily 14 --keep-monthly 6 --prune 

Your provider's snapshots are a good extra layer on top of this, not a replacement. Snapshots are easy to forget about, and they live with the same company as the server.

.env and secrets/ go somewhere offline

These two don't change often, so they don't belong in a nightly job. Copy them once to encrypted offline storage, and again whenever they change (new hot wallet identity, rotated secret):

sudo tar czf - -C /opt/coldtill .env secrets \ | gpg --symmetric --cipher-algo AES256 > coldtill-keys-$(date +%F).tar.gz.gpg 

Put that file wherever you keep things like recovery codes. Not in the same bucket as the dumps: the whole point is that someone who gets the dumps shouldn't also get the key that unseals them.

A note on secrets/: the hot-wallet identity in there is encrypted under your passphrase, but by default the passphrase sits next to it in secrets/hot_passphrase, so the sweeper and payouts processes can restart without you. If that combination in a backup makes you uneasy, you can run without a stored passphrase (coldtill init --store-passphrase=false) and type it at start-up. The trade-off is that if sweeper or payouts restarts, sweeps and payouts wait for you.

Actually restore it

A backup you've never restored is a hope, not a backup. Rent a spare VPS for an hour, install Docker, put back .env and secrets/ from the same point in time as the dump, then:

cd /opt/coldtill sudo docker compose up -d postgres gunzip -c coldtill-2026-11-03.sql.gz | sudo docker compose exec -T postgres psql -U coldtill coldtill sudo docker compose up -d 

Sign in, check two-factor still works, open the Payments page and compare a few rows against the live server. If the two-factor prompt fails, your .env and dump don't match, and you've just learned that cheaply.

After a test restore, the Payments page is the quickest sanity check: same invoices, same states, same filled amounts as production.

Monitoring: what to actually alert on

Uptime is the easy part. Coldtill exposes a health endpoint:

curl -fsS https://pay.example.com/healthz # prints: ok 

Point UptimeRobot, Better Stack, Healthchecks.io or whatever you already use at it. The api container has a Docker health check too.

The more useful signals are in the logs. They're JSON, one line per event, so plain grep goes a long way:

cd /opt/coldtill sudo docker compose logs -f --since 1h api watcher sweeper payouts sudo docker compose logs sweeper | grep '"level":"WARN"' 

Here's what I'd page on, and why:

• "level":"ERROR" from any process. The obvious one.
• hot wallet … needs at least and sweep not possible. The hot wallet pays gas for sweeps. When it runs dry, sweeps stop. Nothing is lost (deposits sit safely on addresses that can only send to your cold wallet), but money piles up where it shouldn't, and payouts stop too.
• filled a nonce gap. Coldtill fixes this itself with a zero-value self-transfer, but if you see it often, something is failing upstream.
• conversion delivery mismatch. Only relevant if you've turned on auto-convert. A swap provider reported something Coldtill's own nodes couldn't match, so a human should look.
• Payouts in needs-review. Coldtill also sends a payout.needs-review webhook, so if you already have a webhook receiver, route that event to wherever your team actually reads things.

A rough watcher, if you don't have a log pipeline yet:

#!/usr/bin/env bash # run every 5 minutes from cron cd /opt/coldtill hits=$(docker compose logs --since 5m api watcher sweeper payouts 2>&1 \ | grep -E '"level":"ERROR"|needs at least|sweep not possible|nonce gap|delivery mismatch' || true) if [ -n "$hits" ]; then printf '%s\n' "$hits" | mail -s "coldtill alert on $(hostname)" ops@example.com fi 

It's crude, it'll repeat itself, and it's still much better than finding out from a customer.

The quiet failure: a chain watcher falling behind

The alert that's easy to forget is the one where nothing errors. If an RPC node is rate limiting you, or quietly serving stale data, the watcher just... sees blocks late. No crash, no error line, payments sitting in "pending" longer than they should.

The dashboard shows a "last block processed" per network under Nodes & networks. If one of those stops moving while the chain keeps going, your watcher is behind. The fix is usually the same: public RPC endpoints are fine for testing and rate limited for everything else, so for mainnet use your own nodes or a paid provider, and add at least two per network so failover has somewhere to go.

The overview shows confirmed volume and recent payments at a glance. A long run of "pending" here is often the first visible sign of a node problem.

A short checklist

Before mainnet, I want to be able to say yes to all of these:

• Nightly dumps run, and land off the server.
• .env and secrets/ are in encrypted offline storage, separate from the dumps.
• I've restored onto a spare server at least once, and two-factor worked afterwards.
• An uptime monitor is watching /healthz.
• Something tells me about ERROR lines, hot wallet warnings and needs-review payouts.
• Someone looks at hot wallet balances, and "last block processed", more than once a month.

None of this is exotic. It's the same discipline you'd apply to any Postgres-backed service. The difference is that with a payment server, the records are the product. The coins will be fine on chain either way; the question is whether you can still tell which customer paid for what.