A Restart Is Not a Reset: Five Persistence Myths and a Boot-ID Probe

My worker wrote a state file at 23:40. By 06:00 the file was gone. No crash log. No alert. Just an empty directory and a new boot ID.

I had tested that job. It passed. But I tested it without ever restarting the machine. That is not a test. That is a demo.

What I actually wanted to know

Three questions. Does my data survive a restart? Does my process come back on its own? Does my health check tell the truth?

I run small jobs on a free server option, and I call free models from the same place. Both are operator-supplied conveniences, not guarantees. So I stopped assuming and started measuring.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The probe: one file, two runs, one restart

Save this as probe.py. Run it. Restart the box. Run it again. Then diff the two JSON files.

# probe.py - run before and after a restart, then diff the outputs import json, os, pathlib, sqlite3, time def boot_id(): p = pathlib.Path('/proc/sys/kernel/random/boot_id') return p.read_text().strip() if p.exists() else 'unknown' def write_probe(path): try: path.parent.mkdir(parents=True, exist_ok=True) path.write_text(json.dumps({'at': time.time()})) return {'writable': True, 'bytes': path.stat().st_size} except OSError as e: return {'writable': False, 'error': str(e)} def sqlite_probe(path): try: con = sqlite3.connect(path) con.execute('CREATE TABLE IF NOT EXISTS t (v TEXT)') con.execute('INSERT INTO t VALUES (?)', (str(time.time()),)) con.commit() rows = con.execute('SELECT count(*) FROM t').fetchone()[0] con.close() return {'rows': rows} except sqlite3.Error as e: return {'error': str(e)} report = { 'boot_id': boot_id(), 'hostname': os.uname().nodename, 'cwd': os.getcwd(), 'deploy_id': os.environ.get('DEPLOY_ID', 'unset'), 'tmp_file': write_probe(pathlib.Path('/tmp/probe.json')), 'home_file': write_probe(pathlib.Path.home() / 'probe.json'), 'sqlite_tmp': sqlite_probe(pathlib.Path('/tmp/probe.db')), 'sqlite_home': sqlite_probe(pathlib.Path.home() / 'probe.db'), } print(json.dumps(report, indent=2))

Now the shell side:

python3 probe.py > run-1.json # restart the server now, without redeploying python3 probe.py > run-2.json diff <(jq -S . run-1.json) <(jq -S . run-2.json)

A changed boot_id means the machine rebooted. Rows going from 3 to 1 means the disk did not survive. That single diff answers more than an hour of guessing.

FAQ: five claims I used to repeat

1. 'If it survived a restart, it is persistent.'

Surviving a restart proves the image was reused. It does not prove the storage is durable.

Many free setups keep the root image and drop instance storage. My /tmp write kept succeeding every time. It also kept vanishing. Write success is not durability.

2. 'It is the same box, so it is the same disk.'

Hostname is not identity. boot_id is. Check both, and log them early.

If your deploy script keys a lock or a cache on hostname, two reboots can hand you the same name and an empty disk. That is a bad combination.

3. 'The model remembers our conversation.'

No. Each model call is stateless. The transcript is your state, not the model's.

If your history file lives in /tmp, your memory dies with the process. Here is the shape I use. Pseudocode, simplified:

history = store.read_json('history.json', default=[]) history.append({'role': 'user', 'content': prompt}) reply = call_model(history) # stateless call, no server-side memory history.append({'role': 'assistant', 'content': reply}) store.write_atomic('history.json', history)

Two rules. Persist before you render. Write atomically, or a crash leaves half a turn.

4. 'It is only a cache, so losing it is fine.'

A cache is only safe to lose if it rebuilds without a human. Can yours?

Ask three questions. What rebuilds it? How long does that take? What breaks during the rebuild? If any answer is 'me, by hand', it is not a cache.

5. 'A 200 from /healthz means my state is fine.'

Liveness says the process answers. Readiness says it can serve. Those are different checks.

A process with an empty database still returns 200. Add one readiness probe that touches real state.

curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8000/readyz

A durability decision table

Data Where it goes Rebuild cost Acceptable loss
Session transcript external store, atomic write high none
Model output cache /tmp with a TTL low, one call all of it
SQLite job queue home dir plus export medium last minute
Secrets env at deploy time none n/a
Logs stdout, shipped off-box n/a small window

Fill the table in before you pick a directory. The table is the design. The paths are just consequences.

The restart drill I run before shipping

  1. Deploy. Let the job write something real.
  2. Restart the server. Do not redeploy. Redeploying hides the bug.
  3. Run the probe. Diff boot_id, row counts, file sizes.
  4. Kill the process mid-write. Check the store is not corrupt.
  5. Point /readyz at real state. Confirm it fails when state is missing.
  6. Write down what you lost. An empty list means you learned nothing.

Steps 2 and 4 catch most of the surprises. They cost about ten minutes.

Limitations, and who should skip this

This probe tests your box, not your provider's contract. It cannot prove durability across regions or disasters.

It assumes a Linux host with /proc. It covers a single node only.

Skip it if you need strong durability guarantees, multi-node consistency, or audited storage. Use managed storage and document that choice. Skip it too if restarts are forbidden in your environment; use a canary node instead.

The probe is cheap. The wrong assumption is not.

If you want somewhere to run the two-run diff before paying for anything, the free server option is where I ran mine. Bring the table with you.