Snapshotting a DigitalOcean Droplet while it's running

DigitalOcean lets you snapshot a Droplet while it's running, and its docs are careful about it. They say a live snapshot "may result in performance degradation and compromise data consistency", and that many databases only guarantee a consistent disk if you stop them or power the Droplet off first. That's sensible advice. I wanted to know what it means in practice: what a snapshot of a busy Droplet contains when you restore it, whether the Droplet slows down while it's taken, and what powering off first actually costs you.

The setup

One Droplet in Frankfurt, the 2 GB / 1 vCPU size with a 50 GB disk, running Ubuntu 24.04 and Postgres 16 with its default settings. On it, two writers ran the whole time:

• Postgres. One numbered row per transaction, about 100 a second. Every insert logged when it was sent and how long the commit took.
• Files. The same fixed-width numbered line, about 100 a second, appended to two files. One file got an fsync after every line. The other was only written, so its lines sat in the page cache until Linux decided to flush them.

Then I took snapshots through the API, timed them by polling the action, restored each one to a new Droplet and looked at what was on the disk. Each restored copy was destroyed as soon as I had checked it, and each snapshot deleted.

I did six live snapshots: three with about 2.3 GB on the disk, one at 13 GB, one at 29 GB, and one after deleting the extra data again. I filled the disk with random data so nothing could compress it. Then there was one snapshot with the Droplet powered off, for comparison.

The Droplet didn't notice

I half expected writes to slow down while the snapshot was being taken. They didn't. During the 13 to 184 seconds each snapshot took, the 99th percentile of Postgres commit times stayed between 2.8 and 5.7 ms, no higher than the 5.5 to 6.5 ms of the rest of the run. The longest gap between two consecutive writes during any snapshot was 38 ms, and outside the snapshots it went up to 42 ms.

So on this Droplet, at this write rate, the "performance degradation" the docs mention didn't show up at all.

The disk is frozen three seconds after you ask

Every live snapshot captured the disk at a single moment, between 3.0 and 3.25 seconds after I sent the request. I can see that moment precisely because both writers stamp their lines with the time: in the restored copy, the last row in Postgres and the last line in the fsynced file were always within 7 milliseconds of each other. Everything written after that moment was missing, and nothing written before it was.

That makes a live snapshot the same as pulling the power cable at that instant. Which is a case Postgres is built for.

Postgres was fine every time

Every restored copy came up with Postgres saying "database system was not properly shut down; automatic recovery in progress". It replayed its write-ahead log for 0.13 to 0.64 seconds and started. The table was complete in all six: no gaps in the numbering, every row up to the moment of the freeze, nothing half-written. The fsynced file was the same. Not one torn or damaged line in six snapshots.

So the docs' warning about databases is worth taking seriously, but for Postgres with its default fsync and full_page_writes on, a live snapshot is a crash it knows how to recover from. A database or an app that doesn't flush its writes is a different story, which the other file showed.

The file without fsync was up to 26 seconds behind

The file that was never fsynced was behind in every live snapshot, by between 0.7 and 26.4 seconds. And it didn't just stop early. Each one ended with a run of zero bytes, between 704 and 3,392 of them, where lines should have been. The file's size had already been saved, always on a 4 KB boundary, but the last stretch of its contents hadn't reached the disk.

The 26 seconds made sense once I looked at how the disk is mounted. DigitalOcean's Ubuntu image mounts the root filesystem with commit=30, so ext4 only commits its journal every 30 seconds, instead of the usual 5. Data that's only in the page cache can be up to 30 seconds old at the moment the snapshot freezes it.

If your app writes logs, uploads or a queue to plain files without fsyncing them, a live snapshot can lose the last half minute of them, and leave zeros behind instead of a clean cut.

Powering off first

With the Droplet shut down through the API, the snapshot had nothing to catch in flight. The restored copy had every row and every line in both files, and Postgres started without any recovery.

The cost is downtime. From the shutdown request until I could SSH back in was 66 seconds: 8 to shut down, 32 for the snapshot, and 26 for the power-on action and the boot. A 2.3 GB snapshot taken live took 13 to 35 seconds, and the Droplet kept serving the whole time.

Bigger disks take longer, both ways

The snapshot size followed the data on the disk, not the 50 GB size of the disk. The time to take it grew, but not in step with the size: 13 times the data took about 5 to 14 times as long. Restoring grew too. A Droplet created from the 29 GB snapshot took 114 seconds to become active, three times as long as one from the small snapshot, plus another 10 to 20 seconds before SSH answered.

Since DigitalOcean charges $0.06 per GB a month for snapshots, based on the size of the snapshot, that 29 GB one would have cost about $1.75 a month to keep.

Deleting data shrinks the next snapshot

After the 29 GB snapshot I deleted the random files and took another one. It was 2.33 GB. I'd expected it to stay big until I ran fstrim, since deleting a file only marks its blocks free in the filesystem. But the same mount options include discard, so ext4 tells the disk about freed blocks straight away. Running fstrim afterwards reported 21.1 GiB trimmed, and the snapshot after that was still 2.33 GB.

What I got wrong

I thought the snapshot would have to include the deleted data, and planned the fstrim step to show how much it would save. It saved nothing, because the image already discards on delete. I should have looked at the mount options before designing that part.

I also tried to measure the clock difference between my laptop and the Droplet over SSH, so I could line the freeze moment up with the API's timeline. Each round trip took half a second, mostly connection setup, so the result was useless. I measured everything against the request time recorded on my side and the timestamps written on the Droplet instead, and both clocks were synced over NTP.

And my first comparison of write latency "during" and "outside" the snapshot showed a 46-second gap outside it. That was my script measuring straight across the window it was supposed to leave out. The real longest gap was 42 ms.

Caveats

This is one Droplet size, one region, one write rate, and Postgres with default settings. Each size was snapshotted once, and the timings at the small size varied from 13 to 35 seconds, so the bigger ones could vary as much. I didn't test Volumes, which aren't part of a Droplet snapshot, or databases other than Postgres.

The whole run, eight snapshots and seven restored Droplets over about 35 minutes, cost less than five cents.

What I'd take from it

• A live snapshot is a power cut at one moment, about three seconds after you ask. Anything that survives a power cut survives it.
• Postgres with default settings recovered in under a second every time, with nothing missing.
• Anything you write without fsync can be up to 30 seconds behind on these images, and the file may end in zeros. Stop those writers, or power off, if those files matter.
• Powering off gives you a clean copy for about a minute of downtime on a small Droplet.
• The snapshot is the size of your data, and deleted data really is gone from it on DigitalOcean's Ubuntu image.