I feel that by now, most of us are increasingly running agents on YOLO/auto-mode (skipping approvals), the friction of approving the actions of every agent and its subsequent subagents is becoming impractical. This increases the need for snapshots. Yes, Git can hold previous copies of your code in case something went wrong, but it is ultimately version control software that assumes a level of human safety.

In fact, Git increases the surface area by which an agent could plausibly mess up your repo:

  • git clean -fdx: deletes untracked files, which Git can't undo.
  • git reset --hard: clears uncommitted changes, and the reflog doesn't bring them back.
  • git push --force: rewrites the remote for everyone.
  • git branch -D followed by git gc --prune=now clears even the reflog's safety net.
  • Anything you gitignore: .env, a local db, ~/.config, untracked projects.

So, what is the alternative? Say that your agent runs in a sandbox with access to a hosted Git repo. Anything you do to protect that repo must either (a) ferry state out of the sandbox, or (b) restrict the agent's capacity within the sandbox itself. Both get messy quite quickly.

I gave a third option a shot: snapshot the entire VM the agent runs on, frequently yet cheaply, and record the screen alongside it, creating a "livestream" effect where you can view and return to previous states. This article will discuss the architecture of this system, the work at the hypervisor level that was done to keep it fast, and why (I think) it serves as a practical undo button for agents.

Architecture

First, I needed to pick a VM technology that would function as a good foundation. I originally started out with Firecracker microVMs, yet later ended up with the more obscure crosvm, which is behind ChromeOS's Linux support.

Few have heard of crosvm despite Firecracker starting out as a fork of it. Something that makes crosvm unique is that it has virtio-fs, which unlike Firecracker's virtio-blk, is flexible enough to mount a host's ZFS dataset. As I'll later explain, this'll be the primitive the system builds upon.

Firecracker was designed for short-lived serverless functions. crosvm was made for Chromebook users, making it one of the rare VMs designed from the ground up for user interaction. As such, it exposes a lot of graphical, audio, and passthrough devices that make it unique for sandboxing agents, where a user often needs to interact with the agent's environment. It also has Google's paranoid "process-per-device" security model, wherein each virtual device runs in its own sandbox, so compromising one doesn't compromise the whole VMM.

The filesystem: ZFS datasets under virtio-fs

The VM's home directory isn't actually a disk image. It's a ZFS dataset on the host, mounted into the guest with virtio-fs. ZFS will do a lot of heavy lifting here.

A ZFS snapshot is copy-on-write. As the name would suggest, it doesn't immediately copy any data, instead freezing the block pointers, and only blocks written afterward take new space. A snapshot of an 80GB /home only takes up the space of the files that changed, and in the case of agents, this is often only kilobytes of source code. This system allows me to frequently snapshot without much consequence.

With a disk image, the host sees a single opaque file, so you either (a) copy the whole thing or (b) get into the messy territory of guest VM tools.

The memory: checkpointing a running VM

Snapshotting files isn't enough though. If you roll back an app to a previous filesystem, yet with the same memory, it'll probably just crash, or worse yet: corrupt something.

So, a full snapshot includes a crosvm checkpoint of the RAM and device state bundled alongside the ZFS snapshot. The issue is taking a memory snapshot without causing a noticeable pause.

You could pause the VM, write out its memory, then resume. With a few gigabytes in use that will take a few seconds, frequently.

Instead the copy runs in rounds while the VM keeps going. The first round copies everything, and later rounds only copy new changes, a little like an incremental backup of memory where each round only copies what changed from the last.

Some statistics from a demo VM:

  • Total time: 4.4s
  • Pre-copy: 2.2s in 5 rounds
  • vCPU pause: 58.9ms
  • Disk hold: 67.1ms

(the disk hold is how long the filesystem is frozen while the ZFS snapshot is being taken)

So from a user's perspective, snapshots are a blip that only takes 60ms.

If the VM is writing memory faster than rounds can catch up, the snapshot waits for a calmer moment instead of forcing a long pause.

The recording: snapshots are a pain if you can't see them

So, now you have a huge list of snapshots called 2026-11-16T11:12:57Z... To state the obvious: a user remembers what the screen showed before things broke, not the exact time.

Alongside snapshots, the desktop's screen is continuously recorded as a video. Because we're talking about SSH here, most of the screen is text, not pixels, which gave me a unique opportunity to make the system even more efficient by creating a custom sort of video format that directly encodes unicode and compositor state, but that's for another article. Scrubbing back through the video doubles as the interface for undo: drag back until you see the state you want, and restore. The timeline maps each moment of the recording to the closest snapshot.

Restoring doesn't clear the current state. It starts a new branch, as if you ran git checkout -b but for the whole machine. If you went back too far, the branch you left is still there. Restoring a snapshot only takes about 7 to 9 seconds on our host.

Basically, it's like a video player:
the timeline bar with a branch hovered

The remainder: resource costs of the system

It's pretty efficient, although there remains a subtle cost (which I think is worth it):

  • Disk: snapshots capture things that probably don't need to be captured, such as a build that takes up hundreds of megabytes. Copy-on-write limits the cost to only occurring once. RAM is also tough as it's gigabytes large and changes frequently. I've implemented a deduplication system which acts in chunks across snapshots, to relieve some pressure there.
  • Retention: despite efficiency, snapshots still take up a lot of space, and so older history is phased out.

Agents need an undo button (closing)

To summarize, an agent with approvals off will eventually do something you didn't want. Git's protection is undone by shell access. Anything outside Git isn't protected in the first place. Trying to restrict the agent's permissions within the VM, or trying to ferry state out of the VM, are both messy solutions.

Snapshotting the full machine is the "nuclear option", and I've found it effective, and hope others will too.

(snapshots are not a backup though, as they are all saved to the same place, so keep a copy of anything important elsewhere)

I've made a public version, Zydra, for anyone to try (without signing up to anything), give it a try :)

ssh zydra.sh ssh -p 443 zydra.sh # From an office network