Scale is complexity.

Exhaustively verifying the correctness of a file system is, in theory, impossible. So what singlefs does is creep closer bit by bit, with pruning and a reuse database: anything already judged never needs judging again, and equivalent cases are judged only once. One layer up sits a multi-agent scheduling and scripting system: a main agent dispatches the work, following the gate scripts to run verifications and experiments.

Is the system stable? Reasonably so — like bolting a steering wheel onto a bare chassis. When things go well it runs fast, but life always has its rainy days. Running it for real, we hit problems like these:

  • The main agent dispatches tasks within milliseconds. Several tasks start at once, fight each other, and eat all the memory: OOM.
  • Tasks that look perfectly reasonable hold onto resources for ages, and the agent's own scheduled check-ins are unreliable, so nobody comes back to look.
  • Every script writes straight into the database, with a real risk of corrupting it; task records are scattered everywhere and slow to look up. And so on.

So we need two things: a task management system and a database service. The task management system coordinates resources and dispatches tasks; the database service validates and writes the data. In the repository the former is called task-server and the latter prune-server.

0. The starting point: every task goes through one entrance

There is exactly one rule: every task that runs project code is launched through the task service. Builds, tests, gates, experiments — light or heavy, CPU or GPU — all go into the same queue.

# Submit a task: the command goes verbatim after -- task-server/task submit -- cargo test --release -p singlefs-verification --test crash_enumeration_fixed_script_stream # Get the outcome: exits on completion, with the task's exit code task-server/task wait <task-id> 

The parameters that used to be passed at submission — thread count, memory cap, which machine, which card — are all gone. The service decides these. If a command specifies one, it is rejected on the spot, and the agent is told which parameter to remove. The reason is simple: one of the things agents are best at is asking for a little more resource for their own tasks.

The service itself is a user-level systemd service: FastAPI on uvicorn, listening only on a unix socket. Tasks and every attempt are recorded in an embedded SQLite database, and each row carries an HMAC check column. Each attempt is its own systemd unit, wrapped in a cgroup memory cap: if it goes over, the kernel kills only that unit and nobody else.

The "just launch it directly" path is blocked at the tool layer. Every Bash command an agent runs passes through a hook first; any cargo, gate, or experiment script not launched via task submit is rejected on the spot. Sub-agents have had their right to run verifications revoked: they write the commands they want run into their report and hand it to the main agent to submit.

Results come back as a code table, each outcome mapped to a fixed exit code, so the agent knows what to do next just from the exit code:

# task-server/task_server_lib.py (excerpt) # Outcome types: name → (exit code of wait, None means the command's own; description) OUTCOME_TYPES = { "succeeded": (0, "command exited 0"), "command-failed": (None, "command itself exited non-zero"), "memory-cap-hit": (250, "hit this task's own cap"), "memory-cap-at-maximum": (236, "hit the cap, already at the max of both hosts"), "card-fault": (237, "scrub found the card faulty"), "never-fits": (241, "won't fit on any host no matter how long it waits"), "stalled": (233, "log didn't grow for 10 minutes; the service stopped it"), "over-four-hours": (229, "cumulative or projected time reached the 4-hour hard limit; bounced by the service"), "killed-externally": (228, "killed by a TERM or KILL not sent by the service"), # ... } 

Another key point: the agent's scheduled check-ins are unreliable. So the service can't rely on the agent remembering to come back. When a task reaches an outcome, has been queued too long, or looks stuck, the service sends a message straight to the inbox socket of the session that submitted the task, waking the agent up. The first line says whether this is an outcome or a reminder; what follows is the evidence and the way forward. The agent never has to poll or set an alarm.

At first everything went very smoothly. The task management system launched tasks centrally within the configured resource limits, queued what didn't fit, and bounced failures and errors back — in short, it handed results back to the agent, which carried on according to what came back. On top of that, a periodic sweep checked task states and terminated anything abnormal.

Then the problems arrived right on cue.

1. Can tasks cut in line?

Say memory is 50GB. Task A is using 32GB, and the next task, B, needs 24GB, so clearly B can't start yet. But C, behind B, needs only 16GB. Can C cut in? What about D after it? If we keep letting later tasks cut in, when will B ever get done?

Design: cutting in is allowed, but it's a matter of time. C's estimated runtime must not exceed A's remaining runtime. On top of that, there's a maximum queue time: once a task has waited two hours, it gets top priority and nobody may cut in front of it.

In the implementation this is called "gap-filling." How much longer the task in the way still has to run is called the "blocking time":

# task-server/task_server.py (excerpt) def attempt_remaining(self, attempt): """How much longer (seconds) this running attempt has: estimate minus elapsed; with no estimate, or once past the estimate, take elapsed.""" estimate = self.attempt_estimate(attempt) elapsed = max(0.0, self.now - float(attempt.get("started_at") or self.now)) if estimate is not None and estimate > elapsed: return estimate - elapsed return elapsed 

The second half is a small trick: if a task has no runtime history, or has already run past its estimate, we guess it will run "at least as long again as it already has."

Whether a task can cut in is decided in two steps:

# gap_fill in task-server/task_server.py (excerpt, self-test branches removed) # Step 1: after B at the head starts, does the share still free on this host fit C? left = snapshot.total(host) - margin - head_booked - beside if booked <= left: return True, "gap-fill: fits, estimate not consulted" # Step 2: only tasks that would eat into B's share are compared by time estimate, _ = self.task_estimate(task) if estimate is None: return False, "no gap-fill: this task has no estimate" blocking, blocker = self.blocking_seconds(host) # blocking time if estimate <= blocking: return True, "gap-fill: estimate within blocking time" return False, "no gap-fill: estimate exceeds blocking time" 

If C never touches the memory B needs, however long it runs it won't delay B, so it starts right away. Only tasks that would eat into B's share are compared by time: C must finish before A does.

The two-hour rule takes just two lines:

def aged(self, task, now): """Still queued aged_seconds after submission: high priority.""" return now - float(task.get("submitted_at") or now) >= self.aged_seconds def queue_order(self, queued, now): """Queue order: high priority first, each keeping its original order (stable sort).""" return sorted(queued, key=lambda row: not self.aged(row[0], now)) 

Big and small tasks alternate per host. A task whose booked amount reaches 85% of a host's usable limit counts as a "whole"; everything else is a "fragment." Fragments fill a batch, the batch is sealed, and the whole gets its turn; once the whole has started, the next batch of fragments is filled. Otherwise fragments keep arriving one after another and the whole never gets to see the host empty.

2. Where does the runtime come from, and is it accurate?

Cutting in requires comparing times. Where do the times come from?

From the database. When each attempt wraps up, the service records its runtime. The estimate is the runtime of the most recent successful run of the same command; if, among the last five runs, one ran longer before failing, that longer one is taken. Failures that exited right at the start don't count: they say nothing about how long the command takes.

Tasks with no record may not cut in. That's the estimate is None → False in the code above.

Is it accurate? No. The same command runs fast when it reuses a lot from the verdict database and slow when it reuses little. So for entry points that run item by item, the service estimates more finely: before the real run, it launches an estimation sub-task in which the entry point reports, for each item, "how many to judge fresh and how many can be reused":

ITEM_PLAN item=<item name> to_judge=<count to judge fresh> reusable=<count reusable> 

After each run, the entry point also reports how long each item actually took:

ITEM_DONE item=<item name> seconds=<seconds> judged=<count judged fresh> reused=<count reused> exit=<code> 

From this the service records, per item, "seconds per fresh judgment, seconds per reuse," and the next estimate is the sum of counts times rates. It's still just an estimate, so it's only used to decide whether to cut in. Sadly, this isn't very accurate either. Ha.

3. How much memory does a task need?

For a task that has never run, first look for its kin; if there's no kin either, give it 4GB. If it hits the cap midway, raise the cap by 1.5× and requeue it.

"Kin" is identified by a command key. For cargo test, for example, the same package, same features, and same build profile make one key; which test target is named does not go into the key — they build the same thing, and most of the memory goes into compilation.

With a record, take the largest of the peaks measured in the last five runs:

# task-server/task_server.py (excerpt) RECORD_WINDOW = 5 def recorded_prediction(self, task, base): """Peaks measured over the last RECORD_WINDOW runs of the same command key: prediction, cap and booking all equal the largest one, not multiplied by 1.5.""" window = sorted(self.recorded_measurements(task), reverse=True)[:RECORD_WINDOW] ... predicted = max(value for _, value in window) cap = math.ceil(predicted / lib.MEBIBYTE) * lib.MEBIBYTE 

Without a record, start at 4GB and grow on each hit:

# task-server/task_server_lib.py CAP_GROWTH_FACTOR = 1.5 UNRECORDED_MEMORY_DEFAULT_BYTES = 4 * GIBIBYTE def grown_cap(cap_bytes): return math.ceil(cap_bytes * CAP_GROWTH_FACTOR / MEBIBYTE) * MEBIBYTE 

4096M → 6144M → 9216M, bumping up one step at a time. If multiplying by 1.5 would reach the whole host's limit, the task is no longer requeued; it's bounced back to the session that submitted it, with suggested fixes: verify the scope, shrink it, or shard it.

For tasks that hit the cap several times in a row, the service adds a paragraph to the message — "check the code for logic loops or memory leaks" — and lists every cap and peak in order: is the peak growing each time, is it pinned at the cap every time? Not fitting in 4GB may mean the task really is big; still not fitting after climbing to 40GB usually means something is wrong with the code.

There's a pitfall we stepped in here: what exactly is the peak measuring? cgroup's memory.peak counts page cache too. One build task used under 1GB in its own processes, but page cache pushed the reading to 50GB. Setting quotas from that number would mean handing the entire machine to that one task. So the peak the service records counts only memory the processes themselves use (anon plus shmem), not page cache.

At this point, the main agent's concurrency problem and the problem of misbehaving tasks hogging resources were solved. And with a task database in place, the lookup and record-keeping problems were solved too. Most importantly, verdicts accumulated from small everyday tests can be reused by later tests, and bit by bit they fill in coverage for the whole project.

But it wasn't over. New problems came hard on the heels of the old.

4. Gate results that couldn't be reused

In the original design, a light gate ran day to day and a heavy gate ran at commit time. Then I discovered the light gate was actually a completely separate codebase. That meant every run started from scratch, reused nothing, and its results were useless to the heavy gate.

Hooking the light gate into the pruning database would amount to rewriting a scaled-down heavy gate. Merging it into the heavy gate instead would mean the heavy gate had to allow runs scoped by range and by named item.

Solution: take the heavy gate's whole verification system apart and make it addressable by name. Delete the light gate.

Now there is just one gate, with two ways to run it:

Mode How to launch When
Unscoped No selection flags; the gate picks what to judge by key on its own At commit
Scoped --item <item>, --check <cell>, named test targets Any time
# List the items that can be named; no build, no run bash .claude/gate.d/crash-point-replay.sh --list-items # Judge only the one crash-enumeration case changed this time bash .claude/gate.d/crash-point-replay.sh --check crash-case-replay --crash-cases-only --item crash-case:<case name> 

The crucial point: both modes are the same script and the same verdict database; only the scope differs. Verdicts produced by everyday named runs are written to the database, and the commit-time run reads them by key and reuses them directly. Each verdict's key consists only of what it depends on: the fingerprint of the function unit with comments stripped, the toolchain, dependency versions, and its own inputs. Paths, line numbers, and which commit — none of these go into the key.

There is only one verdict database, too: the RocksDB held by prune-server. Tasks on both machines read and write through it, and each kind of verdict gets its own key prefix. Any script that opens a separate directory or writes a separate table to store verdicts is bounced in review, no exceptions. This is the database service mentioned at the start.

To keep "everyday" from quietly turning back into "everything," the service added a confirmation step at the entrance. After submission a task doesn't enter the queue right away; the service prints a confirmation slip: how many items it recognized from the command, estimated memory, estimated time. The agent has to copy two of those numbers back verbatim before the task enters the queue:

task-server/task confirm <task-id> --items <N> --tasks <M> 

Unscoped runs and whole-package tests are rejected on the spot anywhere other than at commit.

5. Waiting on very long verifications

What if a verification takes 7 hours? Everything behind it is stuck. One verification, and no other development can happen.

Solution: sharding. Cut the task into small pieces.

The entry script declares it in a header line, and the service knows the job should be sharded:

#!/usr/bin/env bash # task-server-shards: * # task-server-compiles: yes # task-server-card-lease: on-demand 

The service splits a job into three stages, each an ordinary sub-task that queues, picks a machine, sets its memory, and retries on its own:

  1. Plan: the entry point writes a plan.json into the plan directory saying how many shards to cut.
  2. Shards: one sub-task per shard, with TASK_SERVER_SHARD=<shard>/<count> in its environment; it judges only its own share and writes verdicts straight into the verdict database.
  3. Merge: starts only once all shards have succeeded. The merge judges nothing fresh; it only reconciles: everything each shard was supposed to judge can be read from the database right now. If one shard fails, its siblings still run to the end and their verdicts still go into the database. On resubmission, only what hasn't been judged gets filled in.

Beyond sharding there are two hard time rules:

  • After two hours of running, the service doesn't stop the task, but it sends the submitting session a reminder: is the scope right? Does it really need to run this long?
  • At four hours cumulative, the service stops it and bounces it, no negotiation. There's only one way forward: split the item, or optimize the algorithm. Shrinking the verification domain or naming fewer items doesn't count as splitting. ### 6. The crashing GPU

Small pieces brought a new problem. Code bugs plus scheduling problems crashed the GPU outright. And when the GPU goes down, it takes the network with it.

I wrote two separate GPU articles about this a couple of days ago, if you're interested.


  • Before handing over a card, the service scrubs it and reads it back before giving it to the task.
  • After a task ends, the service reclaims the card first: reads VRAM, reads compute processes, reads the kernel log for that period. A card that threw an Xid is benched for 60 minutes, and the task is requeued onto another card.
  • A card judged faulty gets the outcome card-fault, and a human is asked on the spot to reset it. No waiting, no retrying. ### 7. Tasks that could only run on one machine

Because of resource constraints, some tasks could, believe it or not, only run on one machine. Copy the project to the second machine? Changes are far too frequent; every copy and every reconciliation makes token consumption explode.

Solution: share the codebase over NFSv4, with both machines mounting it at the same absolute path.

# This host: /etc/exports, one line per shared root <root> <second host address>(rw,sync,no_subtree_check) # Second host: /etc/fstab, mount at the same absolute path as this host <this host address>:<root> <root> nfs4 rw,hard,nofail,_netdev,x-systemd.automount,x-systemd.mount-timeout=30 0 0 

Three things are shared: the repository itself, the data root (working directories and shard artifacts live under it), and the session log directory. The second machine runs in the same live tree — no copying, no sending, no fetching back, not a single character of any path rewritten. What one machine writes, the other reads directly.

The only exception is the build directory: it lives on each machine's own local disk, is created fresh for every run, and is deleted afterwards. It never goes in a shared directory.

In the service this is a startup condition. On startup the service checks four things: is the configuration present, has this host exported, has the second host mounted, and can each side read a probe file written by the other in both directions. If any check fails, the service doesn't start.

The matching principle went into the project rules: no task must be bound to one machine; if one is, that's a defect. The service has no "this host only" list. If a task dispatched to the second machine goes red, let it go red and go fix the script — don't add exceptions to the service.

Well, that's more or less it. The service's own verification tasks had to queue behind those very long tasks, which made verifying and developing the service very slow. The classic awkward situation: the condition for getting in can only be met once you're in. But in the end these problems were solved.

And then, a new storm appeared.
CPU tasks were in fact solved by this point: CPU and memory form a single resource pool, and scheduling is fairly stable. But GPU tasks held their cards hostage.

8. CPU and GPU tied together

A crash-point replay task first builds and records on the CPU, and only at the very end goes onto a card to judge. Yet it holds the card from the moment it starts. Much of the time, the GPU is just waiting.

Solution: decouple. Assign no card at start; lease one only when the task reaches the step that needs it, and return it immediately after use.

The # task-server-card-lease: on-demand line in the entry file header is that declaration. For such tasks the service assigns no card and does no scrub at start, judging memory and CPU like any ordinary task. In the gate it looks like this:

# .claude/gate.d/crash-point-replay.sh (excerpt) crash_point_replay_take_card() { local cards lease_exit [[ -z "$crash_point_replay_card_leased" ]] || return 0 # already holding one: use it if cards="$(crash_point_replay_card_lease take)"; then lease_exit=0; else lease_exit=$?; fi if (( lease_exit != 0 )) || [[ -z "$cards" ]]; then echo " ✗ $1: $2 needs a card; on-demand lease refused, this run did not start" return 1 fi crash_point_replay_card_leased="$cards" echo " · $1: $2 needs a card; leased $cards on demand (returned when this run ends)" } 

Behind lease take, the service picks a free card by card priority, scrubs it, reads it back, and only then hands it over. If all cards are taken, the request waits in line, first come first served. Any card not returned when a task ends is reclaimed by the service.

The tests in the verification package were split into two stages accordingly:

  1. Record only: runs on the CPU and writes the inputs to be judged into the verdict database. Holds no card.
  2. Judge only: leases a card, reads the inputs back from the database, judges them on the card, and returns the card when done. The other half was done too: once a task has a card and enters its GPU stage, its CPU accounting switches to the number of cores actually in use during that card-holding stage. While it waits on the card's results, its CPU slots go to others.

The time a card spends in hand shrank from "the whole task" to "the part that's actually computing."

Finally, more or less there

  • Memory no longer gets eaten by several tasks starting at once. Admission is decided in exactly one place, the service, and only what qualifies gets started.
  • No task holds resources indefinitely. Stuck ones are stopped, long ones get reminders, four-hour ones are bounced.
  • Every outcome has a name, an exit code, and a next step, and it's delivered to the session that started it.
  • Verdicts from everyday runs aren't wasted; the commit-time run picks them up.
    But there are still plenty of problems:

  • Estimated runtime is historical data, not a promise. A run that reuses a lot and one that reuses little can differ enormously.

  • Starting at 4GB and bumping up by 1.5× trades reruns for convenience. A genuinely new 40GB task has to hit the cap six times before it runs stably.

  • The service is code too. Its self-tests run in a simulated world: fake memory wrappers, a fake second machine, fake cards. Running it for real is a stumbling affair.
    Fortunately, after more than 10,000 tasks, it has more or less settled down. Of course — trust me — countless bugs and optimizations are still waiting down the road.

Problems are like the tree in the yard. When the leaves fall, countless branches are laid bare...