Amoxi: What a durable agent run costs, measured

Runs per second, checkpoint size, and what happens to a lease when a worker freezes. Measured on a small machine, with a script you can run yourself.

Article 6 ended with a hard kill in the middle of a refund, and data/refunds.log still had exactly one line afterwards. A reader, tercel, asked the questions that come next: how does the lease hold up under latency spikes, can you alert on stuck leases with PostgreSQL, do models handle interrupted tools well, and what does all this checkpointing cost? "Would love to see numbers: how many runs per second and average checkpoint size for a realistic support-agent load."

This article has those numbers. Two of the questions also led to changes in Amoxi 0.6.2, which is out now:

npm install amoxi@latest 

The benchmark

The load is the support agent from articles 5 and 6, handling the request it sees most: "my order arrived damaged, can I get a refund?" Every run is the same conversation:

1. The model says it will look up the order and calls lookup_order.
2. It calls check_refund_policy.
3. It explains the customer is eligible and calls issue_refund, which makes the refund inside ctx.step.
4. It streams the final answer with the refund details.

That is 4 model turns and 3 tool calls per run, with realistic payloads: the order is about 1 KB of JSON (customer, items, shipping, totals) and the system prompt about 1.6 KB.

The model is simulated, so the numbers measure Amoxi and not a provider. It runs at two speeds:

• Hosted pace: 600 ms before the first token, then 60 tokens a second, streamed in small batches. A run spends 5.15 seconds waiting for the model, roughly what a mid-size hosted model takes for this conversation.
• Instant: the model answers at once. Nobody has a model like that, but it shows the most runs Amoxi itself can push through.

The machine is deliberately small: a container with 2 vCPUs and 8 GB of memory, Node.js 22, and PostgreSQL 16 on the same machine, competing for the same two CPUs. PostgreSQL runs with its default durability settings, so every write is committed to disk before Amoxi moves on. Each test starts with a warm-up; run times are medians.

With a real model, Amoxi adds about 2%

A run at hosted pace spends 5.15 seconds waiting for the model. The question is what the framework adds on top, and how that changes as more runs execute at once. ai.runs.concurrency sets how many runs one server executes at the same time (8 by default). On PostgreSQL:

The run time stays almost flat while throughput grows with concurrency, because a run is mostly waiting. While one run waits for its model, the worker executes the others. Up to a few hundred runs at once, checkpointing every turn and tool result costs about a tenth of a second per run.

At 1,024 runs at once, throughput stops growing in step: 94 runs per second instead of the 186 that 1,024 runs of 5.5 seconds would give. The two CPUs are now busy with PostgreSQL commits and with picking up queued runs. That is the point to add a second server, which takes runs from the same PostgreSQL queue with no change to your code.

Two things to take from this. Raise ai.runs.concurrency well above 8 if your agents wait on a hosted model; a waiting run uses almost no CPU. And with a real model, the model sets your run time, not the framework.

The ceiling

With the instant model, nothing waits: every run is pure framework work, from claiming the run to the last checkpoint. This is the most runs one server can complete on this machine:

The gap between memory and the durable stores is the price of durability: each run's 27 writes wait for the disk. With PostgreSQL, Node.js spends only 3.7 ms of CPU on a run; the rest of the time goes to PostgreSQL committing those writes on the same two CPUs. A separate database server, or faster disks, raises the ceiling.

In practice you reach the ceiling only with very fast models or very short runs. With the hosted-pace model, 115 runs per second would mean about 600 runs executing at once.

What a checkpoint holds

A "checkpoint" in Amoxi is not one big snapshot. It is a series of small writes, each one recording the step that just finished, so a run that resumes knows exactly where it stopped. One support run makes 27 writes:

The streamed tokens themselves are not saved. They go straight to the browser; only the finished message is written. That keeps the writes per run fixed by the number of steps, not by the length of the answer.

And the size, once the run is done:

The largest single piece is the order lookup's result, about 900 bytes, stored once as an event and once as a conversation message. Your sizes follow your tool outputs: an agent whose tools return large documents stores more per run.

That makes a million support runs about 12 GB of row data in PostgreSQL. On disk it is somewhat more, because indexes are added and the run record is updated in place 12 times: PostgreSQL keeps old row versions until autovacuum cleans them up. Delete or archive old runs as you would any other table that grows.

Leases under latency spikes

tercel's first question was how the 30-second lease (ai.runs.lease-ms) holds up when a model or a payment provider is slow. The short answer: slowness does not expire it.

A worker holding a run renews its lease every lease-ms / 3, so every 10 seconds by default, for as long as the run is going. The renewal runs on a timer, independently of the model call or tool in progress. A refund call that takes two minutes keeps its lease the whole time.

The lease expires only when the worker stops renewing for 30 seconds: its process crashed or was killed, its machine lost the network, or its event loop froze (a long synchronous computation blocks timers too). Then another worker takes the run over and continues from the last checkpoint.

What if the frozen worker wakes up afterwards and tries to carry on? Every update it makes to the run includes a check that it still holds the lease. Once another worker has taken over, the frozen worker's next write is refused and it stops quietly, so two workers never advance the same run's state.

So lease-ms is a trade-off between two things only. A shorter lease resumes crashed runs sooner. A longer lease tolerates longer freezes, such as a host under memory pressure, without a takeover. The default 30 seconds suits most apps.

Watching for takeovers

The second question was whether you can alert on stuck leases or retry storms. In 0.6.2, when a run is taken over:

• The server that takes it logs a warning: Run taken over from a worker that stopped responding, with the run id, the agent and the count.
• The run's run.status event carries takenOver: true, so a UI can say the run resumed.
• The run records claimedFrom: 'stalled' and takeovers, how many times it has been taken over.

An occasional takeover after a deploy or a crash is normal. A rising rate means workers are freezing or being killed. A single run taken over again and again points at a step that brings its worker down every time. With PostgreSQL, two queries cover both:

-- runs whose worker stopped renewing its lease (adjust to your ai.runs.lease-ms) SELECT id, record->>'agent' AS agent, record->>'workerId' AS worker, heartbeat_at FROM amoxi_runs WHERE status = 'running' AND heartbeat_at < now() - interval '60 seconds'; -- runs taken over more than once SELECT id, record->>'agent' AS agent, (record->>'takeovers')::int AS takeovers FROM amoxi_runs WHERE (record->>'takeovers')::int > 1 ORDER BY created_at DESC LIMIT 50; 

The first should return nothing for long; a row there is a run waiting for a worker to take it over. Put either query behind the monitoring you already have.

Tools that are not idempotent

The third question was the hardest: when a tool that is not marked idempotent is interrupted, Amoxi tells the model the action "may or may not have completed". Do models reliably check first, by looking the order up, rather than simply trying again?

I have not measured that, and the benchmark does not answer it: its model is simulated. It depends on the model, the prompt and the tools available, and a model that checks most of the time is not good enough for money. So I would not rely on the model for a tool that moves money or sends something to a customer. Pick one of these instead:

1. Make it idempotent. Wrap the side effect in ctx.step and pass ctx.callId to the provider as its idempotency key, as article 6 did. This is the only option that gives exactly-once effects without anyone checking. Both parts matter: ctx.step alone has a small gap, a crash after the provider call returns but before the result is recorded, and the provider's key closes it.
2. Require approval. @Tool({ approval: true }) pauses the run until a person approves. After a crash, the person sees the action again and decides.
3. Give the model a way to check. A read-only tool such as lookup_order that shows whether the refund already happened turns "may or may not have completed" into something the model can verify.

The default exists for tools you have not thought about yet. For the ones that matter, decide explicitly.

Files or PostgreSQL?

The default file store (data/ai) needs no setup and survives restarts, but it is built for one server. At hosted pace it tops out at about 27 runs per second on this machine, against 44 for PostgreSQL at the same concurrency, because claiming runs from files costs more CPU as the queue grows. For local development and a single small server it is fine. When you need more than one server, or more than a couple of dozen runs per second, switch to PostgreSQL as article 6 showed; your agent's code does not change.

Run it yourself

The benchmark is in the repository, so you can check these numbers on your own hardware, with your own store:

git clone https://github.com/karawash-projects/amoxi.git cd amoxi npm ci npm run build -w amoxi npm run bench:agents -- --store file --model realistic --concurrency 64 --runs 192 

For PostgreSQL, install the driver with npm install --no-save pg, set DATABASE_URL, and pass --store postgres. --model instant measures the ceiling. The script prints runs per second, run time, CPU per run, writes per run and bytes stored per run, and checks that every run issued exactly one refund. Across the roughly 10,000 runs behind this article, every one did.

Recap

• With a hosted model, durability costs about 2% of a run's time, up to a few hundred runs at once per server.
• One small server completes about 115 runs per second when the model is instant; that is the framework's ceiling, not the usual limit.
• A support run makes 27 small writes and stores about 12 KB in PostgreSQL; streamed tokens are not saved.
• Slow models and tools never expire a lease. Only a worker that stops renewing for 30 seconds loses its runs, and it cannot write to them afterwards.
• 0.6.2 logs and counts takeovers, so you can alert on stuck leases and repeated takeovers.
• For tools that move money, use ctx.step with an idempotency key, or approval, rather than trusting the model to check.

Thanks to tercel for the questions behind this article. If you run the benchmark on other hardware, I would like to see your numbers in the comments.

Amoxi is open source under the Apache License 2.0. Code: github.com/karawash-projects/amoxi. Documentation: karawash-projects.github.io/amoxi.