Normal software fails loudly. An exception, a 500, a red test. AI agents usually don't. After a prompt tweak or a model upgrade, an agent can start:

  • calling the same tool three times in a row
  • skipping a step that matters, like checking a policy before acting
  • taking more steps, and costing more, for the same task The final answer might even look fine. Nothing errors, so nobody notices until a user hits the bad path or the bill arrives.

I wanted a test that fails when that happens, so I built it into agentcheck, an open-source TypeScript library and CLI.

The idea: test the trace, not just the answer
An agent's behavior is the sequence of tool calls it makes. If you save that sequence for a run you trust, you can check new runs against rules and against that baseline.

Take a refund agent. A good run looks like this:

lookup_order → check_refund_policy → issue_refund
After a prompt change, a run looked like this:

lookup_order → lookup_order → lookup_order → issue_refund
It looped on the lookup and skipped the policy check before issuing the refund. The customer still got their refund, so a test that only checks the final answer would pass.

Step 1: record a trace
If you use the Vercel AI SDK, convert the result's steps directly. Both the older args/result field names and the newer input/output names work:

 import { generateText, stepCountIs } from "ai"; import { fromAISteps } from "@ydeshpa4/agentcheck"; import { writeFileSync } from "node:fs"; const result = await generateText({ model, tools, prompt, stopWhen: stepCountIs(10) }); const trace = fromAISteps(result.steps, { pricing: { inputPerMillion: 3, outputPerMillion: 15 }, // your model's prices }); writeFileSync("traces/refund.json", JSON.stringify(trace, null, 2));

With any other framework, wrap your tool calls with TraceRecorder. It records args, output or error, duration, tokens and cost:

import { TraceRecorder } from "@ydeshpa4/agentcheck"; const rec = new TraceRecorder({ pricing: { inputPerMillion: 3, outputPerMillion: 15 } }); rec.step(); const order = await rec.recordCall("lookup_order", { id }, () => lookupOrder(id)); rec.addUsage({ inputTokens: 2000, outputTokens: 300 });

Step 2: assert rules

agentcheck check current.json \ --max-steps 4 \ --order lookup_order,check_refund_policy,issue_refund \ --max-repeat 1 \ --max-cost 0.03

Here's the real output for the bad run:

current.json: 5 steps, 4 tool calls, $0.0615 ✗ maxSteps: 5 steps, limit is 4 ✗ maxCostUsd: cost $0.061500, limit is $0.03 ✗ order: expected order lookup_order → check_refund_policy → issue_refund: "check_refund_policy" did not appear after "lookup_order". Actual: lookup_order → lookup_order → lookup_order → issue_refund ✗ maxRepeat: "lookup_order" was called 3 times with identical args (limit 1); possible loop FAIL (4 failure(s))

--order checks a subsequence: other calls can happen in between, but these must appear in this order. --max-repeat counts calls with identical name and arguments, which is a cheap, reliable loop detector. There's also --must-call, --must-not-call, --max-tool-calls and --no-tool-errors.

Step 3: diff against the baseline**
Rules catch what you thought to write down. A diff catches everything else:
agentcheck diff baseline.json current.json

steps: 4 → 5 (+1) cost: $0.042 → $0.0615 (+46.4%) tool sequence: lookup_order - check_refund_policy + lookup_order + lookup_order issue_refund ✗ tool sequence changed: removed check_refund_policy; added lookup_order, lookup_order ✗ steps increased by 1 (4 → 5), allowed 0 ✗ cost increased 46.4% ($0.042 → $0.0615), allowed 10% FAIL: regression

The tool sequence is shown as a line diff, computed with a longest-common-subsequence algorithm, so you can see exactly what changed. You set how much drift is acceptable with --max-step-increase and --max-cost-increase-pct, or allow sequence changes when you expect them.

Both commands exit with code 1 on failure, so they work as a CI step with no extra setup.

Step 4: replay tools offline
Re-running an agent against real tools is slow, costs money and can have side effects (you don't want test runs issuing real refunds). createReplayTools turns a recorded trace into fake tools that return the recorded outputs:

import { createReplayTools } from "@ydeshpa4/agentcheck"; const replay = createReplayTools(baselineTrace); // { strict: true } to require identical args const tools = { lookup_order: tool({ description, inputSchema, execute: replay.tools.lookup_order }), // ... };

Calls are matched by a hash of their arguments. Recorded errors are thrown again, so error-handling paths get tested too. replay.unused() lists the recorded calls the new run never made.

One caveat: this replays the tools, not the model. The model is still called and still isn't deterministic. For a fully offline test, stub the model as well, for example with the AI SDK's mock language model.

What this doesn't do
It doesn't judge whether the final answer is good. That's what eval frameworks and LLM-as-judge setups are for. agentcheck covers the part those usually skip: whether the agent took a sane path to get there, at a sane cost. The two work well together.

Try it

npm install --save-dev @ydeshpa4/agentcheck

Next on the roadmap are a GitHub Action that posts the diff as a PR comment, and adapters for LangGraph.js and the OpenAI Agents SDK. Issues and suggestions are welcome.