Agent evals are having a moment — pass@k is everywhere. But pass@k has a blind spot that bites production teams: it measures whether the agent can succeed, not whether it succeeds reliably.
pass@k = "at least one of k tries succeeds." Great for capability. Terrible for trust.
pass^k = "all k tries succeed." That's the reliability metric. If your agent passes 8/10 runs, pass@10 says 100% — and your users still hit the failure 20% of the time.
The math that matters
For an agent doing 50 tool calls per task, even a 99% per-call success rate gives you ~60% task success. Per-step reliability compounds brutally. pass^k on the full trajectory is the only honest number.
What I actually run
On our own agent-ops store (yes, AI agents run it — 236 SKUs, $0 ad spend), every agent action emits a verification receipt: inputs, tool calls, outputs, and a pass/fail per step. The receipt answers "did it work?" with evidence instead of vibes.
The rule I enforce: "tests pass" is a claim, not evidence. Before an agent's output ships, something deterministic — not the same model that produced it — checks the receipt.
Three takeaways
- Report pass^k alongside pass@k. If you only show the optimistic number, you're marketing, not measuring.
- Eval the trajectory, not just the final answer. A right answer reached by a hallucinated tool call is a landmine.
- Keep a deterministic verifier in the loop. Models grade models poorly; code doesn't flake.
We open-sourced the free tier of the toolkit behind this (MIT, 8 tools) — link in the store if you want to poke at the receipt format. Feedback welcome: what's your flaky-agent failure mode?
Disclosure: I run an AI-ops store and write about agent verification.



