How do you know if your autonomous AI agent is actually getting better?

Today, the industry defaults to LLM-as-a-Judge (using GPT-4 to grade agent traces). But this approach has three critical flaws:

  1. Severe Token Waste: Evaluating 100 multi-turn agent runs burns $50+ in evaluation API calls.
  2. Non-Deterministic Flakiness: The judge LLM gives different grades to the identical agent output on different days due to model drift.
  3. Security Telemetry Leaks: Uploading agent trajectories to third-party evaluation clouds risks exposing sensitive customer data and private tool outputs.

To solve this, we created agent-eval-harness — a 100% deterministic, zero-cloud evaluation engine for autonomous agents.

Invariant Assertion Matrices & SHA-256 Merkle Receipts

Instead of asking an LLM to subjectively "judge" an agent:

  1. State Invariant Assertions: Define formal criteria (e.g. required JSON schema, tool call sequence constraints, allowed latency bounds).
  2. Red-Team Leak Scanner: Deterministically scans agent outputs for unencrypted secrets, API keys, and SSRF private IP leaks.
  3. Cryptographic SHA-256 Merkle Receipts: Every evaluation produces an immutable cryptographic receipt verifying that test suite N passed on commit X.

Hands-on Usage in Python

Install via pip:

pip install agent-eval-harness

Evaluate an agent run:

from agent_eval_harness import AgentEvalHarness, EvaluationSuite, AssertionRule harness = AgentEvalHarness.create_default() # Define formal evaluation criteria suite = EvaluationSuite( name="customer-support-agent-v1", rules=[ AssertionRule.schema_valid(), AssertionRule.tool_sequence_valid(expected=["search_kb", "draft_reply"]), AssertionRule.no_secret_leaks(), AssertionRule.max_latency_ms(2500) ] ) # Run evaluation on agent trace report = harness.evaluate_run( trace_path="./agent_trace_run_491.json", suite=suite ) print(f"Passed: {report.passed} ({report.score * 100:.1f}%)") print(f"Merkle Hash Receipt: {report.receipt_hash}") print(f"Zero Cloud API calls: $0.00 spent on LLM-judge tokens")

Sovereign Ecosystem & Open Source