On November 11, AWS published UAEF on the Machine Learning Blog, an open-source framework for closed-loop evaluation of AI agents. Four days later I pointed it at ten sessions my own agent had answered back in July. One of those answers was false. AgentCore Evaluations had scored it 1.0, "Perfectly Correct." UAEF, given a ground truth I had checked by hand in Cost Explorer, scored it 0.00 with one judge and 0.10 with another.
This article is about that gap, and the gap comes from what each evaluator was asked to compare the answer against.
The builder is back, and so is the test step
When I started in IT and programming, almost everyone was a builder. There weren't many roles. You went from an idea to the software and took care of nearly everything along the way: understanding the idea, design, implementation, test, deploy, and then you started again.
Today there's a turn back toward that. The builder is relevant again, and being able to go from an idea to a 100% working piece of software, moving fast through every intermediate step with AI helping, matters a lot. When the thing you're building is agentic, one of those steps starts carrying more weight than the rest. Evaluating. An agent's output can sound very reasonable and be 100% wrong, and it isn't always easy to tell which one you're looking at.
So when AWS shipped an open-source framework with some maturity behind it, my honest first reaction was relief. I wanted a baseline to evaluate and iterate on what I build. "Maturity" is relative here (the repo is v0.2.0 and marks itself Alpha), but a shared baseline beats the eval script I'd otherwise write alone. My second reaction was to test it against the one failure I already had on file.
The answer that scored perfect
In July I ran Gate 3 of an AgentCore lab. The agent is InvoiceExplainer: a Strands agent on AgentCore Runtime, Nova Pro as the model, and exactly one tool, get_cost_breakdown, which calls Cost Explorer. I wrote 10 questions, verified every expected answer by hand against Cost Explorer for the same window the tool queries, and ran four of AgentCore's built-in evaluators over the sessions.
Question 5 was "Is any service showing a negative cost, and why would that happen?" Cost Explorer had two negative lines in that window. AWS Key Management Service at -0.000000096 per day, EC2-Other at -0.0000000048 per day. Tiny, but real.
The agent answered: "There are no services showing a negative cost in this AWS account."
That's false, and the chain that produced it was mine. My deployed tool summed negative amounts into a credits accumulator, threw away the service name, and formatted the total with four decimals. So -0.0000007 became Credits applied: $-0.0000. The agent read zero and turned it into a claim about the world. My system prompt said "Never invent figures." The agent didn't invent a figure. It invented an absence.
Builtin.Correctness gave it 1.0 with this explanation: "The candidate response correctly states there are no services showing a negative cost... no credits were applied ($-0.0000 effectively means zero)." Builtin.Faithfulness also gave it 1.0. Correctness scored all 10 ground-truth sessions 1.0. Zero of the four evaluators caught the only factual error in the run.
To be fair to AgentCore, I ran Correctness without a reference answer, in its reference-free mode. It judged the answer against the trace, and the trace said $-0.0000. The answer was faithful to a tool output that was wrong. ToolSelectionAccuracy even caught something real that day (a duplicated tool call on q7). But an evaluator that only reads the trace has no way to know the trace is wrong.
What UAEF actually ships
Per the AWS post, UAEF ships 36 built-in metrics across 7 dimensions, normalizes every score to [0,1], and is designed around a closed loop: evaluate, learn what regressed, change the prompt, model or tools, evaluate again. The default regression threshold is 5%, and the judge treats agent content as untrusted input. It's Apache-2.0 at github.com/awslabs/uaef, with a first commit on 2026-09-16, and I pinned commit dc55d8c (v0.2.0, classifier "Development Status 3 - Alpha"). AWS says 12 organizations have used it. That's AWS's claim and I have no way to check it.
The post also says something I'd underline for anyone wiring this into a pipeline: "Judge scores are evaluation signal, not a safety decision, and should not gate a security control."
For my question, the relevant piece is smaller. UAEF takes a GroundTruth with an expected_output. That's a slot where I can put the world instead of the trace.
The replay
I didn't re-invoke the agent. The July ground truth is a snapshot, and today's Cost Explorer would return different data, so a live call would be judged against stale answers. Instead I replayed the saved transcripts through UAEF's AgentCoreAdapter, which accepts pre-collected traces. This is the core of replay.py, trimmed:
from uaef.adapters import AgentCoreAdapter from uaef.api import evaluate from uaef.models import GroundTruth, ToolCall # Rebuild the trace from the July transcript. No span data survived, # so the tool call is attached by hand. Its arguments were never captured. trace = AgentCoreAdapter().transform_to_canonical({ "user_input": q["question"], "final_response": parsed["final_response"], "session_id": parsed["session_id"], "agent_runtime_id": "InvoiceExplainer_InvoiceExplainer-6OivUsDx4c", }) trace.tool_calls = [ToolCall(name="get_cost_breakdown", arguments={}, result=TOOL_OUTPUT, timestamp=now)] # The answer I verified by hand in Cost Explorer. This is the "world". gt = GroundTruth(expected_output=q["expected_response"], expected_tool_calls=[ToolCall(name="get_cost_breakdown", arguments={}, timestamp=now)]) # Run 1: hallucination judged against what the tool returned (the trace) res = evaluate(trace=trace, ground_truth=gt, metrics=["tool_selection_accuracy", "accuracy", "hallucination_score", "answer_relevance", "completeness"], context=[TOOL_OUTPUT]) # Run 2: same metric, judged against the facts I checked by hand (the world) res_w = evaluate(trace=trace, ground_truth=gt, metrics=["hallucination_score"], context=world_facts) Why run hallucination_score twice? Because it returns no score without context, and the choice of context is exactly the variable I wanted to isolate.
Two things bit me before the first real run. The config has an UAEF_ENABLE_CACHING flag that defaults to true, so I set it to false to be safe. I later read the installed package: the flag is read in config.py and not referenced anywhere else, so in 0.2.0 it appears to do nothing for judge calls. Every run made real calls either way (each evaluation took seconds in the log).
The second one took longer. UAEF 0.2.0 sends temperature on every InvokeModel call. In my account, Sonnet 5.5, Opus 5.5, Haiku 5.5, Sonnet 5 and Opus 4.8 all rejected it with "temperature is deprecated for this model". The default judge, Sonnet 4.6, failed for a different reason: an AWS Marketplace AccessDenied. Haiku 4.5 and Opus 4.6 accepted the call. I could have patched the library. I didn't.
## ADR: Run UAEF unmodified, pick judges that accept its request shape Context: UAEF 0.2.0 sends `temperature` on every judge call. Five current Claude models reject it in this account; the default judge hits a Marketplace AccessDenied. Decision: Keep the pinned commit unmodified. Judge = Haiku 4.5 (3 runs) plus Opus 4.6 (1 run) as a control. Why: The point is to test the framework as AWS ships it. A patched UAEF would be my framework, and the results would not transfer to anyone who runs `pip install` from the repo. Consequence: Older judges. Two judges from different tiers, which turned out to be useful. What came back

AgentCore's trace-based Correctness stayed at 1.0 on all ten questions, while UAEF's ground-truth accuracy dropped on q5, the false answer. q7 is a ground-truth mapping artifact, not an agent error.
| Question | AgentCore Correctness | UAEF accuracy (Haiku 4.5 / Opus 4.6) |
|---|---|---|
| q1 total 7-day spend | 1 | 0.85 / 0.90 |
| q2 top service | 1 | 0.95 / 1.00 |
| q3 AgentCore cost | 1 | 0.95 / 1.00 |
| q4 Bedrock spend | 1 | 0.85 / 0.80 |
| q5 negative costs (false answer) | 1 | 0.00 / 0.10 |
| q6 services over one cent | 1 | 0.95 / 1.00 |
| q7 day of largest charge | 1 | 0.00 / 0.00 |
| q8 CloudWatch cost | 1 | 1.00 / 1.00 |
| q9 30-day spend | 1 | 0.85 / 1.00 |
| q10 what to watch | 1 | 0.75 / 0.70 |
With a cutoff at 0.5, both judges flagged q5 and q7. AgentCore Correctness flagged nothing. Haiku's reasoning on q5 opened with "The candidate response is factually incorrect and contradicts the reference answer."
Now q7, which is on me. The agent said it couldn't determine the day because its tool has no daily breakdown, which is true. My lab assertion for q7 accepted exactly that ("Either names 2026-07-26 or states the daily breakdown is not available"). But UAEF only receives expected_output, and I mapped the date into it, not the escape clause. So UAEF's 0.00 on q7 is a false positive created by how I mapped my own ground truth, not an agent error. I left it as is rather than re-tune after seeing the result.
The trace problem follows you into UAEF
This part changed what I took away from the replay.
Look at hallucination_score when the context is the tool output. The false q5 answer got 0.30 from Haiku and 0.50 from Opus. The correct q6 answer got 0.00 from Haiku, below the false one, because it said "no credits applied" and the trace literally contains a Credits applied line. Opus ordered them the right way (q6 at 0.80, q5 at 0.50), but the gap is thin, and its reasoning on q5, with the trace as context, even says the response "correctly identifies that there are no services showing a negative cost." Against the trace, the metric is weak at telling the right answer from the wrong one, and with Haiku it pointed the wrong way.
Swap the context for the facts I verified in Cost Explorer and q5 drops to 0.15 with both judges, while q6 goes to 0.95 (Haiku) and 0.70 (Opus). The framework, the metric and the answers didn't change. The reference did, and so did the verdict.
So no, this doesn't show UAEF is better than AgentCore Evaluations. They got different inputs. AgentCore judged against the trace, UAEF accuracy judged against my hand-checked answers, and they used different judges and different prompts on top of that. Give UAEF only the trace and it misses the same error.

The two judges agree on accuracy against a reference answer (0.06) and drift apart on the open-ended metrics, up to 0.28 on trace-context hallucination.
The judge itself is a variable too. On accuracy, Haiku and Opus nearly agree: mean absolute difference 0.06, and no question differs by 0.25 or more. On completeness, they differ by 0.25 or more on 5 of 10 questions, and so they do on trace-based hallucination. Haiku's three runs were identical in all 50 question-metric cells (spread 0.00). The judge temperature defaults to 0.0 in the config, which fits, but I didn't test any other setting, so I'm reporting it as observed.
What this replay doesn't prove
These are replayed sessions from 2026-07-30 transcripts. The agent was not re-invoked, so I never touched AgentCoreAdapter.invoke() against a live runtime ARN, and I didn't use persist=True (the DynamoDB and S3 experiment tracking that powers the regression loop). The closed loop, the headline feature, is untested here.
The traces are reconstructed. Tool arguments were never captured, so tool_selection_accuracy came back 1.0 by construction and tells you nothing. q7 is the mapping artifact I described. The AgentCore and UAEF numbers come from different judges, prompts and reference data, so the table compares two evaluation setups. AgentCore's Correctness can also take an expected response per trace, as its ground truth evaluations documentation describes. I didn't pass one, so the table sets UAEF with a reference against AgentCore without one, and the like-for-like run is still to do. And I didn't measure judge cost in dollars, so I have no number to give you on that.
What I'm keeping
When the evaluator only sees the trace, your tool is your ground truth. Every bug in the tool becomes a blind spot in the eval. My four-decimal formatter misled the agent first and the evaluator second, and nothing in the trace could expose it.
The ground-truth file was the most boring artifact in the whole lab: ten questions and the numbers I checked by hand. It ended up being the only thing in the pipeline that knew q5 was wrong. Documentation isn't overhead, it's thinking made visible, and here it was also the test oracle.
Mapping it is its own source of error, though. My q7 assertion had an escape clause that expected_output couldn't express, and I paid for it with a false positive. Write ground truth in the shape the evaluator actually consumes.
Pin the judge and run a second one. Scoring accuracy against a reference, my two judges agreed. On the open-ended metrics they split on half the questions. If I'd run only Haiku, I'd have trusted completeness numbers that Opus moved by 0.25 or more on 5 of 10 questions.
And keep AWS's own warning in view. I wouldn't gate anything at runtime on a judge score.
Reproduce it
The PoC lives in uaef-agent-eval-poc, with the ten transcripts, the ground truth and AgentCore's saved scores included. UAEF is pinned to commit dc55d8ca54331176bb16744b9cc680f806075700 in pyproject.toml and installed unmodified.
aws sso login --profile <profile> export AWS_PROFILE=<profile> UAEF_POC_ACCOUNT_ID=<your 12-digit account id> # Full run: Haiku 4.5 x3, then Opus 4.6 x1 as control. # run-all.sh also exports UAEF_ENABLE_CACHING=false (see the note above). ./run-all.sh # No AWS calls: deterministic metric only (tool_selection_accuracy) uv run python replay.py --offline replay.py aborts unless the caller's account matches UAEF_POC_ACCOUNT_ID, so a wrong profile can't bill the wrong account. Results land in results/, and summarize.py builds the table above.
Your turn
Quick: If you evaluate agents today, what do your evaluators compare against, the trace or something you verified outside it? One line in the comments is plenty.
Medium: Take one agent answer you trust and check it by hand against the source system. Tell me if the evaluator and the world agree.
Deep: If you've built a ground-truth set for an agent, share how you keep it in sync with data that changes. Mine went stale within four months, which is why I replayed instead of re-invoking. I'd like to learn how others solve that.
Build. Document. Share. Repeat.

