Domain 3 of the AWS Certified AI Practitioner exam (AIF-C01), "Applications of Foundation Models", is 28% of the scored content. One of its task statements is all about evaluation: how do you know a foundation model (FM) is actually good enough? The exam guide names the tools explicitly, so they are worth learning properly.
Three ways to evaluate
- Automatic metrics on a dataset: fast, cheap, repeatable. You compare model output to reference answers.
- Human-in-the-loop evaluation: people rate the outputs. Slower and more expensive, but the only way to judge things like tone, brand voice or helpfulness.
- LLM-as-a-judge: another model scores the outputs against criteria you define. Scales better than humans for subjective quality.
Benchmark datasets sit under the first option: standard, public datasets that let you compare models on the same questions.
The metrics, matched to the task
| Metric | Think of it as | Typical task |
|---|---|---|
| ROUGE | How much of the reference text the output recalls (word overlap) | Summarization |
| BLEU | How precisely the output matches reference wording | Machine translation |
| BERTScore | Semantic similarity using embeddings, so synonyms still count | Summarization, paraphrasing |
| F1 / accuracy | Correct answers vs. expected answers | Question answering, classification |
A quick memory hook: ROUGE = Recall-Oriented, it is in the name (Recall-Oriented Understudy for Gisting Evaluation). BLEU = Bilingual, Bilingual Evaluation Understudy, so translation.
What Amazon Bedrock Model Evaluation does
Bedrock can run automatic evaluation jobs with built-in datasets. Per task type it measures accuracy, robustness and toxicity. For example:
- Text summarization: accuracy is computed with BERTScore.
- Question and answer: accuracy is computed with NLP-F1.
- Text classification: accuracy is classification accuracy.
- General text generation: accuracy, robustness and toxicity (for example with the RealToxicityPrompts dataset for toxicity).
Robustness checks how much the score changes when the prompt is slightly perturbed. Toxicity checks for harmful content.
Bedrock also supports evaluations with human workers, for criteria a metric can't capture.
Source: Model evaluation task types in Amazon Bedrock.
Trap 1: a good metric is not a good business result
The exam guide also lists business alignment metrics: task completion rate, user satisfaction, cost per interaction, productivity and user engagement. If a question asks whether the model "meets business objectives", the answer is one of these, not ROUGE.
Trap 2: human evaluation is not always the "best" answer
When the scenario says "subjective", "brand style" or "friendliness", humans (or LLM-as-a-judge) fit. When it says "quickly compare several models at low cost", automatic evaluation with a benchmark dataset is what AWS wants.
Trap 3: evaluate the application, not only the model
A RAG chatbot or an agent can fail even with a strong model: wrong documents retrieved, a tool call that fails, a workflow that stops halfway. The exam guide explicitly mentions evaluating applications built on FMs, such as RAG, agents and workflows.
A sample question
A company wants to compare three foundation models for summarizing customer support tickets. It needs a quick, low-cost comparison that measures how well each summary preserves the meaning of a reference summary, even when different words are used. Which approach fits best?
A. Run a human evaluation job with a private workforce
B. Use automatic evaluation with BERTScore
C. Measure BLEU against a translated reference
D. Track cost per interaction in production
Answer: B. Quick and low-cost points to automatic evaluation. "Preserves the meaning, even with different words" points to BERTScore, which uses semantic similarity instead of exact word overlap. BLEU is word-precision based and typical for translation; cost per interaction is a business metric, not a quality metric.
Checklist before exam day
- ROUGE → summarization (recall), BLEU → translation (precision), BERTScore → meaning.
- Subjective quality → human evaluation or LLM-as-a-judge.
- Business questions → completion rate, satisfaction, cost per interaction.
- Read the official AIF-C01 exam guide for the full task statements.
I wrote 6 AIF-C01 practice exams with an explanation for every option. Until November 28 it's $34.99 with this link: https://www.udemy.com/course/aws-ai-practitioner-aif-c01-6-practice-exams/?couponCode=AIFC01LAUNCH. Not affiliated with or endorsed by AWS.