There's a new model almost every day this month. In October alone: GPT-6.1 Terra, Kimi K3.1 Pro, DeepSeek V4.1 Flash, MiniMax M3.1 Flash, Beam 1.0, Mercury 3, and more.

If your LLM feature is a prompt and a feeling, each of those releases is a question you can't answer: should we switch? Is it better for us? Cheaper at the same quality? Worse in some way we'd only find out from users?

If your feature has an eval set, the question takes an afternoon.

What an eval set actually is

Not a benchmark. A list of your inputs with your definition of a correct answer, and a script that scores any model, prompt or design against it.

For a text-to-SQL feature I'm working on, each case is:

- question: "Which leads haven't been contacted in 30 days?" reference_sql: | SELECT l.id FROM leads l LEFT JOIN activities a ON a.lead_id = l.id GROUP BY l.id HAVING max(a.created_at) < now() - interval '30 days' OR max(a.created_at) IS NULL 

The scorer runs both queries and compares the rows they return, not the SQL text. Two different queries can both be correct. A query that runs fine and returns believable numbers can still be wrong.

The cases that matter are the near misses

When I wrote about that design, a commenter (Max) made a point I've kept: some failures are loud and some are quiet. A missing table gives broken SQL, easy to catch. A wrong join between two tables that both exist returns plausible numbers that nobody questions.

An eval set built from easy examples only catches the loud ones. The useful cases are the ones designed to tempt the model:

  • two tables with similar names
  • two join paths that both look reasonable
  • a column that's nullable where the model will assume it isn't
  • a question that sounds like one thing and means another

Write those on purpose. Each real bug you find becomes a new case.

Measure the things that cost money too

An eval isn't only about accuracy. When I compared "retrieve 5 tables" against "cache the whole schema", the deciding number turned out to be cache hit rate: every cold start pays the full write price. I'd assumed about 10 cache writes a day. The eval logs the real number from the API's usage fields.

So score everything that would change your decision:

Measure Why
Correctness (by result, not text) The point
Cost per question Includes cache misses, not just list price
Latency (p50 and p95) Users feel the tail
Refusals / errors A cheap model that fails 5% of the time isn't cheap

Start small

Thirty cases is enough to start. Real questions from real users are best. If you don't have users yet, write the questions you're most afraid of getting wrong.

Run it on every prompt change and every model you're considering. Keep the history. Over time the eval set becomes the most valuable file in the project, because it's the only one that still matters when everything else is replaced.

What I'd tell you before you start

Prompts are temporary. Models are temporary. Your list of "here's what correct looks like for us" outlives both. Build that first, and every new release becomes a quick check instead of a guess.

Model list: "AI Model Releases: October 2026 Tracker", Digital Applied.