Scope an agent evaluation by the shape of the task, not by whether one run looked good. Alex Ratner separates that shape into three axes: environment or input complexity, autonomy horizon, and output complexity. A contest-style coding item with one correct answer needs a unit test. An answer key fits a letter or a single number. A job with a messy environment, many steps, and a long document needs the environment, a rubric, and a grader packaged with the task.

Alex Ratner, co-founder and CEO of Snorkel AI, walked through this on Chain of Thought in April 2026. His team builds data sets and environments for evaluation, tuning, and reinforcement learning, including work with frontier labs, enterprise teams, coding agents, and Harvey's research team on legal deep research.

How do you place an agent task on three eval axes?

Write the task down as a point on environment or input complexity, autonomy horizon, and output complexity before you pick a metric. Ratner treats those three as the directions where agent capability is moving faster than measurement.

Environment or input complexity is how much context the agent must use, at the start and while it works. The simple end is no environment at all: a competition coding problem of one or two sentences. The problem can still be hard. The context is local. The other end looks like a software engineer inside a company: several code bases, internal tools, internal guidelines, JIRA tickets, Slack notes from a PM, and a large code base to search. He called the move from a self-contained written task to that kind of context one of the most important directions. People already run agents there, but the fullest extent of that axis is not yet measured in a scientific way.

Autonomy horizon is the number of steps, plus whether the goal stays still. He said they could have picked a less fancy name. The short end is a coding challenge with one correct answer and a unit test, or a multiple-choice medical or legal question marked A, B, C, or D. The long end is multiple turns, shifting specifications, multiple reasoning steps, and multiple tool calls. Long-horizon tasks are becoming more common. Measuring them out to the length of a real-world task is still at the frontier, and few public benchmarks test it. Most of those, he said, are unsaturated.

Output complexity is how hard the result is to grade. A math item whose inputs fit in one paragraph and whose answer is a letter or a single number is checked against an answer key. A deep legal research output, in his example a 30-page PDF with 50 citations, is tough for humans to verify. On the legal benchmark with Harvey's research team, he described tasks that might take a person five, 10, or 15 hours, with multiple tools, multiple citations, complex reasoning, and criteria that have to be met before the output counts as success.

LLMs have crushed contest problems most people would struggle with. A coding agent might still fall down on a mundane task with messier inputs, messier outputs, and more steps. He tied that pattern to how models are trained with reinforcement learning.

...you can ask a coding agent to do a much more mundane-seeming task, but one that has messier inputs and messier outputs and a longer set of steps required, and it might fall down.

Alex Ratner, Co-founder and CEO at Snorkel AI, on Chain of Thought ep 57

Every AI Agent Has an Evaluati… - Chain of Thought | AI Agents, Infrastructure & Engineering - Apple Podcasts

Podcast Episode · Chain of Thought | AI Agents, Infrastructure & Engineering · April 29 · 43m

favicon podcasts.apple.com

A leaderboard of closed problems will not show that miss. Place the ordinary task on the three axes and test that point. He summarized the evaluation gap as capabilities advancing past our ability to measure them, measurement getting harder because there are many complex capabilities, and a frontier that is jagged and counterintuitive.

Which checks should you run for each axis?

Pick the check from the axis, then run only the checks the task requires. A single aggregate score hides which axis you never measured.

  • For a local input, confirm the prompt is self-contained, so the agent is not graded on context you forgot to supply.
  • For a complex environment, log what the agent could see: repos, tickets, guidelines, tool results. If that context can change during the run, snapshot it at the steps that matter.
  • For a short horizon, an exact match or a unit test is enough when there is one correct answer.
  • For a long horizon, record step count, tool calls, and whether the specification moved. Grade intermediate milestones, not only the last message.
  • For a closed output, compare to the answer key.
  • For a long or structured output, write a rubric that lists the criteria, then run a grader. He said rubrics are how people check a complex answer, with human or automated validation. He listed unit tests, style checks, and milestone objectives as ways to encode checks on a complex output.

On data his team produces, almost 40 to 50 percent of the time that goes into a data point or an environment, at the quality bar they hit, goes into review and quality control. He described that review as labeling: take the outputs and check them against the spec. Budget that review when the output axis is high.

A minimal sketch to adapt, not a drop-in library.

from dataclasses import dataclass @dataclass class AgentTaskEval: name: str input_complexity: str # "local" or "environment" autonomy_horizon: str # "single_answer" or "multi_step" output_complexity: str # "answer_key" or "long_form" def checks_for(task: AgentTaskEval) -> list[str]: checks = [] if task.input_complexity == "environment": checks.append("trace_env_context") else: checks.append("check_local_prompt") if task.autonomy_horizon == "multi_step": checks.append("record_steps_and_goal_shifts") else: checks.append("match_single_answer") if task.output_complexity == "long_form": checks.extend(["apply_rubric", "run_verifier"]) else: checks.append("compare_answer_key") return checks

Fill the three fields from the task, not from the model you hope to ship. checks_for only chooses the family of check. You still write the rubric, the unit test, or the environment trace yourself. If two tasks differ on one axis, give them different specs. Averaging a multiple-choice item with a multi-hour research job produces a number that describes neither task.

When should the environment ship with the dataset?

Ship them together when the agent has to act in an environment. Ratner called the split between data and environments a false dichotomy for that kind of work.

His comparison was a standardized test. A year or two ago, the payload was multiple-choice questions plus an answer key. The tasks he described now look like a student who must use the library computer, look up references, and write a report. The environment is the desk and the computer. The task is the assignment: topic, themes, citations. You still need a rubric, because you cannot ask whether A equals B. You may keep a reference answer. You need graders, which he said are often called verifiers. If the task includes back-and-forth, the environment can include simulated user personas.

So you need the task, you need the environment, you need the potentially the reference answer, you need the rubric or the grading kind of guidelines, and you need the grader or the verifier.

Alex Ratner, Co-founder and CEO at Snorkel AI, on Chain of Thought ep 57

The naming is semantics. Call it an environment, data plus an environment, or data. The valuable package is the combination. Done right, you design the task from the environment and the environment from the task. A cloned tool with no task, no rubric, and no verifier does not tell you whether the agent did the job. A question-and-answer file with no environment does not test an agent that has to look something up.

For a coding agent, that package is concrete. The environment is the repo, the ticket, and the tools. The task is the change request, including any spec that arrived later on Slack. The rubric might mix tests, style checks, and milestone objectives. The verifier is whatever applies that rubric, human or automated. Store those pieces under one task id so a failure points at an axis: missing context, a horizon that ran past the tests, or an output the rubric cannot score.

Public benchmarks still belong in the set. Ratner said people overfit them, and that overfitting does not remove their value. Measure in more than one way: use-case-specific checks, private benchmarks, and public ones. Public benchmarks show where a capability works and where the next gap is. They are not a substitute for the package on your own task.

FAQ

What is the evaluation gap?
Ratner described capabilities advancing faster than measurement, with many complex skills to score, and a jagged frontier that does not match what feels hard for a person. LLMs can do well on contest problems. A coding agent might still fail a mundane task with messier inputs, messier outputs, and more steps.

What is autonomy horizon?
He defined it as how many steps a task takes, including goals that change. A unit-tested coding answer or an A-B-C-D question is the short end. Multi-turn work with shifting specifications is the long end, and he said few public benchmarks test real-world length.

Do I still want public benchmarks if models overfit them?
He said overfitting does not diminish a benchmark's value, and that public benchmarks cannot be the only tool. Keep use-case-specific and private checks next to them so you can see where the agent works on your tasks.

Takeaway

  • Score every task on environment or input complexity, autonomy horizon, and output complexity before you choose a metric.
  • Use an answer key only when the output is a closed value. Use a rubric and a verifier when the output is long or cited.
  • Record step count and goal changes when the horizon is multi-step.
  • If the agent acts in an environment, store the task, the environment, the rubric, and the grader together.
  • Keep a contest-style item so you know the checks run, then add a messier input and a longer step chain before you trust the result.

The full conversation, with the transcript, is on Chain of Thought.

Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.

Drafted with AI assistance from the episode transcripts.