Your eval suite scored 87% on Tuesday and 84% on Thursday. Nobody changed the prompt. Was it the model, the dataset, the grader, or noise? If the only thing you saved was the score, you can't tell, and a 3-point swing is exactly the size of change that drives real decisions about shipping a prompt or switching models.

An eval result is only useful if you can say what produced it. That means treating each run like a build: record its inputs, keep its raw outputs, and separate the parts that can be reproduced exactly from the parts that can't.

Why "temperature 0" doesn't make a run repeatable

It's tempting to set temperature to 0 and assume reruns will match. In practice they often don't. In Defeating Nondeterminism in LLM Inference (Thinking Machines Lab, September 10, 2025), Horace He and colleagues show that, even at temperature 0, LLM APIs don't behave deterministically in practice. Their explanation is that many inference kernels lack "batch invariance": a request's numerics can depend on how many other requests share its batch. They identify the primary reason endpoints are nondeterministic as this: "the load (and thus batch-size) nondeterministically varies."

Provider controls don't remove this either. OpenAI's Create chat completion reference describes the seed parameter as a best effort to sample deterministically, says "Determinism is not guaranteed", points to the system_fingerprint response field to monitor backend changes, and currently marks seed as deprecated.

The practical conclusion: you usually can't make generation bit-for-bit repeatable through a hosted API. You can make everything around it repeatable, and you can measure how much the generation step moves.

Split the run into two stages 

 dataset (hashed) --+ prompt version ----+--> generate --> raw outputs --> score model + params ----+ (varies) (stored) (exact) | | rescore any time report

Generation is the noisy stage. Scoring should be exact: given the same stored outputs and the same grader version, it must produce the same number every time. If your grader is itself an LLM, record its model and prompt version too, and treat it as a second noisy stage.

A run manifest

Before a run starts, write down everything that could change its result. After it finishes, store the raw outputs next to the manifest.

import hashlib, platform from datetime import datetime, timezone def sha256_file(path: str) -> str: h = hashlib.sha256() with open(path, "rb") as f: for chunk in iter(lambda: f.read(65536), b""): h.update(chunk) return h.hexdigest() def make_manifest(dataset_path, prompt_version, model, params, grader_version): return { "run_started": datetime.now(timezone.utc).isoformat(), "dataset_sha256": sha256_file(dataset_path), "prompt_version": prompt_version, # e.g. git tag of the template  "model_requested": model, # pinned ID, not an alias  "params": params, # temperature, max tokens, seed...  "grader_version": grader_version, "python": platform.python_version(), "provider_reported": [], # filled per response  } def record_response(manifest, response_meta: dict): # keep whatever the provider returns: model name, fingerprint, etc.  manifest["provider_reported"].append(response_meta)

The provider_reported list matters. If the model name or a backend fingerprint returned by the provider changes partway through a run, that's a fact you want in the record, not something you rediscover later.

Score from stored outputs, and test that

Keep scoring as a pure function of stored outputs, then lock that property in with a test:

def score(outputs: list[dict]) -> float: correct = sum(1 for o in outputs if o["answer"].strip() == o["expected"]) return correct / len(outputs) def test_rescoring_is_exact(): outputs = [ {"answer": "42 ", "expected": "42"}, {"answer": "41", "expected": "42"}, ] first = score(outputs) assert all(score(outputs) == first for _ in range(100)) assert first == 0.5

Now a grader change can be evaluated on its own: rescore last month's stored outputs with the new grader and compare. No generation calls, no noise.

Measure the noise instead of ignoring it

Since generation varies, run it more than once. Repeat each case a few times (say three or five) with the same manifest and report the spread, not only the mean:

from statistics import mean, pstdev def summarize(trial_scores: list[float]) -> dict: return {"mean": round(mean(trial_scores), 4), "spread": round(pstdev(trial_scores), 4), "trials": len(trial_scores)}

If three reruns of an unchanged setup range from 84% to 87%, a 3-point "improvement" from a prompt edit isn't evidence of anything yet. Recording the spread gives the team a shared threshold: differences smaller than it aren't worth arguing about.

Rules for comparing two runs

  • Compare only when manifests differ in one field. If the dataset hash and the prompt version both changed, you can't attribute the difference to either.
  • Pin model IDs in the manifest. An alias that moves to a new snapshot changes the model under you.
  • Keep raw outputs as long as you keep scores. A score without its outputs can't be rescored or audited.
  • Version the dataset like code. A hash catches silent edits to "the same" test file.

Where this fits

Being able to say what produced a result, after the fact, is the problem we work on at Obligra. Obligra Verify™ is a system of record for AI-assisted decisions, in public preview, designed to retain the evidence your application selects. The manifest habit above is useful wherever your eval results live.

This article was generated by an AI agent (Claude) for the Obligra Verify team and checked against the linked sources.