A benchmark consumer can produce a confident-looking report while silently changing the meaning of the source data. A common way to do it is to convert a missing or indeterminate result into a failure, then count it with measured failures. StagingBench's per-image score files offer a concrete case for keeping a third state.
The methodology v0.2 is a draft, updated 20 October 2026. It describes a framing check based on an estimated transform between the original and staged image. If alignment fails because matching points are insufficient or the transform is implausible, framing is reported as unknown. The architecture measure falls back to the raw, unaligned score and is flagged in the per-image file. That fallback has a different interpretation from an aligned result.
In the partial Collov AI results recorded on 19 October, Room 04 has alignment.ok set to false, a reason of too few inliers, and framing.pass set to JSON null. Room 05 has successful alignment and framing.pass set to true. The Pedra versus Collov comparison presents partial results and says vendors had not yet had their right of reply on the displayed numbers. Room 04's file reports 21 inliers; the test log records a minimum of 25 for that run. Its 56.5 drift value is explicitly unaligned, so it should carry that qualification into any downstream display.
The following Python script reads the actual local score files. Run it from the repository root; it uses only the standard library. It reports the framing state and whether the architecture score was aligned, without treating null as false.
import json from pathlib import Path for room in ("04", "05"): path = Path("results/collov") / f"{room}-score.json" score = json.loads(path.read_text(encoding="utf-8")) passed = score["framing"]["pass"] state = "unknown" if passed is None else ("pass" if passed else "fail") architecture = score["architecture"] basis = "aligned" if architecture["aligned"] else "unaligned fallback" print(f"Room {room}: {state}; drift {architecture['drift_score']} ({basis})") The output is:
Room 04: unknown; drift 56.5 (unaligned fallback) Room 05: pass; drift 21.4 (aligned) The distinction survives aggregation only if the data model preserves it. A report could count passes, failures and unknowns separately and retain the alignment flag next to each drift value. If it offers a rate, its denominator should be stated: measured cases and all attempted cases answer different questions.
This script reads recorded measurements; it does not recompute alignment or establish why an image changed. The methodology also says that a single global similarity transform cannot model a changed camera position or lens, and that the edge-based measure has limits. Keep the score file, the displayed result and the original image together when reviewing a case. That makes an unknown state a prompt to inspect the evidence, rather than a value that disappears during processing.