Reproducible is not correct

Here is a measurement you can reproduce in eight lines, plus a dozen-line helper. It is from the example plans that ship with my floor-plan compiler — a U-shaped house built around a courtyard:

import { readFileSync } from "node:fs"; import { describe } from "@chanmeng666/archlang"; // Ray casting: count crossings of a ray from `point` towards +x. Points on an edge count as inside. function pointInPolygon({ x, y }, polygon) { let inside = false; for (let i = 0, j = polygon.length - 1; i < polygon.length; j = i++) { const a = polygon[i], b = polygon[j]; const cross = (b.x - a.x) * (y - a.y) - (b.y - a.y) * (x - a.x); const onEdge = cross === 0 && Math.min(a.x, b.x) <= x && x <= Math.max(a.x, b.x) && Math.min(a.y, b.y) <= y && y <= Math.max(a.y, b.y); if (onEdge) return true; if ((a.y > y) !== (b.y > y) && x < a.x + ((y - a.y) * (b.x - a.x)) / (b.y - a.y)) inside = !inside; } return inside; } const d = describe(readFileSync("examples/courtyard-house.arch", "utf8")); for (const r of d.rooms) { const c = { x: r.bbox.x + r.bbox.w / 2, y: r.bbox.y + r.bbox.h / 2 }; console.log(r.label, r.floor_polygon.length, "pts, bbox centre inside floor?", pointInPolygon(c, r.floor_polygon)); } // Living 4 pts, bbox centre inside floor? true // Entry 4 pts, bbox centre inside floor? true // Dining 4 pts, bbox centre inside floor? true // Gallery 8 pts, bbox centre inside floor? false <-- here // Bedroom 1 4 pts, bbox centre inside floor? true // …

The Gallery is an eight-point room wrapped around a courtyard. Its bounding box has a centre, that centre is a perfectly good coordinate, and it is outside the room. Anything that reached for that point — to place a label, to start a route, to decide which side of a wall is "outside" — got a confident, silent, wrong answer.

That is not one bug. It is a defect class, and my project shipped six instances of it before anybody noticed. This post is about the three classes I have actually shipped, and what each one taught me about the difference between a gate that is green and a system that is right.

Class 1: a position derived from a shape's box instead of its shape

The six instances, all fixed in one release, all silent while they lived:

  • the point a room's name and area are drawn at;
  • the anchor a circulation route starts from — one plan reported a 10.9 m walk as 5.6 m;
  • the witness lines of automatic dimensions, which floated metres off a sloped facade;
  • swing into <room>, which gave a door a swing into a wall the room does not border;
  • furniture placed against wall, which backed a piece onto a wall outside its own room;
  • and the compass direction a window faces, which on a courtyard plan was reported backwards for every window, because the code found "outward" by comparing against the midpoint of the building's bounding box, and on a courtyard plan that midpoint sits in the courtyard.

Every one of them passed the whole test suite. Every one of them produced plausible output. The linter — the part of the system whose entire job is to say when a plan is unsound — reported none of them, because a wrong label position is not an unsound building, it is a correct building described wrongly.

What actually finds this class: not a test, a grep. Search for consumers of room.size or r.size.w with no nearby branch on the room's polygon. That pattern is a shape consumer reading the box when the ring is the honest datum. The fix is local and closed-form every time — probe one wall thickness off each face and ask which side has floor — and the general rule is the one sentence I wish I had written three releases earlier: a derived position comes from the shape, never from its bounding box or centroid.

Why a rectangle hides it. On a rectangular room the box centre is inside the room, and every one of those six code paths is correct. The entire class is invisible until a room is L-shaped, U-shaped, polygonal or round. If your fixtures are all rectangles, your suite is measuring the easy half of your input space.

Class 2: a generator that reproduces the same wrong text forever

I have a CI gate that regenerates every generated artifact and fails if the committed file differs. Grammars, error-code documentation, CLI reference, JSON schemas, a one-page language spec. It is a good gate. It has been green for a very long time.

It is also, on its own, vacuous — and this is the part that took me longest to see.

The gate compares a generator's output against the committed file. That proves the artifact is reproducible. It says nothing whatsoever about whether the generator is right. A generator that hardcodes a language fact instead of deriving it will reproduce the same wrong text on every run, for every release, with a green check beside it the whole way.

Both halves of that happened here, and both are in the public changelog:

  • the spec generator shipped a CLI reference three versions out of date, and omitted a keyword entirely, for three releases;
  • the grammar generator hardcoded a number regex that did not include the unit suffixes the lexer had accepted for months. The published grammar and the actual parser disagreed about the language, and nothing could see it, because each was internally consistent.

The rule that came out of it: a generator must derive from the source of truth — the same tables the compiler itself reads — and never retype it. And each generator needs a guard that fails when a source-of-truth entry has no rendering at all, because a missing row is exactly what a reflexive gate cannot notice.

The general form is worth stating on its own, because it is not specific to compilers:

A reflexive check is sound for a derived artifact and vacuous for a retyped one, and it cannot tell you which kind it is looking at.

Class 3: a surface no test ever executed

The third class is the cheapest to describe and the most embarrassing to find.

One release found five of them at once. A watch command that had not actually watched for files since it was written, across roughly two dozen releases. A PDF export that drew none of the wall hatching the SVG export drew. A formatter that rewrote pocket doors as hinged ones — silently changing the plan it was asked to tidy.

All five were found by running the surface. None of them by reading the code, and none of them by the test suite, which exercised the functions underneath each surface perfectly well and never once invoked the surface itself.

If you have a CLI, a formatter, a watcher or an exporter that is tested only through its internals, you have this class. The test that would have caught all five is unglamorous: run the actual command, on an actual file, and assert on what comes out.

What I do differently now

Three habits, one per class:

  1. Every new language form ships with a byte-identity law, pinned by a test. A plan that does not use the new feature must render, describe and lint exactly as it did before — proved with a hash sweep across every shipped example, not by looking at two drawings. When a golden moves, that is a finding to explain before it is a diff to bless.
  2. A green gate is asked what it proves. "Reproducible" and "correct" are different words and the gate only ever said the first one. Where a check is reflexive, the second check is a human deriving the artifact from the source of truth by hand, once, and comparing.
  3. Non-rectangular fixtures exist in the suite now. The courtyard plan at the top of this post is in the repository precisely because it is the shape that exposes class 1. A test corpus made only of easy inputs is a measurement of the easy inputs.

None of this makes the system correct. It makes a specific family of silent wrong answers loud, which is a smaller claim and the only one I can support.

Reproduce it

Save the snippet at the top of this post as check.mjs, then:

mkdir bbox-check && cd bbox-check npm init -y && npm install @chanmeng666/archlang@1.57.0 cp -r node_modules/@chanmeng666/archlang/examples . # the example plans ship in the package cp ../check.mjs . && node check.mjs

which prints, against version 1.57.0:

Living 4 pts, bbox centre inside floor? true Entry 4 pts, bbox centre inside floor? true Dining 4 pts, bbox centre inside floor? true Gallery 8 pts, bbox centre inside floor? false Bedroom 1 4 pts, bbox centre inside floor? true Bedroom 2 4 pts, bbox centre inside floor? true Bedroom 3 4 pts, bbox centre inside floor? true Study 4 pts, bbox centre inside floor? true Kitchen 4 pts, bbox centre inside floor? true Bathroom 4 pts, bbox centre inside floor? true Utility 4 pts, bbox centre inside floor? true

The compiler is MIT, has zero runtime dependencies, and compiles floor plans written as text to SVG, DXF and PDF: https://archlang.uk. There is a playground if you would rather not install anything: https://playground.archlang.uk.

If you have shipped one of these three, I would genuinely like to read about it. The first class in particular seems like something everyone with geometry in their system hits exactly once.