Eighteen domains, each with its first measured point and its last; the worst started at a quarter, one stays under a hundred.

👋 Hi, I'm Anton - a software engineer. These days I work on a platform: contracts, generated code, data paths and the tooling around them. Running notes live on my GitHub: github.com/brilliant-almazov.

Every number below is the acceptance agent's own count on local test stands, first point and last; the fixes for the open rows in the one domain under a hundred are planned in other services and are marked that way.

This is part 3 of the series on one week of integration acceptance across a whole chain of services, after months of doing it the slow way. Maybe you already do it better, maybe you'd do it another way - this is how it went, and where it stops.

In real life

The whole system is a zoo. The services are the residents, the bus is the paths, and acceptance is the vet - Pickle, a guinea pig with a card for each of eighteen enclosures. Nora is the keeper and hangs signs.

At the end of the week Pickle hung a tag on every enclosure, and each tag had two numbers on it: before and after. Nora asked why keep the before at all - surely the after is what matters. Pickle said: "So we remember where we started." Biscuit, the courier dog, read his own tag and said, with some pride: "I started at a quarter."

Pip counted seventeen green tags. Miso the cat pointed out that hers had been green from day one, which is true and also very like her. And then there was one tag that was not green. Pickle looked at it and said "four lines still open". Ember the dragon, worried, said the only thing worth saying: "Do not paint it green."

A four-panel comic: Pickle says before, after, both numbers, next to a sign reading before 25, after 100, and Nora asks why keep the before; Pickle says so we remember where we started, and Biscuit says I started at a quarter; Pip says seventeen green tags, and Miso says mine was green from day one; Pickle, worried, says four lines still open, and Ember, worried, says do not paint it green

The tags are the per-domain results of the acceptance plan. The "before" is the first point the plan measured for that domain; the "after" is the last day of the week. And the one amber tag is the subject of the second half of this article: a domain at 96.4 %, with its four open rows named instead of hidden.

Why keep the before

A plan that only shows the current number looks the same whether a domain was always fine or was rescued from a quarter. Those are different stories, with different risks. A domain that started low and reached a hundred in days was changed a lot in days; it deserves a second look in the next regression run. A domain that was at a hundred from the start told us little this week.

So the plan keeps, for every domain, the first measured point - the date and the count - next to the current one. Nothing is overwritten. The figures below show all eighteen, split in two halves by where they started.

Paired columns on a 0 to 100 scale for the nine domains that started lowest: corner cases 24.8 to 100, batch frequency 33.3 to 100, watchdog 36.4 to 100, send switch and hold 46.2 to 100, collector statuses 53 to 100, storage with three binaries 56.8 to 100, task queue collector 59.4 to 100, collector invariants 60.3 to 100, priorities and buffer 71.4 to 100; a pill reads measured, rows passing, percent of rows in scope

Paired columns for the nine domains that started highest: provider emulator 77 to 100, gathering on the emulator 83.3 to 96.4, request log and object store 83.3 to 100, priorities and manual push 91.7 to 100, final processing 96 to 100, webhook of provider B 96.8 to 100, content fetch, freeze and scheduled frequency at 100 throughout; pills read measured and one domain under 100, 96.4 percent

The eighteen, in numbers

Here are all eighteen as the plan recorded them: rows passing out of rows in scope, at the first measured point and on the last day.

Domain First point Last day
collector corner cases 29 of 117 · 24.8 % 61 of 61 · 100 %
batch frequency 2 of 6 · 33.3 % 13 of 13 · 100 %
watchdog 4 of 11 · 36.4 % 11 of 11 · 100 %
send switch and hold 12 of 26 · 46.2 % 24 of 24 · 100 %
collector statuses 53 of 100 · 53.0 % 82 of 82 · 100 %
storage, three binaries 42 of 74 · 56.8 % 74 of 74 · 100 %
task, queue, collector 19 of 32 · 59.4 % 31 of 31 · 100 %
collector invariants 35 of 58 · 60.3 % 38 of 38 · 100 %
priorities and buffer 15 of 21 · 71.4 % 95 of 95 · 100 %
provider emulator 47 of 61 · 77.0 % 59 of 59 · 100 %
gathering on the emulator 95 of 114 · 83.3 % 107 of 111 · 96.4 %
request log and object store 20 of 24 · 83.3 % 23 of 23 · 100 %
priorities and manual push 11 of 12 · 91.7 % 12 of 12 · 100 %
final processing 24 of 25 · 96.0 % 40 of 40 · 100 %
the webhook of provider B 30 of 31 · 96.8 % 30 of 30 · 100 %
content fetch 32 of 32 · 100 % 100 %, unchanged
freeze 10 of 10 · 100 % unchanged
scheduled frequency 7 of 7 · 100 % unchanged

Read the second column of counts carefully, because the bottom of each fraction moved too. Collector corner cases went from 117 rows in scope to 61. Priorities and buffer went the other way, from 21 rows to 95. The plan's own note on the whole week says the denominators changed because synthetic rows left the scope by the owner's word, each row with its quote; it does not give the reason domain by domain, and neither do I. A percentage on its own hides both moves; the counts show them.

A donut: 17 of 18 domains; legend: at 100 percent 17, at 96.4 percent 1; Pip says seventeen green tags; a pill reads measured

How a tag goes from amber to green

Every domain that moved, moved the same way. The first point is measured and kept. A fix lands - in the collector, in the configuration of the stand, sometimes in the acceptance agent's own probes. The fix reaches the stand at once, because every pushed commit goes there without waiting for CI. The rows of that domain are rerun on the live build. And only then does the tag change.

An animation: a letter travels from first point to a fix, to a rerun, to the tag; the pills change from the first point is measured and kept, to a fix reaches the stand, to the rows are rerun on the live build, to the tag shows both numbers

The rule that makes this honest is the one from the first part of this series: a verdict counts only on the build that carries the fix. One run earlier in the week was measured on a build without the fix under test and showed a false loss of four rows. After that, every live build got a new run, and over a week of such reruns twenty more rows of the plan passed. The next part goes through what those fixes were.

The one that is not green

Gathering on the emulator started at 83.3 % - 95 of 114 rows - and finished the week at 96.4 %, 107 of 111. That is the only domain under a hundred, and the four rows that keep it there are the four open rows of the whole plan: 749 of 753.

Three of the four are tied to defects the acceptance agent found in services other than the collector, and which are planned to be fixed there:

# the open rows of gathering on the emulator (paraphrased) 2 rows results are delivered to a queue that has no consumer, so delivered results stay unread -> planned, another service 1 row a status call with tenant 0 is refused ("tenant_id must be > 0"), so the empty-result mark is lost -> planned, another service 1 row not named in this article rule a row stays failing until it passes on the live build

A dark card with the tag of the one enclosure that is not green: domain gathering on the emulator; first point 95 of 114 rows, 83.3 percent; now 107 of 111 rows, 96.4 percent; open 4 rows, each with a reason: 2 rows a result queue with no consumer, another service, planned; 1 row a status call with tenant 0 refused, another service, planned; 1 row not named on this card; the highlighted line reads the tag stays amber until the rows pass on the live build; pills read measured and open rows planned

Why not a hundred, then? Because the alternative to 96.4 % here is not 100 %, it is a lie. These rows describe behaviour the chain should have; the chain does not have it yet; the fix belongs to a service the acceptance agent does not own. Taking the rows out of scope would turn the tag green and make the plan worse at the one thing it is for. The rule from part 1 covers it: a row leaves the scope only with the owner's words that name it, and nobody said those words about these four.

Eighteen tiles with faces in three rows of six: corner cases, batch frequency, watchdog, send switch, statuses, storage; task and queue, invariants, priorities and buffer, emulator, gathering in amber and worried, request log; priorities and push, final step, webhook B, content, freeze, schedule, all proud; pills read measured 17 at 100 percent and gathering on the emulator 96.4 percent

The sign on the door

Nora, proud, waving next to a big sign reading a tag shows both numbers and stays amber; Nora says nobody paints a tag green; on the right the mechanism: the first point is kept, rerun on the live build, open rows named, not dropped; pills read measured and 4 open rows planned

What went wrong

The worst starting point was a quarter: collector corner cases at 29 of 117. By the last day the same domain had 61 rows in scope, all passing - the scope moved as well as the code, and the table above keeps both. Collector statuses, invariants and the task-to-collector path all started near a half or just above. A week earlier none of these numbers existed; the chain was "mostly working", which is the sentence the whole series is about replacing.

What I do not claim

  • Not that the eighteen domains are the same size or the same weight. They range from six rows to over a hundred.
  • Not that a domain at 100 % is finished. It passes every row in its scope today, on local stands.
  • Not that the shrinking denominators were free. Each row that left the scope carries the owner's words; I do not argue those choices here.
  • Not when the four open rows close. Their fixes are planned in other services.

What of this is method

  • Keep the before. A first point next to the last tells how much a domain was changed, not just where it is.
  • Show counts, not only shares. A moving denominator is invisible in a percentage.
  • An amber tag stays amber. Open rows are named and owned, not removed.

Eighteen domains in a week - Part 3. Next: Whose Bug Was It.

If your test reports show only today's percentage, how do you see which area was rescued last week? If your plan's denominator moves, who signs off on each row that leaves it? If you have an area stuck just under a hundred, is it waiting on another team, and does the report say so? What does your one amber tag say?