The release gate said "two OTel backends proven." The check proving it ended in || true.

It had been passing for weeks. Against a service that was never started, never configured, and never received a single span.

This is from the agentwatch v0.2.1 release — the agent-security tool in agentic-suite. The full forensics are in the v0.2.0 field test report.

The claim and the check

agentwatch emits OpenTelemetry GenAI spans. The release gate for v0.2.0 has an expanded item, number 10:

Two OTel backends proven — an agent-span tree must render in Jaeger and Tempo, not just whichever backend happened to be wired up first.

The field-test case behind it, FT-BACKEND-2, checked the Tempo side. Its assertion was, in essence:

curl -sf http://localhost:3200/api/search/tags || true

Port 3200 is Tempo's search API. The curl asks "is Tempo up and does it answer?" And then || true answers the question permanently: yes, whatever curl said.

That's a check that cannot fail. It's also a check someone wrote with good intentions — the || true is there so the case doesn't hard-fail on a flaky timing window. It's defensive shell. And it converted a release-gate claim into a string in a script.

What was actually running

The stack definition (lib.sh) listed the v0.2.0 profile services. Tempo wasn't among them. The collector config (deploy/otel-collector-config.yml) had one exporter: otlp/jaeger.

So the topology behind the "two backends" claim was:

Claimed Actually
Jaeger receiving spans ✅ up, receiving
Tempo receiving spans ❌ never started, never configured, no exporter pointing at it
Field-test status PASS

The check passed because the check was unpassable. Not because the system worked.

The part I like: hardening made it fail

Here's where the story turns. The v0.2.0 field test had 94 cases, and after the run I did a hardening pass — going case by case, asking of each assertion: could this fail if the claim were false?

FT-BACKEND-2 couldn't. So the || true came out, and the check became a real query: Tempo's search API must answer for the service agentwatch emits as.

Then the case ran red. And the red was not a test bug. The red was the finding.

That inversion took me a while to appreciate. We're trained to treat a red field test as "the test is broken until proven otherwise." But a check that can fail, failing, is the system telling you something a green check never will. The report we published says it plainly: the old || true hid that the R4 claim was unmet. The green wasn't evidence. It was upholstery.

The fix was three lines of infrastructure

Not a test fix — an actual product fix:

  1. tempo added to the profile services (and a tempo-config.yml to match).
  2. A new otlp/tempo exporter in the collector config, added to the fan-out list.
  3. The sibling case (FT-OTEL-1) hardened too — it now asserts a real parent/child agent-span tree renders, a CHILD_OF relationship, in Jaeger and Tempo, rather than grepping a service name out of one backend's API.

Re-run: the stack comes up 16 services including svc-tempo, the trace resolves in Tempo, and both cases pass — now for reasons. The v0.2.1 release gate table reads gate 10 as PASS (fixed) with the Tempo receipt attached.

The class of bug, generalized

|| true is the obvious member. But the same genus showed up all over the census we did of the 94 field-test assertions:

Shape Looks like Can fail when claim is false?
`\ \ true`
test -f <path> "artifact exists" Only if packaging breaks entirely
python -c "import x" "module ships" Only if the import system itself fails
grep -q <word> on your own output "feature works" Only if the word stops appearing
Driver runs, prints ok: ... "scenario passed" Only if the driver exits non-zero

We tallied it: across 94 cases, 52 were driver-run, 11 existence/syntax, 9 substring-greps, 5 module-imports, 1 outright || true — and only about 8 asserted a real condition. That's the sentence I'd want on a wall somewhere: most of your green is plumbing, and plumbing can't fail.

The v0.2.0 campaign grounded 90 of 92 passing cases on checks that can fail. That's the actual product of that release, more than any feature.

What I keep from this

  • A check that cannot fail is not a check. It's documentation with an exit code.
  • || true on a probe is fine (you don't want probes failing your build). || true on a gate is a lie with good intentions.
  • When a hardened assertion goes red, the first hypothesis should shift over time: early in a project it's "test wrong," late in a project it's "system wrong." This one was late.
  • Release-gate claims need an assertion per claim, not an assertion per area. "Backends" isn't a claim. "Jaeger and Tempo each render the tree" is.

All of it shipped in agentwatch v0.2.1 — on PyPI.

The question worth asking of your own release gates: for each one, what's the check, and could that check fail tomorrow if the claim stopped being true? If any answer is "no," you've found the next field test.

References