DevOps
The shell step that could not fail no matter what we did to it
Sergey Shinder DEV Community
23 views
Our integration suite had been green for eleven weeks. That is not a boast, it is the symptom. A colleague onboarding to the repo wrote a deliberately broken test to see what a failure looked like, pushed it, and the pipeline went green in the usual four minutes.
The stage was a single shell step, npm run test:integration | tee integration.log || true, and it contains two separate mistakes, both added by people trying to be helpful. The pipe was there so the log could be uploaded as an artifact, and without set -o pipefail the exit status of the pipeline is the exit status of tee, which is always zero. The || true had been added fourteen months earlier because a cleanup command at the end of the script occasionally returned 1 and nobody wanted to debug it that week. Either one alone would have hidden every failure. We had both.
I went looking for the rest and found seven more steps across four repositories that could not report failure: three with the same pipe, two with || true, one that ran the real command inside a subshell whose status was discarded, and one where a for loop over services swallowed each iteration's result and exited on the status of the loop itself.
The fixes were mechanical. Every script step now starts set -euo pipefail, enforced by a lint job that reads the workflow YAML and fails on any run: block that does not. || true is banned outside a small allowlist, and each allowed use carries the reason in a comment on the same line. Logs are captured by redirecting to a file and cating it afterwards, not by piping the command that matters.
The part I would keep if I could only keep one thing is the canary. There is now a job in every pipeline that runs a command which exits 1, and the pipeline asserts that this job is marked failed. It runs on every build and costs two seconds. It would have caught this in an afternoon rather than in eleven weeks.
A test suite you have never watched go red is not evidence about your code. It is evidence that a script exited zero.
– Sergey Shinder
Read original: https://dev.to/sergey_shinder_ab2d943365/the-shell-step-that-could-not-fail-no-matter-what-we-did-to-it-f45
← Previous
prefill and default look the same in Console. Only one of them lets an agent skip your required field.
Next →
Two Sources of Truth Will Always Disagree
Related
Upgrading a Locked-Down Linux Fleet with Nothing but kubectl
DevOps
53
Dev.to (EN Zone)
PasteSwitch: 108 crypto-theft ads on a hijacked HBO Max Reddit account
DevOps
39
DEV Community
Code Hosting vs CI/CD: GitHub Actions, GitLab CI, Jenkins, CircleCI, Argo Workflows and More — What Actually Fits Your Stack?
DevOps
40
Dev.to (EN Zone)
My build finished in 0 seconds and thirteen test suites passed. They had run yesterday's binary.
DevOps
37
DEV Community
Comments0
No comments yet — be the first