TL;DR: A pipeline that builds and deploys automatically but only runs a thin smoke test, or tests inconsistently across stages, isn't really "CI/CD," it's continuous deployment with continuous integration's name borrowed. The practice that actually earns the name, testing automatically and comprehensively at every stage, is also the one quietest to talk about, because "we added more tests" doesn't demo as well as GitOps or progressive delivery. A Dubai proptech team's change-failure rate dropped from 25% to 5% specifically because this gap got closed, not from any deployment-mechanics change.
Ask a team with a mature-looking pipeline whether they have "good CI/CD," and most say yes, pointing at automated builds and automated deploys. Ask a more specific question, does every change get tested at the unit, integration, and end-to-end level before it ships, automatically, every time, and the answer gets noticeably less confident. That gap, build-and-deploy automation without comprehensive test automation, is common enough that it's worth treating as its own category of pipeline maturity problem, separate from the GitOps and progressive-delivery conversations that get more attention.
What "automate the whole pipeline" actually requires
| Stage | What should be automated | What commonly gets skipped |
|---|---|---|
| Build | Compile/package on every commit | Rarely skipped, this part is usually solid |
| Unit test | Fast, isolated tests on every commit | Partial coverage, "we'll add more tests later" |
| Integration test | Tests across service boundaries | Often missing entirely, requires more setup than unit tests |
| End-to-end test | Full user-flow validation | Frequently manual, or run only before major releases |
| Deploy | Push to production automatically | Usually solid once the above exists |
The pattern: automation effort concentrates at the build and deploy ends of the pipeline, because they're the most visible and the most straightforward to wire up. The testing middle, especially integration and end-to-end, requires more setup work and more ongoing maintenance as the system changes, which is exactly why it's the layer most likely to stay thin.
Why this specific gap is expensive
A pipeline with solid build/deploy automation but thin testing ships fast and unreliably, which is a worse combination than shipping slowly and reliably, because the speed just gets you to a broken state faster. This is the mechanism behind a counterintuitive finding in delivery research: teams that ship most often also tend to have the lowest change-failure rates, which only makes sense once you notice that the teams shipping safely at high frequency invested specifically in the testing layer, not just the deployment mechanics. Deployment speed without test coverage doesn't produce that pattern, it produces fast, frequent breakage instead.
The four numbers, and which one this practice actually moves
| Metric | What it measures | What drives it |
|---|---|---|
| Deployment frequency | How often you ship | Pipeline automation, deploy mechanics |
| Lead time for changes | Commit to production | Pipeline automation, approval overhead |
| Change failure rate | Share of deploys causing a problem | Test coverage and quality, specifically |
| Time to restore service | Recovery speed | Rollback mechanics, observability |
Change failure rate is the metric most directly downstream of comprehensive automated testing. A team can have excellent deployment frequency and lead time (fast build/deploy automation) while still carrying a high change-failure rate, if the thing shipping fast isn't being tested thoroughly on the way out.
Case study: the number that moved once testing actually got comprehensive
A Dubai proptech company's releases had become a monthly ordeal, bundling weeks of changes, taking a full evening of manual work, with about one in four causing a production problem. Five-month rebuild:
| Problem | What was built | Outcome |
|---|---|---|
| Manual monthly releases | Fully automated build/test/deploy pipeline | Deploys: monthly → daily |
| Slow path to production | Trunk-based workflow, automated tests at every stage | Lead time: 8 days → 90 minutes |
| Frequent broken releases | Comprehensive automated testing: unit, integration, end-to-end, run on every change | Change-failure rate: 25% → 5% |
| Slow incident recovery | Monitoring tied to the pipeline, GitOps rollback | Recovery time: 5 hours → 12 minutes |
All four rows improved, but the change-failure-rate row is the one that traces most directly back to test automation specifically, rather than to deployment mechanics. Going from one-in-four releases causing a problem to one-in-twenty is a testing-coverage story first, the deploy automation made releases possible more often; the test automation is what made each of those releases safe.
FAQ
How do we tell if our pipeline has this gap without a full audit?
Ask what percentage of changes get integration and end-to-end test coverage, not just unit tests. If the honest answer is "most of our testing is unit tests plus a manual QA pass before big releases," that's the gap.
Is it better to fix testing gaps before or after adopting GitOps and progressive delivery?
Before, or at least alongside. GitOps and progressive delivery reduce the blast radius and recovery time of a bad release, they don't reduce how often a release is bad in the first place, that's specifically what comprehensive testing does.
Does more test coverage necessarily mean slower pipelines?
Not if tests are parallelized and layered correctly, fast unit tests run on every commit, slower integration/end-to-end tests run less frequently or in parallel stages. The proptech case study's lead time dropped to 90 minutes with full test coverage in place, not despite it.
Originally published on the Sherdil Cloud blog, the full piece (with the complete six-practice breakdown and four DORA metrics in depth) is here. For the hands-on pipeline build, see CI/CD pipeline from scratch; for the cloud foundation underneath it, DevOps in the cloud.
About the author: Muhammad Usman is Head of DevOps at Sherdil Cloud, AWS DevOps Engineer Professional, Certified Kubernetes Administrator (CKA), and Alibaba Cloud Certified, building cloud and DevOps infrastructure for enterprises across Pakistan, the UAE, and the United States since 2014.
