"Just re-run it."
Four words that quietly destroyed the trust of an entire engineering team.

If you've ever typed those words into a Slack thread, this story is for you.

A Red Build Nobody Believed

It was 4:47 PM on a Thursday. Priya, a backend engineer at a 12-person B2B SaaS startup, pushed a one-line fix for a typo in an email template. The pipeline went red.

She didn't read the logs. She didn't even open the failed job. She clicked Re-run all jobs, grabbed a coffee, and came back to a green build.

Nothing in the code had changed. The build had simply decided to pass the second time.

That moment, repeated dozens of times a week, is what we'll dissect in this case study. By the end of this post you will have:

• A clear definition of what a flaky test actually costs you (in hours, not vibes)
• A repeatable way to measure flakiness in your own pipeline
• A practical migration path from brittle scripted tests to a more resilient approach
• Code you can paste into your own repo today

The team's product was a subscription management dashboard. Their stack looked like most modern startups:

On paper, this was a healthy setup. In practice, the end-to-end suite had become the thing everybody worked around instead of relying on.

The Problem: Quantifying the Pain

The team's engineering manager finally did what every engineering manager should do before complaining: she measured.

She pulled 30 days of CI history and wrote a small script to classify every failed run by whether a re-run of the same commit later passed.

#!/usr/bin/env bash # flaky-report.sh # Counts runs that failed, then passed on retry for the same commit SHA. gh run list --limit 500 --json databaseId,headSha,conclusion,attempt \ | jq -r ' group_by(.headSha)[] | select(any(.[]; .conclusion=="failure") and any(.[]; .conclusion=="success")) | .[0].headSha ' | sort -u | wc -l 

The result was uncomfortable:

Total pipeline runs (30 days): 412 Runs that failed at least once: 171 (41.5%) Failed-then-passed on same SHA: 139 (81% of failures) Real failures (code was broken): 32 

Eighty-one percent of the failures were not bugs. They were noise.

The real cost

She then did the math that got leadership's attention:

# cost_of_flakiness.py flaky_failures_per_month = 139 avg_minutes_lost_per_failure = 22 # context switch + rerun + wait engineers_blocked_ratio = 1.4 # others waiting on the same pipeline hourly_cost = 55 # blended, USD hours_lost = (flaky_failures_per_month * avg_minutes_lost_per_failure * engineers_blocked_ratio) / 60 print(f"Hours lost per month: {hours_lost:.1f}") print(f"Approx. monthly cost: ${hours_lost * hourly_cost:,.0f}") 

Output:

Hours lost per month: 71.3 Approx. monthly cost: $3,923 

That is nearly half of one engineer's monthly capacity, burned on re-running tests that were not even telling the truth. And the invisible cost was worse: engineers had started ignoring red builds. Eventually, a real bug slipped through because everyone assumed "it's probably flaky again."

What They Tried First (And Why It Didn't Work)

Before looking for a new approach, the team tried the usual suspects.

Attempt 1: Add retries. They set retries: 2 in the Playwright config.

// playwright.config.ts import { defineConfig } from '@playwright/test'; export default defineConfig({ retries: process.env.CI ? 2 : 0, timeout: 30_000, }); 

Result: the pipeline went green more often, but run time increased by 35%, and the flakiness was now hidden rather than fixed.

Attempt 2: Add hard waits. Somebody sprinkled await page.waitForTimeout(3000) everywhere.

Result: slower tests, same flakiness, and a codebase full of magic numbers.

Attempt 3: A "flaky test quarantine." They tagged unstable tests and skipped them in CI.

Result: the quarantine list grew from 9 tests to 31 in six weeks. A quarantine that only grows is just a graveyard.

Diagnosing the Root Cause

The team finally sorted the 139 flaky failures by cause. This is the table that changed their thinking:

Nearly half of the pain came from tests that were coupled to implementation details of the UI. A selector like this one was typical:

// Looks innocent. Breaks the moment a designer touches the DOM. await page.click('#root > div.layout > main > section:nth-child(2) button.btn-primary'); 

A developer changes a wrapper div, nothing visible changes for the user, and the test dies.

The Turning Point

The insight was simple but uncomfortable: the tests were asserting on how the page was built, not on what a user could do.

The team started evaluating tools that describe tests in terms of user intent and use AI to locate elements, instead of hard-coding locators. That is where they tried an AI-driven QA platform. If you want to try the same approach on your own flaky suite, you can start with BotGauge here and run your first AI-generated tests before committing to any large migration.

The workflow they adopted looked like this:

1. Describe the scenario in plain English instead of writing selectors.
2. Let the platform generate and execute the test across the UI, API, and database layers.
3. Rely on self-healing so that cosmetic UI changes don't produce false failures.
4. Wire it into CI so results gate the merge just like unit tests do.

A scenario that used to be 40 lines of Playwright became something like this:

Scenario: Customer upgrades from Basic to Pro plan Given I am logged in as a Basic plan customer When I open the Billing page And I choose the Pro plan and confirm payment Then I should see a confirmation message And my plan should show as "Pro" in Account Settings 

No nth-child. No waitForTimeout. The test reads like the requirement it protects.

The Migration Playbook

They did not rewrite everything at once. Here is the order that worked, and it's a good template for you:

Week 1 Measure flakiness (the scripts above). Rank tests by failure count. Week 2 Migrate the 10 flakiest critical-path tests (login, checkout, upgrade). Week 3 Run old and new suites in parallel. Compare signal, not just speed. Week 4 Retire migrated Playwright tests. Keep the rest until their turn. Week 5+ Continue by risk tier. Delete the quarantine list as you go. 

Wire the new suite into the pipeline like any other quality gate:

# .github/workflows/ci.yml name: CI on: [pull_request] jobs: unit: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - run: npm ci && npm test e2e: needs: unit runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Trigger AI end-to-end suite run: ./scripts/run-e2e-suite.sh env: E2E_API_TOKEN: ${{ secrets.E2E_API_TOKEN }} 

The Results (After 8 Weeks)

Illustrative numbers from the scenario, tracked using the same scripts from earlier:

The most important number is "Failures that were real bugs." When a build turns red now, people read the logs. That is what a healthy signal looks like.

Key Lessons You Can Steal

1. Measure before you migrate. Without the 30-day report, this would have stayed an anecdote.
2. Retries hide problems, they don't solve them. Use them as a temporary bandage and track them as debt.
3. A growing quarantine is a smell. Set a hard cap and an expiry date for every quarantined test.
4. Test intent, not implementation. If a designer's CSS tweak can break your test, your test is testing the wrong thing.
5. Trust is the real product of a test suite. Speed and coverage matter, but only if people believe the result.

Your Turn

Run the flaky-report.sh script against your own repository this week. If more than 15% of your failures disappear on re-run, you don't have a testing problem, you have a trust problem.

Have a flaky-test horror story of your own? Drop it in the comments. I read every one.