Coding Got Faster. Verification Became the Bottleneck

Technical Note

What happens when writing code is no longer the binding constraint?

We had about 25 engineering agents in the fleet and used Claude Code with three Claude Ultra accounts. Agents wrote most of the pull requests in our main repository, working alongside people. An agent could open a draft and GitHub would start running 30,000 tests. The next push could run all 30,000 again. Then the merge could kick off another full run. We were billed for 60 to 75 runner-minutes each time the full suite ran on GitHub-hosted runners.

The three Claude accounts cost us $600 a month. Our October tracker put GitHub Actions at about $1,600. We were spending more than two and a half times as much on GitHub Actions as on the subscriptions we used to write the code.¹

An agent could push a revision and start on another task. The PR stayed put until the tests came back green. In one October read, median wait in the merge queue was 12 to 14 minutes.² That's the time on the clock for the PR. GitHub was billing runner time across jobs that could run in parallel.

Verification had become the binding constraint. The cost moved with it. Before we looked for cheaper runners, we looked at how often we were asking GitHub to run the same tests.

1. Audit job attempts before editing YAML

Start with one repository and one day. With a signed-in GitHub CLI and read access to Actions, list the runs created that day, then inspect all job attempts for an expensive run. Replace the example repository, UTC date, and run ID.

REPO=owner/repo DAY=2026-10-28 gh api --paginate "repos/$REPO/actions/runs?created=$DAY&per_page=100" \ --jq '.workflow_runs[] | [.id, .name, .event, .head_sha, .run_attempt] | @tsv' RUN_ID=123456789 gh api --paginate "repos/$REPO/actions/runs/$RUN_ID/jobs?filter=all&per_page=100" \ --jq '.jobs[] | [.id, .name, .head_sha, (.labels | join(",")), .started_at, .completed_at, .conclusion] | @tsv' 

The runs endpoint gives you the trigger and the run's associated SHA. The jobs endpoint gives you the individual jobs, including earlier attempts with filter=all. For a full audit, count job attempts by when they ran, including retries of runs created before the window. On a busy repository, split the run query into narrower time windows so you don't hit GitHub's 1,000-result cap for filtered searches.

Group job time by trigger and runner class. Eight shards running for ten minutes in parallel account for roughly 80 runner-minutes before billing adjustments. If one disappears near the end and you rerun the entire suite, you can nearly double that compute without making the second run look any slower. Job durations tell you where to look. Use billing data for what you paid, and track the included allowance separately from usage.

2. Give drafts a fast check and keep the full merge gate

We moved to GitHub Enterprise for a larger included-minutes allowance and split testing into two tiers. Draft pull requests ran the tests they added or changed. Pull requests ready to merge still had to pass the full suite. Billed minutes per run fell from 76.4 to 31.4, a 59% drop. Total billed minutes barely moved at first. We were running CI more often.³

The merge queue gave us a way to avoid a second full run on code that had already passed. The queue tests the commit intended to land. After merge, CI looks for a successful queue run for the exact commit on main. If it can't find one, it runs the full suite. That last check matters: a green result for a different commit doesn't count.

If you're changing your own workflows, start with the event. A draft push and a commit about to land on main don't need the same check.

Wire the required workflow to merge_group. In the full test job, record git rev-parse HEAD after checkout so you can tell what actually ran. For the post-merge shortcut, require a green run from the trusted queue workflow, the tested SHA matching the pushed SHA, and a passing result for every required full-suite job. A workflow that checked out a different ref or skipped required jobs doesn't count. Neither does a lookup that times out. Before enabling the shortcut, try a known green exact match, a green run on a different SHA, and a failed lookup. Only the first should skip the suite.

We're still working through the slow parts of the suite. For the tests we had to run either way, we looked at what we paid for runner time.

3. Move one workflow to Spot and test both failure paths

GitHub Actions still ran the workflow. We moved eligible test jobs off GitHub-hosted runners and onto EC2 Spot, starting with the merge queue and later moving jobs for PRs from branches in the same repository. We used isolated, short-lived CI runners in AWS, with one job per runner and the runner discarded after the job. Spot gave us a cheaper way to buy test time. It could also take that time away in the middle of a run. If seven shards pass and the eighth loses its runner, rerunning the whole suite means paying to repeat seven results you already have.

We called the launcher Spot Wave-Rider. Instead of asking AWS for a particular machine, we told EC2 Fleet what the job needed and let it choose among matching instance types in the zone specified by the launcher. We used the capacity-optimized strategy, which looks for pools with spare capacity rather than just picking the cheapest one. If Spot capacity is unavailable at launch, the launcher can try On-Demand. A test shard lost to an interruption is retried once on a GitHub-hosted runner. A failed test isn't eligible for that retry.

Those are two different failure paths. At launch, the job hasn't started and needs somewhere to run. Mid-test, you want to preserve the shards that already passed. In the eight-shard example, retry shard eight against the same tested commit and require its result along with the original seven. If you can't establish that a runner was lost, or if the retry fails, the required check stays red. Test the interruption path before moving a required workflow: stop a runner during a pilot job and check that only the missing shard is retried. Then make a test fail and check that the gate stays red. Keep untrusted PR code off your self-hosted pool; GitHub's runner security guidance explains why a cheap runner can be a bad trade.

4. Check the bill and the queue together

Run the pilot against a recent baseline with the same repository scope. Change one lever at a time where you can. Watch first-pass queue success and time in queue as well as merge count. For cost per merged PR, add what you actually pay GitHub and AWS for that scope, including compute spent on retries, and divide by the number of merged PRs. A lower GitHub bill can hide a longer wait or a new AWS bill.

What we measured

GitHub's daily charges before credits went from about $83 to $27. That's a 68% drop, and we were still merging about 50 PRs a day. I still need to match up the days on the bill with the jobs we moved and see what AWS charged us to run them. Then I can say how much the move actually saved us.

We started with agents that could write code faster than we could verify it. We gave drafts a quick check and kept the full suite at the merge gate. If the exact commit had already passed, we stopped running the suite again after it landed. Then we moved some of the remaining work to Spot and built a way to retry a lost shard without starting over. I still want the agents to push as many revisions as they need. What I care about is how long it takes to get a tested change onto main and what it costs us to get it there.

Once coding became fast, the binding constraint moved to verification and testing. We cut repeat test runs, kept the full suite at the merge gate, and moved some of the remaining work to Spot with a way to retry just the shard we lost. After this, the binding constraint on our speed, velocity, and productivity is the human judgement required to recognize the correct architectural patterns and make the right calls.

1. The $600 is our monthly Claude subscription cost. The rounded October Actions figure comes from our internal spend tracker and is still being reconciled against a separate GitHub billing read. The comparison covers those two expenses, not the full cost of engineering or verification. ↩

2. Source: internal merge-queue tracking, October 17, 2026. This is a snapshot of queue wait, not total PR lead time or a measurement of how long agents spent writing code. ↩

3. Source: internal CI usage tracking for October 16, 2026. The 59% figure compares billed minutes per run, not total daily minutes. ↩

4. Sources: GitHub Actions usage billing API (Actions minutes, gross charges before included allowances) and GitHub merged pull request search, both read November 6, 2026. The October cost-per-merge range is from the daily rows for October 28-31. The ratio divides gross Actions charges for the billing scope by the observed repository's merge count for the same day. It does not assign a charge to an individual PR. Merge counts do not measure queue latency or test quality. ↩