Coding agents can open more PRs in a day than your team can review in a week. An AI software factory is supposed to turn all that code into features your team can ship. In practice, this moves the bottleneck from writing code to checking it.
Below, we will discuss all eight building blocks, the tools that support them, and what happens when one is missing.
TL;DR: An AI software factory consists of multiple coding agents working in parallel loops, each fed by a queue of specs. While code generation scales with compute, verification doesn’t. As a result, the amount of autonomy you can safely give your agents depends on how well you can verify the running app. Unit tests and AI review bots check the code, but something independent still needs to check the product from the user's perspective.
What Is an AI Software Factory?
An AI software factory is a system of coding agents running in parallel loops, fed by a queue of specs and tickets. Every change passes through automated checks and a review gate before it reaches production.
The definition most people use comes from Addy Osmani. It revolves around these three ideas:
- A loop is one agent doing one job on repeat: gathering context, making a change, checking the result, and starting again.
- A harness is everything around that loop, such as the sandbox it runs in, the tools it can reach, its memory, and the rules for what "done" means.
- A factory is many harnessed loops running at once, with humans overseeing the whole system.
How much human involvement remains largely depends on your level of autonomy. Dan Shapiro's Five Levels, published in January 2026, provide a shared scale. The names and the right-hand column are my own summary of his descriptions:
| Level | Name | Who reads the code |
|---|---|---|
| 0 to 2 | Spicy autocomplete to pair programming | A human writes or approves the code |
| 3 | Developer as manager | A human reviews the agent's diffs |
| 4 | Developer as product manager | A human writes the spec and checks that tests pass |
| 5 | Dark factory | Nobody reads the code |
Osmani adds a useful split. In a light factory, humans stay involved in design and review. In a dark factory, code ships without any human reading it. Only machines can verify the result.
What Are the Building Blocks of an AI Software Factory?
Below, you’ll find the full set, with what each block does and what happens when you leave it out:
| Building block | What it does | Example tools | What breaks without it |
|---|---|---|---|
| Work queue and specs | Turns intent into tasks an agent can build | Linear, Jira, GitHub Spec Kit | Agents build the wrong thing quickly |
| Coding agents and harness | Writes the change within defined limits | Claude Code, Codex, Cursor | Loops wander, stall, or edit what they shouldn’t |
| Shared context | Rules files and Model Context Protocol (MCP) servers |
| The same mistakes repeat on every PR |
| Preview deployments | Gives every PR its own running build | Vercel, Netlify | There’s nothing real to verify before the merge |
| Code-level checks | Unit tests, continuous integration (CI), static analysis, AI review bots | GitHub Actions, CodeRabbit | Broken code gets merged |
| Running-app verification | Checks user journeys in the running app, through a browser or a mobile build | QA.tech, scripted end-to-end suites | Correct code ships a broken product |
| Risk-tiered review gate | Sends each change through the least expensive review that’s still safe | Branch protection, merge queues | Reviewers get overwhelmed or rubber-stamp changes |
| Rollback and production signals | Undoes bad changes and feeds incidents back into the queue | Feature flags, observability tools | Escaped bugs pile up unnoticed |
What Does Running-App Verification Look Like in Practice?
Say a coding agent ships a redesigned checkout flow. Unit tests pass, the review bot leaves two style comments, and CI is green. But on the preview deployment, the new layout pushes the saved-card selector below a sticky banner, preventing returning customers from selecting a card. Nothing in the diff looks wrong, but once the change has been merged, every returning customer faces the same problem.
This is the gap QA.tech, an AI testing platform, is built to address. Its QA agents receive a user goal ("pay with a saved card and see the confirmation") and interact with the live app much like a real user would. The test case is written as a goal, so a renamed selector doesn't automatically break it. A web or mobile test case can also check email and SMS or make API calls along the way. That means the checkout test can verify that the receipt email actually arrives.

In a factory, verification occurs at two points:
- Before a PR exists. The coding agent calls the MCP server from inside Claude Code, Cursor, or Codex. It starts a run, reads the result, and fixes the failure before anyone opens a PR. The MCP docs list the available tools, such as
start_run_with_test_casesandget_run_test_case_trace. - On the PR. PR testing runs against the PR's preview deployment, or, if you have no previews, against staging straight after the merge. It selects the existing test cases the change affects and creates new ones only where coverage is missing. It then posts the result on the PR: pass, fail, or "unable to verify", with a status check you can require before the merge.

When Does a Dark Software Factory Not Make Sense?
For most teams, a fully dark factory is not the right goal. Osmani warns about comprehension debt, which he defines as "the widening gap between how much code exists and how much any human still understands." A dark factory adds to that debt as fast as it produces code, even while the tests remain green. That risk is particularly high in large, decade-old systems, which is where a lot of mid-market and enterprise teams work.
A light factory is often the better choice when:
- The domain is regulated, and a human must sign off on the intent.
- The codebase is old and poorly understood, so humans need to keep reading in order to continue learning.
- Usability and product judgment matter more than raw throughput.
In every case, QA agents take over the repetitive verification while people remain responsible for judgment calls. As for the test strategy, I recommend a small, pinned regression set for your critical paths, with dynamic testing on every PR and exploratory testing for everything else. Rebuilding your old suite one to one only carries its existing brittleness into a new tool.
Where Should You Start in 2027?
Build the blocks in the order of what each one unlocks:
- Give every PR a preview deployment, so there’s something real to verify before merging. If that isn’t possible yet, run verification on staging straight after the merge.
- Verify the running app on every PR using a verifier that didn’t write the change.
- Add risk tiers to your review gate, so low-risk changes with clean evidence can merge based on an automated check alone, while high-risk changes go to a human reviewer who starts from the available evidence.
- Raise agent autonomy only as fast as your verification can keep up.
If you’d rather not build and maintain step two yourself, the PR testing and MCP setup described above can handle it. For a closer look at where verification fits in the factory, see what an AI software factory is and where its quality-control station goes.
FAQ
What is an AI software factory?
An AI software factory is a system of coding agents running in parallel loops, fed by a queue of work. Every change passes through automated checks and a review gate before it reaches production.
What’s the difference between a light and a dark software factory?
A light factory keeps humans involved in design and review, while a dark factory ships code that no human has read and that only machines have verified.
What’s the biggest bottleneck in an AI software factory?
Verification is the biggest bottleneck. Generating code scales with compute, while checking that the product works has traditionally depended on human attention.
Do AI code review bots replace end-to-end testing?
No, because review bots read the diff but never open the app. As such, they can’t confirm that a user journey works.
Can a coding agent verify its own pull request?
A coding agent can run a first pass on its own work, but the final check should come from a verifier that wrote neither the code nor the tests.
Do I still need unit tests in an AI software factory?
Yes, because unit tests remain the fastest layer. They check the code, while running-app verification checks the product.
Will QA agents replace QA engineers?
No. QA agents are in charge of repetitive checks, whereas QA engineers focus on writing acceptance criteria and deciding what the agents should verify.


