I made three AI seats argue over the spec before any code. Here's what 10 runs showed.

Most of the bad code I've gotten from coding agents wasn't badly written. It was the right code for the wrong plan. The agent misread the task, assumed something about the repo that wasn't true, and then executed that assumption very well.

So I built a tool that spends its effort before the code. It's called Open Council. Three seats argue over a PRD until they all sign off on the same document:

• Lead PM writes the brief and the PRD.
• Chief Engineer checks every claim against the real code. Every objection has to cite a file:line.
• Chief Architect rules on deadlocks and reviews the agreed document. It runs with no tools, so it judges the text, not the repo.

When all three sign off, I approve the PRD with a token. Only then does the Engineer implement it, in an isolated clone. The harness runs the project's gates (its real test commands) and opens a PR only if they pass. Anything else ends as BLOCKED or STALLED, never DONE.

Last week I had ten councils run on version 0.2.1 to see how this holds up. Here's what happened, including the parts that didn't work.

The setup

All ten ran in solo mode, which means every seat is Claude Code (Lead PM on Sonnet, Chief Engineer on Opus, Chief Architect on Sonnet with tools off). They used the quick profile, with at most 4 rounds. The targets were local copies of two small repos: a toy slugify library I keep for testing, and Open Council itself. Local copies have no GitHub remote, so no run could open a PR. That last step ends BLOCKED by design.

8 of 10 signed off. 7 of those 8 passed their gate. The median time to a sign-off or a stall was 6.5 minutes.

The catch nobody asked for

Six of the runs were on the slugify repo. In all six, the Chief Engineer's first objection had nothing to do with the task.

The repo's package.json had "test": "node --test test/". On Node 22 that argument is read as a module path, not a folder, so npm test ran 0 of the 2 existing tests. It had been quietly broken since before I started. No task mentioned it.

In run 1 the Engineer reproduced it, tried two working alternatives, and made fixing the test script the first commit of the plan. Its reasoning was simple: you can't claim the new tests pass if the runner doesn't run tests.

This is the thing I wanted from the design. A single agent asked to "handle empty strings in slugify" would most likely write the fix and the tests and report that they pass. A seat whose only job is to check claims against the repo found that the claim couldn't be checked at all.

What went wrong

Two runs stalled, and the same way. In both, the Engineer had no objections left. Then the Architect raised one more concern. In run 4 it asked for a README line the task never mentioned. The revision that answered it introduced a new problem: one of the new assertions would fail on a correct implementation. The Engineer objected in the last round, the budget ran out, and nothing reached me for approval. The harness wrote the last draft and the open objections to a file and stopped.

A stall is the system working, because nothing wrong got built. But two stalls with the same shape tell me the Architect's late concerns cost more than they're worth. I already made one change for this: a concern that could have been raised in an earlier round is waived on a resubmission. These two runs show it isn't enough yet.

One run was blocked by its own plan. In run 7 the council agreed on a plan and named its own test command, node --test test/slug.test.js. But the project config runs npm test, which was still broken in that repo, and the Architect had ruled that fixing package.json was out of scope. So execution ran the configured gate, it failed, and the run ended BLOCKED. Nothing was pushed. The code itself passed the plan's own 7 tests when I checked afterwards.

The harness did the right thing by not calling it done. The real bug is that a PRD can name a gate the project doesn't run, and nothing flags that before execution. That's an open issue now.

Solo mode isn't the real test. Every seat in these runs was the same model family. The point of separate seats is that they don't share blind spots, and the council is designed to put Claude, Gemini and Grok in different seats. I haven't run that mixed setup enough to say whether it catches more. Ten runs on two small repos, with three tasks repeated, is a smoke test, not a rate.

What I took from it

1. Review the plan, not just the diff. The most useful objection in these runs was about the repo's state, not the task. It only came up because a seat was told to check claims against the code before anyone wrote any.
2. Make "done" something the harness checks, not something the agent says. Run 7 had passing code and still ended BLOCKED, because the gate the project actually runs failed. I'd rather read a BLOCKED than find out later.
3. Publish the failures. The stalls taught me more than the sign-offs, and they're the part I'd want to see before trusting a tool like this.

Trying it

You need Node 18 or later, macOS or Linux, and Claude Code. In a git repo:

npm install -g open-council-cli council init --solo council doctor council backlog add "your task here" --priority 1 SESSION=$(council open my-task --item 001) council deliberate "$SESSION" 

When it signs off, it prints the PRD's path and the exact council approve command. That step is yours to run. Then council handoff "$SESSION" and council run "$SESSION" implement it and open the PR.

The code, and full replays of these runs with every objection, are at https://github.com/goshtasb/OpenCouncil. It's MIT licensed. If you run a council and it stalls or ships something wrong, that's the report I'd most like to read.

This article was written with the help of AI, and the ten runs were driven by an AI agent on my behalf. I checked the numbers against the run logs before publishing.