One Jira Ticket Through an Org of AI Agents: The Architecture

Following a single story from the board's goal to merged code, through PM clarification, estimation, budgets, risk gates and scorecards.

In the last post I sketched an org chart for AI agents: seats that a human or an agent can fill, capacity split down by percentage, probation and benching, and a human overseer on every agent seat.

The most common question was: "OK, but what actually happens when a ticket shows up?"

So this post follows one ticket all the way through. It's PLAT-412: "Rate limit the public /export API", and we go from the board's quarterly goal to merged code in production.

👉 Interactive diagrams: probita-ai.github.io/agent-org-model/architecture.html

The one rule behind everything

Agents run in one layer. Everything that limits them is plain software.

• Every tool call passes a policy engine.
• Every model call passes a metered gateway.
• The kill switch revokes keys at those two points. Stopping an agent never depends on the agent cooperating.

No model sits in the decision path for budgets, permissions or approvals.

Every level adds something before handing down

Every seat uses the same card: what comes in, what it decides, what it hands out, when it escalates, and what it's scored on. That consistency is what makes it possible to swap a human for an agent (or back) in one field.

Inside the coder: 12 phases

This is where most people's mental model of "an AI coder" is too thin. "Read ticket, write code" skips the parts that make human engineers good.

1. Intake. Reserve budget, read the brief from the lead.
2. Understand. Build an understanding doc: restated goal, code map, blast radius, unknowns.
3. Definition of Ready. Is the ticket buildable?
4. Clarify with PM. (More below.)
5. Estimate. (More below.)
6. Plan. For medium-risk work, the reviewer approves the plan before code. That costs about $0.40 and saves about $15 of wrong-direction work.
7. Implement. Sandbox, test first, checkpoint after every green commit.
8. Self-check. PR checklist, mostly deterministic.
9. Hand off. PR plus two structured briefs (tester, reviewer).
10. Review loop. Max 3 rounds, then escalate.
11. Release gate. A human approves production.
12. Close and learn. Record actuals; propose a lesson to crew memory.

The single most useful rule is in phase 2: map every acceptance criterion to a concrete test. Any criterion that can't be mapped is marked ambiguous. That one check catches most unclear tickets before anyone writes code.

The coder also has hard stop rules, enforced by the orchestrator rather than the prompt:

• Same error 3 times → escalate with the trace
• 80% of budget → pause and re-estimate
• 100% → the gateway refuses further calls
• Diff touches files outside the plan → flagged for review
• More than 120 steps → pause

Talking to the PM

PLAT-412 says "limits configurable per plan tier." Per user? Per org? Per API key? The ticket doesn't say. Which is realistic.

The coder sends one batch of questions, not a drip:

type: question_batch estimated_pm_minutes: 4 questions: - id: q1 blocking: true text: "What should the limit be keyed on?" options: a: "API key (matches how /export authenticates)" # default  b: "Organization" c: "User" why: "Changes the Redis key design and what customers see." - id: q3 blocking: false text: "Build the dashboard, or emit a metric ops adds?" options: a: "Emit metric only" # default  b: "Build the panel (+2 points)" 

The rules:

• Every question has options and a default, so the PM can answer in one click.
• Blocking vs non-blocking. If nobody answers in a business day, non-blocking questions proceed on the default and are marked as assumptions on the ticket.
• A PM-assist agent pre-answers what it can from docs and ADRs. A human confirms.
• The coder can propose new acceptance criteria wording, but can't change it. Only the PM seat holds that permission.

And the PM picked organization, not the default. That's the whole argument for asking instead of assuming. The answer is recorded as a decision, and decisions are injected into every later step, so the reviewer doesn't "fix" it back.

Every question also draws from a human-minute budget. The PM publishes 300 minutes a week for this crew. Agents are scored on human minutes per point, so lazy questions have a cost.

Estimation: four numbers, not one

A human team estimates in story points. Agent work needs more, because the scarce resources are different.

The method:

1. Extract features: files touched, services, coverage, external calls, risk.
2. Reference class: the 5 most similar closed jobs, with their actual costs.
3. Adjust, with every multiplier written down:

base P50 (weighted median of similar jobs) $12.00 × 1.15 external call with no cache × 1.05 customer-facing (flag + rollout) P50 ≈ $14.00 × 2.0 spread factor (from this agent's own history) P90 $28.00 

1. Cross-check against the sum of task points.
2. Blind planning poker: coder, tester and reviewer estimate without seeing each other. Here: 5, 5, 8.
3. Exactly one round of reasons, then the lead decides. Unbounded agent debate is a cost sink.

Actual: $13.20, inside P90. That feeds back into calibration: an agent whose P90 holds about 90% of the time earns a tighter spread factor. One that always says "13 points, P90 $200" is safe but useless, and loses score too. To stop point inflation, the reviewer assigns "actual points" after the fact.

There are split rules too. More than 8 points, a P90 over $40, more than 2 services, or a migration plus a behaviour change means the coder must propose a split to the PM.

The surprise: sprints run out of humans first

When the lead agent planned the sprint, dollars fit fine ($980 P90 against $1,020 plannable). PM minutes didn't (410 against 300), so a ticket was dropped.

That changed the whole design. Human attention is budgeted the same way as money: reserved per job, metered and reported. The useful question about an agent isn't "how fast is it?" It's "how many human minutes does each point of its work cost?" That number decides how many agents a team can actually supervise.

Context doesn't survive handoffs by accident

The biggest weakness of deep agent hierarchies is context loss. The fix is briefs: structured packets where each level adds and never just forwards.

• CTO adds the goal and metric.
• PM adds the customer story and acceptance criteria.
• Lead adds constraints ("existing Redis only") and known context (ADR-017).
• The thread adds binding decisions ("key by org, decided by Dana").
• Coder adds "where to look hardest" for the reviewer.

Briefs link to sources rather than pasting summaries, so the receiver can always go to the original.

Budgets, gates, and firing

Capacity. Shares flow down by percentage. Each job also reserves its P90. It pauses at 80%, and at 100% the gateway refuses calls. Unused budget goes back on close.

Risk gates. Every tool call is classified, not every agent. Reading a repo is low risk and allowed. Opening a PR is medium and sampled. A migration or a prod flag change is high and needs human approval or a narrow, expiring permit. Secrets and pushing to main are never allowed for agents.

Scorecards. Real outcomes outweigh another model's opinion: escaped defects, CI results and human overrides weigh more than reviewer-agent verdicts. On this ticket, the probation agent ran in shadow mode, keyed the limit by API key, and ignored the PM's decision. Its score was 0.58 against a 0.75 bar, so it stays on probation. Nothing it produced left its sandbox.

What I'd build first

Bottom up, as the first post argued:

1. Skeleton: seat registry, ledger, metered gateway, tool proxy, kill switch.
2. One coder agent on real tickets, humans in every other seat.
3. Calibrate estimates on 50+ jobs.
4. Add tester and reviewer, briefs, human sampling.
5. Probation and bench on real signals.
6. Only then, an agent lead at L2, and only if it pays for itself.

Open questions

• If agents are scored on cost per point, will they inflate points? Is reviewer-assigned "actual points" enough to stop it?
• Should crews be required to use a different model vendor for the reviewer seat, to avoid shared blind spots?
• How would you measure human minutes without making people log time?

Full diagrams: architecture page · Text version: docs/architecture.md · Repo: probita-ai/agent-org-model

Which of these phases would you trust an agent with today, and which would you keep human?