Merged Isn't Delivered: Building CommitmentProof, a Two-Agent Switch Workflow

A pull request is merged. 27 of 27 checks are green. Support is ready to tell the customer "it's fixed." Can they?

The short version

• CommitmentProof checks the proof behind a software promise before anyone tells a customer it is kept.
• Two separate Codex agents work in one Switch-connected Discord room with a human. The Evidence Agent collects source-linked GitHub proof. The Coordinator Agent applies a deterministic completion policy and drafts the next step.
• On a real merged PR, microsoft/vscode #331010, the policy returned NEEDS_EVIDENCE. The PR was merged and 27 of 27 checks passed, but nothing proved deployment or QA.
• Instead of closing the promise, the Coordinator proposed an exact GitHub issue. A human replied approve, created issue #2 by hand, and the Coordinator verified it through GitHub's read-only API.
• The workflow concluded with a human-approved GitHub follow-up, independently verified by the Coordinator Agent. Every decision and action remained traceable inside the Switch collaboration

Built for Flip the Switch by WeMakeDevs, Track 01: Best Business Workflow. Repo · Live dashboard · Demo video

Watch the demo: two agents and a human working in the real Switch room.

The problem: merged is not delivered

Support teams close customer promises on engineering status. A developer merges the fix, CI goes green, and someone writes "this is resolved" to the customer. That message is a claim about production. The evidence behind it is a claim about the repository.

The gap between the two is where trust breaks:

• The merge may not have shipped to the customer's environment yet.
• CI proves the tests passed, not that the customer's original scenario now works.
• Nobody may have reproduced the bug and confirmed it is gone.

We wanted a workflow that asks one business question and refuses to guess: can we confidently tell this customer their promised fix has been delivered?

A real example

We ran the workflow on a real public pull request, microsoft/vscode #331010. We do not operate VS Code's production environment, which makes it a fair test: the code can only see what GitHub shows.

Two green rows look like success. The policy answer is NEEDS_EVIDENCE: do not tell the customer the fix is delivered until the missing proof exists. That is the system working, not failing.

The live dashboard on microsoft/vscode #331010: pull request and CI verified, deployment and QA missing, completion claim not permitted.

Why two agents and a human

We split the job so that no single participant both gathers the proof and decides what it means. The agent that collects evidence never declares a promise kept. The agent that judges the evidence never fetches it. A human makes every external decision.

We gave each agent a distinct responsibility inside our Switch-connected Discord room. The Evidence Agent focuses on collecting verifiable technical evidence, while the Coordinator evaluates those findings, prepares the next action, and involves the human before any external change.

This separation makes the workflow auditable: everyone in the room can see where the evidence came from, how the decision was reached, and who authorized the final action.

- name: "{team}-evidence" instructions: | Investigate the commitment with read-only source access. Return PR merged status, required test status, deployment verification, QA confirmation, and a link to each actual source. Never infer success from missing evidence. Send findings to @{team}-coordinator in the room. Do not close tasks or message customers. - name: "{team}-coordinator" instructions: | Receive findings from the evidence agent. If any check fails or is unknown, keep the customer promise open and prepare an evidence-linked follow-up draft. Never publish customer communications or modify GitHub without explicit human approval. 

The split is not decoration. Raw evidence stays separate from the decision about whether a promise is safe to close, and a person can see both in the same conversation.

The architecture

One Switch room, two agents, one human. The only write to GitHub is the human's.

Both agents only read GitHub. The single write in the whole workflow, issue #2, is a human action that comes after an explicit approve.

The dashboard's recorded Switch session: every step labelled as recorded, human-approved or manual, never as a live feed.

How we built it

Every piece of evidence carries its source

github_evidence.py is a read-only collector built on the GitHub REST API. It starts by assuming nothing. Every field begins as unknown, and deployment and QA stay unknown because GitHub cannot prove them:

unknown = {"value": None} evidence: dict[str, Any] = { "pr_merged": dict(unknown), "tests_passed": dict(unknown), "deployment_verified": dict(unknown), "qa_confirmed": dict(unknown), "collection_errors": [], } 

It then reads the pull request, takes the head commit, and pages through every check run on that commit. A single failed, cancelled or timed-out run makes the test evidence negative. Anything queued, skipped or neutral leaves it unknown, never green:

statuses = [check.get("conclusion") for check in checks] all_success = all(status == "success" for status in statuses) any_failure = any(status in {"failure", "cancelled", "timed_out", "action_required", "startup_failure", "stale"} for status in statuses) check_url = f"https://github.com/{encoded_repo}/commit/{head_sha}/checks" value = False if any_failure else (True if all_success else None) evidence["tests_passed"] = {"value": value, "source_url": check_url} 

Every True comes with an https:// link a reviewer can click. That is what the agents paste into the room.

A policy that fails closed

commitment_proof.py is the completion policy. It is 58 lines of plain Python with no model inside it, so the same evidence always gets the same answer. Its docstring is the whole philosophy: "Unknown values fail closed. Review eligibility is NOT completion."

for name, description in ( ("pr_merged", "pull request merged"), ("tests_passed", "required tests passed"), ("deployment_verified", "deployment verified"), ("qa_confirmed", "QA confirmation"), ): entry = evidence.get(name) if not isinstance(entry, dict): reasons.append(f"No evidence for {description}") continue value = entry.get("value") source = entry.get("source_url") if value is False: blocked = True reasons.append(f"Negative evidence: {description}") elif value is not True: reasons.append(f"Unresolved: {description}") elif not isinstance(source, str) or not source.startswith("https://"): reasons.append(f"Source link missing for {description}") 

It can return three verdicts:

The last column is the point. Even with perfect evidence, can_claim_complete stays false. The best the policy can do is hand the decision to a human.

The dashboard

The public dashboard is React, Vite and TypeScript on Vercel, with Python API functions for live, read-only investigations. It shows the four evidence cards, the verdict, and the recorded Switch session. It labels that session "Actual handoff – not a live event feed", because that is what it is.

The run, message by message

Everything below happened in #commitment-proof in our CommitmentProof Lab Discord server on 7 November 2026. The agents are real Switch participants with their own usernames, not characters in a dashboard.

1. The Evidence Agent hands off

The Evidence Agent's handoff said the earlier check read showed 27 of 27 completed runs successful, that deployment and QA remained unknown, and that assess() returned NEEDS_EVIDENCE / REQUEST_EVIDENCE with can_claim_complete=false. Switch recorded the handoff as Evidence Agent message sw_f3edfcaadde246219504809024217a72 → Coordinator follow-up draft sw_62aebc7a04614e88bcd9d06065b6ec04.

The handoff in #commitment-proof: the evidence summary, then the Coordinator's follow-up draft with NEEDS_EVIDENCE.

2. The Coordinator proposes, it does not act

The Coordinator turned the gap into a concrete request. This is the exact proposal it posted:

Title: Verify deployment and QA for merged VS Code fix #331010

The fix in microsoft/vscode PR #331010 is merged, and all 27 completed CI checks succeeded. Deployment to the affected customer environment and QA reproduction of the original scenario are still unverified. Please attach deployment evidence identifying the deployed build/environment and record QA reproduction results before the customer is told the fix is delivered.

Reply APPROVE to approve this exact title and body for you to create manually, or REJECT to decline it. No issue has been created. The automated Discord approval backend is not deployed.

Notice the last two sentences. The agent states what it has not done and why.

3. A human decides

The human replied approve. The Coordinator confirmed it was approving this exact title and body, and again said it had not created the issue. Switch recorded the decision as approval sw_353065a05331d2295d29b05961ea6c17.

The exact proposal, the human's "approve" at 8:54 PM, and the Coordinator acting only after it.

4. The human acts, the agent verifies

The Coordinator confirms it has not created the issue; the human creates issue #2 and asks for verification.

The human created issue #2 and asked the Coordinator to check it, adding: "Clearly state that issue creation was manual, not automated." The Coordinator read the issue back through GitHub's read-only API and reported:

Verified the issue at https://github.com/bh00mi-11/CommitmentProof/issues/2 via GitHub's read-only API. It exists and is open. The title matches exactly: "Verify deployment and QA for merged VS Code fix #331010". The body matches the approved proposal, including the unresolved deployment/QA request and both evidence links (the merged PR and 27/27 successful checks). Issue creation was performed manually by you after approval; it was not automated by CommitmentProof.

GitHub's preview of issue #2 and the Coordinator's read-only check.

The business task moved from a question to a traceable outcome: an open, evidence-linked follow-up that a person approved and an agent checked.
Why this matters to real teams
Engineering, QA and support often operate with different definitions of “done.” CommitmentProof connects those signals before a customer-facing commitment is closed.

Instead of simply reporting a merged pull request, the team receives an evidence-backed decision and an actionable next step. Switch makes that collaboration visible, with specialized agents performing distinct responsibilities and a human retaining final authority.

The result is a reusable workflow for reducing premature delivery claims, improving accountability, and giving teams a clearer path from technical progress to customer confidence.

This is a potential business impact argument, not a claim that you've already measured those outcomes.

The end state on the dashboard: human approved, issue created manually, verified by the Coordinator.

Implementation boundaries and next steps

We would rather you hear the limits from us than find them yourself.

• The demo uses a public PR we do not run. VS Code #331010 shows the evidence logic on real data. The "customer" framing is our scenario, and the completed business action is the verified follow-up issue, not delivery of the VS Code fix itself.

Why it fits Track 01

What we learned

• "Unknown" needs its own state. Most status tools have two colours. The most useful thing our policy does is keep a third one, unknown, and refuse to round it up to green.
• Let the model talk, let the code decide. The agents read, summarise and draft well. The verdict comes from 58 lines of deterministic Python, so a reviewer can predict it and test it.
• Make agents say what they did not do. "I have not created the issue" turned out to be the most trust-building sentence in the whole conversation.
• Separate roles make the chat readable. With one agent gathering and one judging, anyone scrolling the room can see where a claim came from and who decided what.

Where this goes

• Connect a real deployment source (release records or environment health checks) so the Deployment card can turn green on proof.
• Let QA attach reproduction results in the room, so a human can close the loop without leaving Switch.
• Deploy the signed approval service, so an approved proposal can be written to GitHub automatically, still only after an explicit human approve.

Try it

Open the live dashboard, switch to Live investigation, and enter microsoft/vscode and 331010. When we ran it, it returned the same four cards and the same verdict: merged, green, and still not delivered.

Where would you draw the line on what an agent may decide on its own? Tell us in the comments, and if the idea is useful, a star on the repo helps.

Prove it before you close it.

Links

• Code: github.com/bh00mi-11/CommitmentProof
• Live dashboard: commitment-proof.vercel.app
• Verified follow-up: CommitmentProof issue #2
• Investigated PR: microsoft/vscode #331010 and its check runs
• Demo video: youtu.be/_T1D_nQKd2g