When a production outage hits at 2 AM, every second counts. I built IncidentRoom — an autonomous SRE war room that turns any team chat channel into a self-coordinating incident response system using Switch AI.
Here's what actually happened during our live demo: a SEV-1 alert fired, three AI agents collaborated in a shared Mattermost room, identified the root cause commit in under 60 seconds, ran real bash commands to prepare a rollback, waited for a single human APPROVE command (voice-dictated via Wispr Flow), executed the fix, and auto-generated a full post-mortem — all in under 2 minutes.
The Problem We Solved
Traditional incident response is chaotic:
- Context is fragmented across monitoring tools, Slack threads, and git history
- Engineers waste precious minutes manually correlating stack traces to recent PRs
- Stakeholder updates distract the on-call engineer from actually fixing the issue
- Post-mortems are written hours or days later from memory
MTTR (Mean Time to Resolution) suffers not because engineers are slow — but because coordination is broken.
Enter Switch AI
Switch is an open-source platform that connects AI coding agents (Claude Code, Codex, OpenCode, Cursor, Antigravity) into team chat channels as first-class participants. Agents can read messages, respond to @mentions, run bash commands, hand off context to each other, and await human approval — all inside Mattermost, Slack, Teams, or Discord.
This is the missing piece for incident response.
Architecture: 3 Agents, 1 Room, Zero Chaos
📢 Alert fires in #incident-war-room │ ▼ @triage-agent ──→ 11 tool calls ──→ Root cause: commit 8f2b31a │ ▼ HANDOFF @sre-agent ──→ npm pack stripe@3.4.0 ──→ "Awaiting APPROVE" │ ▼ Human: "APPROVE" (via Wispr Flow voice) @sre-agent ──→ Rollback executed ✅ │ ▼ @comms-agent ──→ Post-mortem generated 📄 The Three Agents
@triage-agent — The Incident Detective
Receives the alert via @mention, parses the stack trace, correlates against recent git commits, and isolates the root cause. In our live demo:
triage-agent BOT · Worked for 3m 16s. 11 tool calls. 🔍 TRIAGE — root cause identified: commit 8f2b31a ("feat(payments): upgrade stripe sdk v3.4") Root cause: services/payments/stripe_gateway.js:84 passes a null options object to the Stripe create call; stripe-node v3.4 now unconditionally reads options.idempotency_key, so createSession throws before reaching Stripe. @sre-agent — The Remediation Specialist
Receives the handoff, evaluates blast radius, executes real bash commands to prepare the rollback — but never deploys without human approval:
# Actual command run by @sre-agent npm pack stripe@3.4.0 --pack-destination /tmp/opencode/stripe340 Then waits. No autonomous deployment. Human-in-the-loop is non-negotiable.
@comms-agent — The Incident Commander & Scribe
Handles stakeholder communication and auto-generates the post-mortem with MTTR, timeline, root cause, and action items.
Voice-Powered with Wispr Flow
Here's the detail that makes this truly hands-free: I used Wispr Flow to voice-dictate every incident command without touching the keyboard.
While monitoring dashboards, I spoke:
"Alert: Save one HTTP 500 error spike in checkout payment gateway at triage agent. Please investigate the root cause. Commit. Approve, roll back, and notify all stakeholders. @Comms agent generate the postmortem report."
Wispr Flow transcribed this at 100 wpm, 405 total words across the incident session. That's the entire incident — triage, approval, and post-mortem command — dictated hands-free.
The Live Results
| Metric | Result |
|---|---|
| Time from alert to root cause | < 60 seconds |
| Triage agent tool calls | 11 calls, 3m 16s |
| Human touch-points | 1 (APPROVE) |
| Wispr Flow words dictated | 405 words @ 100 wpm |
| MTTR | Under 2 minutes |
| Post-mortem generation | Automatic |
Community Wisdom That Shaped This
Building this, I learned from the community:
- Agentic incident response is where autonomy meets the pager — reinforced that read-only defaults and approval gates are non-negotiable
- NexusOps: Multi-Agent SRE Command Center — validated the multi-agent pattern for SRE workflows
The key insight both posts validate: agents need approval gates before any write operation in production. We enforced this strictly — @sre-agent never executes a deployment without an explicit human APPROVE in the room.
How to Run It Yourself
git clone https://github.com/swatiicfai/IncidentRoom cd IncidentRoom cp .env.example .env # Add your Switch server URL npm install node src/simulate.js # Trigger a live SEV-1 simulation You need:
- Switch Console running locally
- OpenCode (or any Switch-compatible agent) installed
- Mattermost running in Docker (Switch provides this out of the box)
What's Next
- Webhook integration — Datadog/Sentry/PagerDuty alerts fire directly into the room
- Canary verification —
@sre-agentchecks error rate baseline post-rollback before declaring resolution - Multi-room escalation — auto-create a dedicated room per SEV-1 instead of using a standing channel
- Agent memory — triage agent remembers past incidents to pattern-match faster
🔗 Resources
- GitHub: https://github.com/swatiicfai/IncidentRoom
- Demo video: https://youtu.be/gvqgkVetPTc
- Switch AI: https://github.com/sandbox-quantum/switch
- Post-mortem: See POSTMORTEM_LIVE.md
Built for the Flip the Switch Hackathon by WeMakeDevs × Switch AI.
This post was written with AI assistance (Antigravity/Gemini) during the hackathon session.
