When an AI workflow breaks, responders need to know what it tried to do and which actions may already have happened. A useful incident runbook should help contain the issue first, then preserve evidence and restore service safely.
1. Stop risky side effects
Disable the affected trigger or tool permission if the agent could repeat a harmful action. Pause queued jobs when duplicate execution is possible. Keep a human path available for urgent requests.
2. Establish the scope
Record the first known failure time, workflow version, model and prompt changes, affected integrations, and example run IDs. Check whether the issue is a bad answer, tool failure, latency spike, or unintended external change. Avoid copying sensitive prompts or user data into broad incident channels.
3. Recover deliberately
Use a known-good workflow or manual process while investigating. If external writes occurred, reconcile them with the system of record before replaying jobs. Do not blindly retry a non-idempotent action.
4. Verify and learn
Test the fix against the failed case and nearby edge cases. Re-enable traffic gradually, watch the completion and error signals, and document who owns the next step. Turn the incident into an evaluation case, alert, or permission boundary.
The runbook should name an incident lead, a technical owner, rollback criteria, and customer communication owner. Keep it short enough to use under pressure and review it after changes to tools or workflows.
More notes on operational AI delivery: https://umer-amin.vercel.app/. What is the first action your team takes when an agent starts behaving unpredictably?