This post was written as part of the Kiroween 2026 hackathon (#KiroweenHackathon26).

A customer writes in: the earbuds arrived broken, can I get a refund? An AI support agent looks up the order, checks the refund policy and asks the payment service to refund $129. The service commits the refund, but its response arrives after the client has already timed out. The app retries, as every retry loop does. The customer gets $258 back and is told $129.

Nobody did anything exotic. A timeout after commit is one of the most ordinary failures in production. What's new is that an LLM sits in the middle deciding what happens next, and the tools it calls move money and delete servers.

When that goes wrong, you usually have a log and a lot of guesswork:

  • You can't safely re-run it. Re-running the incident refunds the customer a third time.
  • You can't easily reproduce it. Making the service time out at exactly the right moment is hard.
  • You can't prove the fix. Once you change the code, no test shows this exact failure can't happen again.

So for Kiroween I built GhostReplay Lab: a flight recorder and crash-test lab for AI agents.

The loop: record, replay, break, prove

GhostReplay runs one loop end to end:

Record → Replay offline → Inject a failure → Detect unsafe behaviour → Fix → Prove with pytest

  1. Record. Every model call, tool call, clock read and generated id goes through one door (AgentRuntime). It is captured in order, with secrets redacted before anything is stored.
  2. Replay offline. Your agent's real code runs again, but every response comes from the recording. Tool functions are replaced by stubs that raise if called, and the network is blocked on the replaying thread. The replay is checked to be identical to the recording, event for event.
  3. Break it on purpose. Pick any recorded step and inject one of 15 fault kinds: a timeout after commit, HTTP 429, stale data, clock skew, a replaced model decision and more.
  4. Detect. Invariants such as refund.no_duplicate or infra.capacity_preserved check the run and explain, with evidence, why it was unsafe.
  5. Prove. The incident becomes a pytest regression case that FAILS on the buggy agent and PASSES on the fixed one, and it runs in CI.

The rule I cared most about: never guess. If an injected fault makes the agent take a path that was never recorded, strict replay stops at the first divergence and tells you, instead of inventing what the agent "would have done". Faults that need a real side effect run as a clearly labelled branch execution that re-executes for real.

Two very different incidents, one framework

I entered the Skeleton Crew category: one shared framework and two separate applications built on it.

The Double Refund Incident. A support agent backed by a real SQLite database and a local HTTP refund service with transactional commits. The bug: the app retries a timed-out refund without an idempotency key. The fix: a stable idempotency key plus a check for existing refunds before any retry. One refund, invariant PASS.

The AI Agent That Almost Deleted Healthy Servers. A DevOps agent talking to three real HTTP instance services and an allowlisted controller. After a bad deploy, stale monitoring says the healthy instances are impaired. If the workflow follows its runbook literally, it terminates all three. The fix: a safety gate in code that probes each target directly and requires approval. All three servers stay up.

Neither app contains any replay, fault, diff or regression logic of its own. A refund bug and an infrastructure outage are caught by exactly the same code.

What surprised me: the real model was more careful than my workflow

The model decisions in the demos come from genuine Claude tool-calling runs, recorded live and shipped as read-only captures. Every click on the site re-executes those runs for real against the local services. Two things came out of those live runs that I didn't expect.

Claude refused the dangerous runbook. It read the logs, noticed the instances were probably fine and suggested a rollback instead. Great, but that's the model's judgment on one day with one prompt. So the demo shows that real decision, then injects the unsafe one (clearly labelled) to test the workflow. Without a safety gate, the workflow destroys the fleet. Test the workflow, not the model's mood.

The bug lives in the application, not the model. When the tool schema allowed it, Claude passed its own idempotency key and the duplicate refund disappeared. The buggy app now never offers one, which is exactly how this bug looks in real systems.

How I used Kiro

Kiro was how I organised and extended the project.

Steering. The steering documents in .kiro/steering/ hold the product's rules. The most important is honesty: never present injected or scripted behaviour as real, and label every event with where its response came from. They also cover the repository structure, the stack, coding conventions and the security model. When Kiro added features, the new UI followed those rules, for example badges that say whether a run's decisions came from a live model or a scripted policy.

Specs. Every component has a spec with requirements.md, design.md and tasks.md: replay engine, failure injection, both labs, the dashboard and the SDK. Each task names its verification commands, and the rule is to write or extend a test first, implement, run the checks, then tick the task.

Running tasks in Kiro. I ran open spec tasks from Kiro's task view (Kiro 1.4.121). That work added:

  • two new fault kinds, reordered_response and clock_skew (which targets clock reads, so refund windows and expiry checks can be broken on purpose);
  • an infra.approval_chain_valid invariant that catches self-approved changes;
  • honest recovery for failed probes in the safety gate;
  • model usage and "Copy as JSON" in the inspector, and live/scripted badges in the explorer.

Kiro keeps an execution history for each spec in tasks.meta.json, committed alongside the code.

Hooks. Agent hooks in .kiro/hooks/ run the framework guarantee tests when the framework changes and the scenario regressions when a lab changes. They also type-check the dashboard, keep its TypeScript types in sync with the Pydantic schema, lint on stop and scan for secrets.

The specs made the biggest difference. A task like "add clock_skew" touches the Pydantic schema, the fault catalog, the dashboard types, the Failure Lab UI and the tests. Writing that down as requirements, design and numbered sub-tasks with verification commands kept each change small, testable and consistent across Python and TypeScript.

Try it

In the browser. Open https://ghostreplay.hayaix.tech and click Watch an AI Agent Fail. The six-step guided demo takes about three minutes, with no sign-up and no API key.

With your own agent. The Python SDK wraps your tools and model calls with decorators:

from ghostreplay import GhostReplay, fault ghost = GhostReplay("my-agent", storage="./ghostreplay-data") @ghost.tool("payments.charge", destructive=True) def charge(order_id: str, amount_cents: int, key: str | None = None) -> dict: ... @ghost.agent("buggy") def run(task: dict) -> dict: ... incident = ghost.record(task, agent="buggy", faults=[fault("timeout_after_commit", "payments.charge")]) ghost.replay(incident) # offline: none of your functions is called ghost.write_regression_test(incident, "tests/regression", agent="fixed", import_module="my_agent")

Agents will keep making expensive mistakes. The least we can do is catch them once, replay them safely and make sure they never happen again.

Built for Kiroween 2026. #KiroweenHackathon26