Originally published on Hashnode.
How Do I Debug an AI Agent Run After It Finished?
Logs show what an agent printed. A recorded run shows what it did, and it can be trusted because replay uses the same code path as live.
By Mithilesh Gaurihar. Last verified: October 2026.
Quick Answer. Debugging an AI agent run after it finished needs a recording of what each step was handed, what it returned and how long it took. RocketRide records every agent pipeline run and replays it in the VS Code extension: play it back at up to 25x, pause, step request by request, and open any tool or model call to read its input. Replay matches live because both use the same client code over the same server-stamped events. Recordings last up to 7 days, or 30 when deployed.
Why Don't Logs Tell Me What My Agent Did?
Once an agent runs for other people, the questions turn forensic. What did it do in the thirty seconds before yesterday's batch went wrong? Which tool call was slow, and what was it handed?
A log can't answer that. A log line records what a step chose to print. It doesn't record what the step was handed, what it returned, how long it waited, or what else was in flight at the same moment. Rebuilding concurrency from interleaved print lines is archaeology, and it gets worse under load, when more lines arrive from more places at once.
What Is Agent Run Replay?
Agent run replay is playing back a recording of a finished agent run, call by call, in the order and with the timing it happened, so the run can be inspected without running it again. For a replay to answer the questions a log can't, the recording has to keep five things:
- the input and output of every model and tool call
- when each call started and how long it took
- which calls overlapped
- the stretches where nothing was in flight
- timestamps set where each event happened, so playback can't drift from the original
A tool that keeps less than that is closer to a log with a timeline. In RocketRide, an agent runs inside a pipeline: a connected set of steps called nodes, such as a model, a tool or a database. So an agent run is a pipeline run, and the recording covers every node in it. RocketRide records every run, and any recorded run can be replayed, paused, rewound and fast forwarded. The recording keeps all five at the default trace level, with very long payloads shortened, and the rest of this post shows each one on a real run.
What Does Replaying a Finished Run Look Like?
The replay controls sit on the Development tab of the VS Code extension, under a row of panes: Status, Tokens, Flow, Trace, Errors, Log and Utilization. Below the panes is a strip of activity with a red needle, and clicking the time readout opens the list of runs. The pipeline in these screenshots serves a MySQL database to an agent as tools. The agent itself runs elsewhere and calls in, so what the recording holds is the tool side of that conversation: every query the agent asked for, and the silences in between. It ran three times that morning: at 08:32:44 AM for three seconds, 09:31:14 AM for ten minutes, and 10:42:57 AM for seventeen minutes.
Pick the 09:31 run and play it at 10x, and its ten minutes pass in one. The traces scroll through, the usage counters move, and the flow and utilization panes fill in, all as they were recorded hours earlier. The controls do what a video recorder's would: play and pause, a speed selector from 0.25x to 25x, previous and next run, step back and forward. The needle stays fixed and the strip slides under it.
The step buttons jump by a size I choose: a fixed interval such as 30 seconds, or Request, which jumps to the next finished call. The same controls work on a run still in progress: press GO LIVE, start the pipeline, and a green bar grows on the strip while the Log tab shows what is happening. Pausing drops out of live and leaves me on the same controls I use for a finished run.
Where Did the Time Go in My Agent Run?
The Status tab shows "Where the time went" for the run: calls, total time and worst time for each node. Replaying the 09:31 run, the only node was MySQL: 30 ms across six calls, 5 ms on average, 30 ms at worst. Beside it is a list of the slowest calls, labelled Slowest completions, and each row opens straight into the trace for that one request.
The tile at the top of that same screen reads 0 completions, which looks like a contradiction next to a list of six. It isn't: the word means two different things on one screen. The tile counts items that finished moving through the pipeline, the throughput number a document-processing run would fill in. This pipeline ingests nothing; it sits there and answers an agent's questions, so that counter stays at zero all run. The list underneath counts calls. Worth knowing before you read a zero as a broken run, and worth remembering that the tiles describe a pipeline's data throughput while the list describes its work.
Opening one of those calls showed the exact SQL the agent sent and the rows that came back. A slow request stops being a statistic and becomes a specific call I can read.
The measurement that surprised me is idle time: the Longest idle tile for this run reads 3 min. The runtime opens a quiet window when the last in-flight call ends and closes it when the next one begins, so the number is the longest stretch in which nothing at all was running. Overlapping calls never count as idle.
What it does not tell you is why. The runtime measures the silence; it has no view of the cause. For this pipeline the explanation is almost certainly the other side of the connection: it serves tools to an agent running elsewhere, so three quiet minutes means the agent spent three minutes not asking it anything, probably reasoning or working on something else. On a pipeline that fetches its own work, the same gap would point somewhere else entirely.
That is still the useful part. When an agent feels slow, the instinct is to hunt for the slow node, and often there isn't one. No per-node timer can show a gap, because a gap is the absence of the thing being timed.
How Can a Replay Be Trusted to Match the Live Run?
The obvious way to build replay is to store events and write separate code that reads them back into a view. That leaves two code paths, live and replayed, that must agree forever. They drift quietly, and the replay stops being evidence.
RocketRide takes the other route. Every event is time-stamped on the server when it happens, and the same code draws the screen whether the events arrive live or come back from the recording. The shortening and sampling further down this post apply to both: the live view is reading the same recorded events, so it shows exactly what a replay shows. Replay is not a reduced copy of live, and live is not richer than what you can get back tomorrow. A replayed row is the row a viewer would have seen live, and the Status numbers come from the server too, so they hold wherever the needle sits.
Can My Team Replay a Run I Deployed?
Yes. In RocketRide Cloud, a pipeline deployed to a team is recorded with that team, and any member with the monitor permission, the right to watch a team's runs, can open the recording and replay it. Members get that permission by default, and org admins can open any team's deployments in their organization. When a deployed agent misbehaves overnight, whoever is on call opens the same run I would, with no exported log and no screen share.
Development runs work the other way. A run I start while building, or a deployment to my personal space, is stored under my account. No teammate, team admin or org admin has a way to open it through RocketRide, so half-finished prompts and test data stay mine.
How Do I Hand a Recorded Run to a Coding Agent?
A recording holds whatever the run handled. At the default trace level that means each call's inputs and outputs, so a run that moved customer records recorded those records. Keys held in a node's configuration are not written into a recording, but a secret that travels as data, in a URL, a request body or an error message, is recorded like anything else. Treat a downloaded run with the same care as the data its pipeline touched, and check what is in it before sending it anywhere, including to a coding agent.
The Log tab has two buttons, Download Log and Download Events, which save the whole run as a text log and as a file of every recorded event. The use I keep coming back to is blunt: download the log, drop it into Claude Code, and ask what went wrong. Turning on the Pipeline Debug Output setting adds the runtime's internal debug messages, which is when an agent reading the file earns its keep.
There's also a route with no download. The RocketRide server's MCP endpoint (Model Context Protocol, the standard way coding agents connect to outside tools) exposes recorded runs as tools, so a coding agent connected to it can list past runs, read their events and open any single request's trace directly.
Models hallucinate, and a person still needs a fast way to check what the model claims. The download or the MCP tools make the agent quick. The recording is what lets me check its answer.
What Do I Gain From a Replay Over Logs?
The largest gain is not having to know in advance what to print. The recording captures what each step was handed and returned whether or not anyone thought to log it, so yesterday's incident can be reviewed without reproducing it.
| Question after an incident | From logs alone | From a recorded run | What it gives me |
|---|---|---|---|
| What was the step handed? | Whatever it printed | The full input and output of each call, at full trace level | The exact SQL or prompt, instead of a guess |
| Which step was slow? | Subtract timestamps by hand | Calls, total and worst time per node | The slow node named in one view |
| Was anything waiting? | Nothing: a gap prints no line | Longest idle stretch and when it began | The upstream wait no per-node timer shows |
| What ran at the same time? | Interleaved lines | The whole run on one timeline | Concurrency I can watch instead of rebuild |
| Is the provider the problem? | A timeout line, if one was logged | A request that stays open on the timeline | A long wait on the provider is visible as a wait, which is a starting point rather than a verdict |
One more gain sits outside the table: there's nothing to add to the agent's code, because the recording happens in the runtime.
What Trace Level Should I Run My Agent At?
The Pipeline Trace Level setting in the VS Code extension takes four values, none, metadata, summary and full, and defaults to full. Each level records everything the one below it does, plus more:
| Level | What gets recorded | Use it for | What it gives me |
|---|---|---|---|
| full (default) | Each call's input and output, per node, with long text shortened | Debugging, and a newly deployed pipeline's first weeks | Any slow call opened and read |
| summary | What each node hands to the next and the final results, without per-call payloads | Production, once the pipeline is proven | The shape of every run with a lighter recording |
| metadata | That each call happened, in order, with sizes for text and table writes; no inputs or outputs | A proven pipeline where only the shape of a run matters | Timing and sequence with none of the payloads |
| none | Log output and counters; no flow traces | Runs nobody will need to look inside | The smallest recording |
Setting the level on a deployed pipeline
The setting above covers runs you start from the editor. A deployed pipeline carries its own trace level, chosen when you publish it, and when nothing is chosen it records at full, the same as a dev run. So the pipelines running unattended are the ones recording the most, which is useful for the first weeks after a deploy and worth revisiting once it is proven.
Reading a recording through the MCP tools follows exactly the same permissions as reading it in the editor: the tools are a thin wrapper over the same commands, so monitor rights on the owning team are still required. Connecting a coding agent does not widen who can see a run.
Where Does a Replay Not Help Me?
Four limits to design around.
- History is recent, not permanent. Runs age out after 7 days in development and 30 days when deployed, and sooner on a busy pipeline, because each pipeline's history is also capped at about 1 GB and 512 runs. A run that matters for compliance or a post-incident record needs a downloaded copy.
- The recording is slightly thinner than live. Long text, such as a big prompt, is cut to its first 2,000 characters in a trace, and the Status numbers are recorded as periodic samples rather than every change.
- It records; it doesn't stop the pipeline. Stepping moves the needle through what was recorded. It can't hold a live pipeline at a breakpoint, and it can't show what a change would have done.
- Access is team membership. Nobody is told who opened a recording, and there's no link to share one. Removing someone's access means changing their team membership or permissions.
FAQ
What is time travel debugging?
Time travel debugging records a program's execution so it can be stepped backward and forward afterwards, instead of re-running the program to reproduce a bug. Classic tools do this for a single process. RocketRide applies the idea to a whole AI pipeline run: each step's events are recorded on the server, and the VS Code extension replays them on a timeline with pause, step and speed controls.
Distributed tracing vs replaying a run: what is the difference?
Distributed tracing follows one request across services and answers "where did this call go". Replaying a run in RocketRide shows a whole pipeline execution on one timeline, including the stretches where nothing ran, so it answers "what was the pipeline doing at 8:32". Per-request traces sit inside the recording, and the two work well side by side.
Does replaying an AI agent run call the LLM again?
No. Replaying an AI agent run in RocketRide reads the recorded events back through the same view, so no model is called, no tool runs and nothing new is charged. The counters that climb during replay, including Tokens charged (RocketRide compute, not model tokens), were recorded in the original run. To see what a changed prompt or setting would do, start a new run and compare the two recordings.
Langfuse vs LangSmith vs a built-in replay: which do I need?
Often more than one. Langfuse is open source (MIT) and can be self-hosted; LangSmith is LangChain's hosted platform, with self-hosting on its enterprise plan. Either covers long-term retention, dashboards, alerting and evaluation. A RocketRide replay is recent history inside the IDE, kept 7 to 30 days. RocketRide's open source OpenTelemetry bridge, the rocketride otel command, exports pipeline traces to both, so they run side by side.
