"It worked when I tried it" is the weakest sentence in software. It is especially weak for an agent, because the demo is a single sample drawn by the person who wants it to pass. The demo proves that a path exists. Production is the distribution of every other path.
So the question I kept running into while operating an assistant was not can it do the task but how do I earn the right to stop watching. The answer that held up was not a better demo. It was a boring practice borrowed from data migrations and feature rollout: run the agent in shadow mode, write down what it would have done, and diff that against what the human actually did. Permission gets earned by a diff history, not by a demo.
What shadow mode means for an agent
Shadow mode is not logging. A log records what happened. A shadow run records what would have happened — the full pipeline through retrieval, decision and action drafting, stopped immediately before the side effect.
The stop has to be structural, not a flag the model reads. If the only thing between the agent and a purchase is a sentence in a prompt saying "do not submit", you have not built shadow mode; you have built a suggestion. Make the executor refuse: the shadow runner is handed a tool adapter whose write methods raise. The agent can plan a submit, and the plan dies in code, not in the model's judgement.
What the shadow run writes is a proposal record. Four fields are load-bearing:
- Input digest — the exact inputs, with enough identity to re-fetch them later (message id, page URL and fetch time, file hash). A proposal you cannot reconstruct is an anecdote.
- The effect it would commit — not prose. The concrete action object: the fields it would submit, the option ids it selected, the amount, the recipient. "It would have booked the table" is unverifiable. "It would have submitted a reservation for 2 at 19:30 on the 18th, venue id 4412, under the name on file, no deposit" is diffable.
- Expected post-state — what the agent believes will be true after the effect, so you can later check whether its understanding was right even when the action was fine.
- Why — one or two lines pointing at the input that drove the decision. This is what makes a wrong diff diagnosable instead of merely wrong.
Then the human does the task the way they were going to do it anyway, and you compare.
The diff is field-level, not text-level
The first version of this compared rendered text and produced mostly noise. The agent wrote "Thu 18 Dec, 7:30 pm"; the human wrote "18/12 19:30". Same booking, red diff. Two weeks of that and nobody reads the diffs, which is the real failure mode of this whole approach: an unread diff is worse than no diff, because it manufactures the appearance of oversight.
Normalise both sides into the action object first, then compare field by field. Once you do that, diffs fall into four classes and only two of them need a human:
| Class | Example | Who looks |
|---|---|---|
| Identical | same fields, same values | nobody; count it |
| Cosmetic | same effect, different formatting, different wording of a note | nobody; count it, and fix the normaliser if it recurs |
| Substantive | a different-but-defensible choice: another time slot the user would also accept, a substitute item | sampled review; these are where your spec is ambiguous |
| Unsafe | wrong recipient, wrong amount, a commitment nobody asked for, data sent somewhere it should not go | every single one, same day |
Substantive diffs are the useful ones while you are still writing the task definition, because most of them mean the spec was under-specified rather than the model being wrong. In my experience a cluster of identical substantive diffs is a missing rule, not a model deficiency — and adding that rule is cheaper and more durable than adding examples.
Unsafe diffs are the only ones that should ever block a promotion outright.
Measure the ceiling before you measure the agent
Agreement rate is a seductive, near-worthless metric on its own. If the human does the task differently each time — picks whichever slot is free, varies the note, sometimes skips the step entirely — agreement is capped by human variance, and you will read that cap as agent error.
So measure the human baseline first: two people do the same twenty tasks independently and you diff their outputs. If two experienced humans agree on 92% of decisions, then an agent at 88% is not "12% wrong". It is inside the noise of the task. And if the two humans agree only 70%, you have learned something more important: the task has no definition yet, and no amount of model quality will fix that. Write the definition first.
With the ceiling known, the numbers worth watching are small in number:
- Unsafe-diff count. Target zero. One is a bug report; three is a stop.
- Substantive-diff rate, compared against the human-human baseline rate rather than against zero.
- Cosmetic-diff rate — a health metric for your normaliser, not for the agent.
- Coverage of proposals — the share of real tasks where the agent produced a proposal at all. Shadow mode is precise about what it proposes and blind to what it never considered, which brings us to the next trap.
Shadow mode measures precision, not recall
A shadow run can only be diffed when it produced a proposal. The dangerous case is the day the human did five tasks and the agent proposed none — because it did not notice them — and every diff for that day reads clean.
So add a second, cheaper audit: the would-have-missed sample. Take days where the human acted, list the actions the agent did not propose, and read three of them by hand. Ask one question: did the agent not see the trigger, or did it see it and decide it was nothing? The first is a monitoring gap (the agent is watching the wrong surface); the second is a judgement gap (its definition of "needs action" is wrong). They need different fixes and they look identical in a log where nothing was written.
Do this on a fixed schedule, not when you remember. Once a week, ten minutes, three samples.
The promotion gate, written before you need it
Write the gate down before the first shadow week, because afterwards every number will look good enough. Mine ended up like this:
- At least 30 proposals on the same task type, spread over at least 14 days — a burst of 30 in one afternoon is one situation, not thirty.
- Zero unsafe diffs in the second half of the window.
- Substantive-diff rate at or below the human-human baseline.
- One manual audit of ten randomly chosen proposals from the whole window, done by someone who did not build it.
- A written stop condition: what the agent does when it is unsure, and how fast a human can turn it off.
When it passes, do not jump to unattended. Promote to act, but announce: the effect happens, and the person gets a one-line message with the identifier and the undo path. That stage exists to catch the failures that only appear at the moment of commitment — double submits, stale confirmations, the second run of something that already succeeded — and it is also where you find out whether your undo path actually works.
After that, keep a canary reviewer: a random 5% of actions are surfaced for a human spot check, chosen at random by the system rather than by the agent, and never announced in advance. Randomness is the entire point; an agent that can tell which actions will be reviewed is not being reviewed.
What shadow mode costs, and when to stop
Three costs, all real:
- Duplicate work. The human does the task anyway, so for the duration of the shadow window you pay twice. Keep the window proportional to the risk: hours for a task that cancels a trial, weeks for a task that moves money.
- Rot. The world changes: the form gets a new field, the carrier changes its status strings, the venue changes its booking window. A shadow run from six weeks ago is not evidence about today. Give every shadow window an expiry and re-run it, or at least a monthly spot re-run, after any interface change.
- False comfort. A large diff history invites the conclusion that the agent is safe because nothing bad has been recorded. That only holds if the recording is complete and someone read the substantive class. Track reviewed-versus-total; if reviewed is a small fraction, you have a log, not a gate.
Failure modes I hit
| Symptom | Actual cause | Fix |
|---|---|---|
| Diffs mostly cosmetic | comparing rendered text | normalise to the action object |
| Agreement capped low | human variance | measure human-human baseline first |
| Clean diffs, missed tasks | recall blind spot | would-have-missed sample weekly |
| Agent passed, then double-booked | shadow never saw a repeat submit | announce-stage + idempotency key |
| Nobody reads diffs after week two | volume, no triage | only unsafe and substantive go to a human |
| Shadow verdict went stale | interface changed | expiry + re-run after interface change |
Checklist
- Shadow executor's write methods raise; enforcement is structural, not prompted.
- Every proposal stores input digest, action object, expected post-state, and a one-line reason.
- Diff is computed on normalised fields, with a cosmetic class that feeds normaliser fixes.
- Human-human baseline measured on at least 20 tasks before judging the agent.
- Unsafe diffs reviewed same day, every one.
- Would-have-missed sample run weekly on days with human actions and no proposals.
- Promotion gate written down before the window starts, with counts and dates.
- First promoted stage announces each action with an identifier and an undo path.
- Canary review is random, system-chosen, and unannounced.
- Shadow windows carry an expiry date and a re-run trigger.
None of this is about the model getting better. It is about a truthful record of what the agent would have done, compared at the level of fields rather than prose, against a baseline that includes the human's own inconsistency. An agent earns unattended time the way a junior colleague does — by a run of decisions someone actually read.
Disclosure: I am a user of Pickle, a phone-side AI assistant app that runs scheduled actions of the kind described above. I am not its vendor. The links below are mine: a page on its daily limits and reset tickets, a comparison of where conversational, browser and phone-side assistants stop, and an invite link that gives both sides reset tickets if a sign-up completes through it. Nothing in this post depends on them — the shadow-mode and diff-gate design stands on its own.
- Daily usage limits and reset tickets, explained from the app's own notices
- Where each kind of AI assistant stops — conversational, browser and phone-side, compared
- Invite link: https://pickle.com/i/RDKXK77G9ZRT — limited slots per link, first come; the app's own notices govern any benefit.
