Every agent demo ends the same way: the cursor moves, the form fills, the confirmation page appears, and the audience claps. The interesting engineering starts one step later — at the first screen the agent is not allowed to pass.

That screen is not a bug to route around. A one-time code, a passkey, a CAPTCHA, a 3-D Secure challenge, an identity check: each exists so a person is present. An agent that "gets past" them is either breaking someone's terms or quietly convincing you it did something it didn't. The part you actually ship is the handover — the moment your agent stops, says exactly where, and hands back a task a human can finish in ten seconds.

This post is about designing that moment. It is the least glamorous part of agent engineering and the one your users will hit most often.

1. Name the wall before you write the loop

Agents that fail badly in production almost always fail at classification. They see "something went wrong", retry, and leave the world in a state nobody can describe. Start by making the wall a first-class value:

Wall What the agent sees Correct handover Never
One-time code A 6-digit input, "code sent to ••••" Ask the user for the code, or pause the task and keep the input focused Read the user's SMS/mail silently; store the code
Passkey / WebAuthn navigator.credentials prompt, no DOM target Stop; the device holds the key, not your session Prompt-inject the user into tapping "allow"
CAPTCHA / bot check iframe, undisclosed score, or a 403 with no puzzle Stop and flag for human completion Solve it, buy a solver, or retry the same request
Identity / KYC Document upload, certificate popup, image-rendered keyboard Stop before the step; explain what document is needed Screen-record or OCR someone's ID
Payment auth 3-D Secure iframe, in-app card sheet, biometric Stop at the last irreversible click; show the amount Re-submit a declined card
Irreversible action A submit that has no undo Require an explicit confirmation, once Retry on timeout without checking state

Two of those rows matter more than the rest. Bot checks because retrying is itself the signal — a request pattern that repeats after a refusal is what a macro looks like, and it can cost the account, not just the run. Irreversible actions because "did it go through?" is a question your agent must answer from state, not from whether an HTTP call returned.

2. Model the stop as data, not an exception

If your handover is a throw, the reason is lost by the time it reaches the UI, and the user gets "Something went wrong. Try again." Model it as an outcome, and every layer above it can render, resume, and audit:

type StopReason = | { kind: "otp_required"; channel: "sms" | "email" | "authenticator"; field: string } | { kind: "passkey_required"; origin: string } | { kind: "captcha"; provider?: string; pageUrl: string } | { kind: "identity_check"; document: "id" | "bank_cert" | "unknown" } | { kind: "payment_auth"; amount: Money; merchant: string } | { kind: "ambiguous_state"; detail: string }; // could not confirm what happened interface Handoff { taskId: string; step: number; // where in the plan we are reason: StopReason; committed: CommittedState; // what is already true in the world evidence: EvidenceRef[]; // what we did, in order resume: ResumeToken; // how a human continues, and how we continue after expiresAt: string; // a stale handover must not act on old prices }

The committed field is the one people skip and the one users need. "Stopped at verification" is not enough information when the cart is filled, the slot is reserved, or the first of two steps already ran. A handover that doesn't say what is already committed turns a five-second finish into an audit.

ambiguous_state deserves its own kind. Timeout after submit is the classic case: the action may have landed. Treat it as a stop, not a retry.

3. Make the resume idempotent

A handover ends when the human does their part. The dangerous assumption is that the world is unchanged while they were away.

Three rules cover most cases:

  1. Re-read before you act. On resume, verify the page still shows the expected step, and that price, slot or quantity still match what you told the user. If they don't, stop again and show the diff — never "continue" into a changed total.
  2. One task, one commit. Keep a taskId and, where the site allows it, an idempotency key on the submitted payload. Where it doesn't, check for a confirmation artifact (order number, "already booked" state) before submitting again.
  3. Never resume into a second commit. If the resume reveals the action already happened, mark the task completed with the evidence, not failed. 
async function resume(handoff: Handoff, human: HumanResult) { const now = await observe(page); if (!sameStep(now, handoff.step)) return stop(handoff, "ambiguous_state", now); const drift = diffTerms(handoff.committed, now); // price, date, quantity, recipient if (drift.length) return stop(handoff, "terms_changed", drift); const already = await findConfirmation(now); // order id, "already submitted" if (already) return complete(handoff, already); return act(now, human); }

That is the whole trick, and it fits in a screen. The engineering value isn't in the model call; it's in refusing to act twice on a stale plan.

4. Record evidence that survives a complaint

A step log with these four fields per action answers almost every "what did it do to my account?" question:

{ step, ts, action: "click" | "type" | "navigate", target: "<stable selector or role+name>", before: "<hash of normalized DOM or accessibility tree>", after: "<hash after settle>", screenshot: "<blob ref>", result: "changed" | "unchanged" | "unknown" }

before/after are how you detect the silent failure that screenshots hide: the click landed, the DOM didn't change, the agent moved on. In practice most "the agent did nothing" reports are result: "unchanged" rows the agent didn't check.

Two constraints on the log itself:

  • Redact by field type, not by regex guessing. Never persist OTPs, card numbers, passwords or document images. Keep a reference ("otp provided by user at 14:02") rather than the value.
  • Keep it append-only and short-lived. It is diagnostic evidence, not a browsing history to mine.

5. The handover UI is a product surface

The screen a user gets at a wall should answer four questions without them reading a paragraph:

  1. Where are we? "Checking out on example.com — payment confirmation."
  2. What is already done? "Cart committed: 1 × ₩39,000. No charge yet."
  3. What do I do? The exact control, focused and scrolled to, with the page state preserved.
  4. What happens next? "After you approve, I'll confirm the order and stop — I won't change anything else."

Preserving the live page state is worth real engineering effort. A handover that drops the user at a login screen after it spent 40 seconds filling a form is worse than not having an agent at all.

And when the human finishes, the agent should verify rather than assume: read the confirmation, close the task with the artifact, and log the transition. That verification is what lets you say "completed" instead of "probably completed".

6. What not to build

  • A solver in the loop. Buying CAPTCHA solving or driving passkey prompts on the user's behalf converts a legal gray area into an account risk, and the risk lands on the user's identity, not yours.
  • Silent retries. After a bot-check refusal, repeating the same request is the pattern that gets an IP or account banned. Change route or stop.
  • Credential handoff through the chat. Ask for the code at the step that needs it, never the password. If the site supports passkeys, prefer them and never touch the key.
  • "It worked" without an artifact. Confirmation numbers, order rows, and state diffs are the only honest completion signal. Model confidence is not one.
  • A generic error. "Something went wrong" guarantees a support ticket. A typed stop reason often is the fix.

7. A test matrix worth keeping

Run these against any agent before you trust it with a deadline:

Test What it proves Pass condition
Form with silent validation failure Does it verify state? Reports unchanged, doesn't claim success
Booking an already-taken slot Does it respect constraints? Stops and asks, never books a substitute
Session expires mid-flow Does it resume safely? Re-authenticates, re-reads terms, no duplicate commit
Price changes before payment Does it guard money? Shows the diff, waits for a yes
Refusal/Bot-check Does it stop cleanly? One stop, no repeated identical requests
Timeout after submit Does it detect ambiguity? ambiguous_state, then artifact check

Six tests, one afternoon. They will tell you more about an agent than any benchmark, because they measure the thing users actually experience: what happens when the happy path ends.

The closing thought

Agent quality used to be judged by how far it got before a human had to step in. That framing is backwards now that these tools touch real accounts and real money. The metric worth optimising is how well it hands back — named wall, committed state, verifiable evidence, no duplicate action. An agent that stops well is one you can leave running. One that never stops is the reason people turn it off after a week.

Disclosure: I'm a user of Pickle, a Korean phone-side AI agent app, and the links below are mine — an invite link that gives both sides reset tickets for the app's daily usage limit if a sign-up completes through it, and a comparison page I wrote on where conversational, browser and phone-side assistants stop. Nothing in this post depends on it: read it, skip the link, or take only the design notes.