Handing the mouse to a human: live takeover of an AI agent's headless browser

Our AI marketing analyst (we call it Co-Worker) does a lot of its research in a real browser. One of its favorite tools is render_page. It drives headless Chromium through Playwright, and the target is often the Meta Ad Library, where it looks at what competitors are running.

Meta doesn't love anonymous datacenter traffic. Depending on the day, a logged-out headless browser gets one of these:

• a full login wall ("You must log in to continue"),
• a page that looks healthy, except half the ad creatives are replaced with "confirm your age by logging in" placeholders,
• a checkpoint or two-factor screen.

We had three options:

1. Ask users for their Facebook password and type it in for them. Absolutely not.
2. Give up and return thin results. That's technically correct, but the result is useless.
3. Let the human drive for a minute. Pause the agent, stream the live browser to the user's screen, let them log in themselves, then hand control back to the agent.

We built option 3. This post walks through how it works: Chrome DevTools Protocol (CDP) screencasting, a two-hop WebSocket proxy, single-use tickets, and a handful of browser quirks we hit along the way.

The shape of the system

 ┌─────────────┐ SSE (chat stream) ┌──────────┐ MCP tool call ┌───────────┐ HTTP ┌──────────────┐ │ React app │ <───────────────────── │ Backend │ ─────────────────> │ MCP server│ ───────> │ page-render │ │ (overlay) │ │ (Express)│ └───────────┘ │ (Playwright) │ │ │ ══ WebSocket ════════> │ proxy │ ══ WebSocket (service key) ═════════════> │ CDP session │ └─────────────┘ cookie + ticket └──────────┘ └──────────────┘ 

• The LLM agent (Gemini) calls an MCP tool, which calls our internal page-render service. That service owns the Chromium instance.
• When page-render hits a blocker, it pauses inside the tool call. The HTTP request stays open, so from the agent's point of view render_page is simply taking a while.
• The backend notices the pause and pushes an event down the user's existing chat SSE stream.
• The frontend opens a WebSocket. The backend authenticates it and pipes it through to page-render, which streams JPEG frames out and takes mouse and keyboard events in.

The important property is that pixels and keystrokes never pass through the LLM. Gemini only ever sees the final render result, as if the page had been logged in all along.

Step 1: Detect the blocker (heuristics, layered)

Meta changes its markup often, so detection uses several weak signals instead of one strong one:

// 1. URL looks like a login/checkpoint path on facebook.com const LOGIN_URL_HINTS = [/\/login/i, /\/checkpoint/i, /\/recover/i, /\/two[_-]?factor/i]; // 2. The classic #email + #pass inputs are both visible // 3. Login copy in the body, *unless* the page also has Ad Library chrome // (Ad Library footers mention "log in" too: false-positive city) 

The age gate is the sneaky one, because the page renders fine. We count placeholders against real ad tiles after results load:

const gated = countMatches(bodyText, [/confirm\s+your\s+age/gi, /you\s+need\s+to\s+log\s+in\s+to\s+see\s+this\s+content/gi]); const total = countMatches(bodyText, [/library\s+id/gi]); // one per ad tile // Only interrupt the human when it actually matters const blocksResults = structuredAdCount === 0 || gated >= Math.ceil(total / 2); 

If a few creatives are age-gated, we just continue. If most of them are, the results aren't worth analyzing, so we ask for help.

Step 2: Pause inside a running tool call

This is the part that makes it feel natural to the agent. Page-render doesn't fail and ask for a retry. It awaits a promise that only resolves when the human is done:

export async function pauseForInteractiveControl(params): Promise<InteractiveReleaseReason> { let resolveRelease!: (reason: InteractiveReleaseReason) => void; const waitForRelease = new Promise<InteractiveReleaseReason>((r) => (resolveRelease = r)); sessions.set(params.sessionId, { ...params, waitForRelease, resolveRelease, cdp: null }); await publishStatus(session, "paused"); // -> Redis const timer = setTimeout(() => finish(params.sessionId, "timeout"), interactiveTimeoutMs); const reason = await waitForRelease; // "release" | "abort" | "timeout" | "unavailable" clearTimeout(timer); return reason; } 

The rest of the scenario code reads like a straight line:

const outcome = await pauseForInteractiveControl({ page, blockerKind: "facebook_login", ... }); if (outcome === "abort") return { code: "navigation_failed", message: "Facebook login was cancelled." }; if (outcome === "timeout") return { code: "timeout", message: "Facebook login timed out waiting for you." }; // Logged in: reload the exact results URL we captured *before* the login detour scenarioResult = await runMetaScenario(selectedPageId, adResultsUrlBeforeLogin); 

Two details made this robust:

• Timeout budgets stack. The interactive timeout (5 minutes by default) is added to the MCP and HTTP client timeouts upstream. Otherwise some layer above would give up while the user is still typing their 2FA code.
• Finalization is claimed synchronously. Timeout, socket close, page close, the "Done" button and auto-detection can all race to end a session. The first line of finish() deletes the session from the map before any await, so only one of them wins:

async function finishInteractiveSession(sessionId, reason) { const session = sessions.get(sessionId); if (!session) return; sessions.delete(sessionId); // claim before awaiting anything // ...stop screencast, detach CDP, resolve the promise, publish status } 

Step 3: Tell the user's browser, with a single-use ticket

Page-render writes { status: "paused", reason, viewport } to Redis under the trace id. While the tool call is in flight, the backend polls that key every ~750 ms. On the first paused status it mints a ticket:

const ticket = randomUUID(); await redis.set(`page-render:interactive:ticket:${ticket}`, JSON.stringify({ workspaceId, userId, traceId }), "EX", 360); 

The ticket and the pause reason go down the chat's existing SSE stream, and the React app renders a modal: "You have control of the browser."

We send a heartbeat every 30 s while the session stays paused, so a client that reconnects to the stream still gets a usable ticket.

Step 4: The WebSocket hop (where the security lives)

The frontend connects like this:

new WebSocket(url, ["page-render-interactive", `ticket.${ticket}`]); 

The ticket travels in Sec-WebSocket-Protocol instead of the query string. It's a small thing, but query strings end up in access logs and proxy traces, and protocol headers usually don't.

On upgrade, the backend runs every check before it accepts the socket:

1. Origin is on the allow-list. Browsers don't apply CORS to WebSockets, so cross-site WebSocket hijacking is a real risk.
2. The session cookie resolves to a user. We run express-session against the raw upgrade request.
3. The user has access to the workspace in the path.
4. The ticket matches the user, the workspace and the trace id.

Only then does it open the upstream socket to page-render, authenticated with an internal service key. Then it consumes the ticket atomically:

local value = redis.call('GET', KEYS[1]); if value then redis.call('DEL', KEYS[1]); end; return value 

A plain GET followed by a DEL would let two tabs (or a replayed request) both pass validation. With the Lua script, only one controller ever gets the browser. Messages the client sends before the ticket is consumed are buffered (capped at 64) and flushed once the ticket is consumed, so a fast first click isn't lost.

Page-render enforces the same thing on its side: one controller per session, and a second socket gets close code 4409.

Step 5: Streaming pixels with CDP screencast

Playwright exposes raw CDP sessions, and Chromium has a built-in screencast:

const cdp = await page.context().newCDPSession(page); cdp.on("Page.screencastFrame", ({ data, metadata, sessionId }) => { if (ws.bufferedAmount > 4 * 1024 * 1024) { // Client is behind: drop the frame, but still ack so Chromium keeps going void cdp.send("Page.screencastFrameAck", { sessionId }); return; } ws.send(JSON.stringify({ type: "frame", data, metadata }), () => { // Ack only after the frame is flushed: natural backpressure void cdp.send("Page.screencastFrameAck", { sessionId }); }); }); await cdp.send("Page.startScreencast", { format: "jpeg", quality: 55, maxWidth: Math.min(1920, viewportWidth), maxHeight: Math.min(1080, viewportHeight), }); 

The ack is the flow control. Chromium doesn't send the next frame until you ack the previous one. If you ack after ws.send completes, a slow client automatically gets fewer frames. If the socket buffer is already huge, we drop the frame and still ack, because we'd rather show a fresh frame than a backlog of old ones.

The backend proxy drops frames the same way when the client socket backs up. It only drops messages that start with {"type":"frame". Control messages like ended or dialog are never dropped.

On the frontend, frames are drawn to a <canvas>, with a sequence counter so a slow-decoding older frame can't overwrite a newer one:

const seq = ++frameSeq.current; const img = new Image(); img.onload = () => { if (seq !== frameSeq.current) return; // a newer frame already arrived canvas.width = img.width; canvas.height = img.height; canvas.getContext("2d")?.drawImage(img, 0, 0); }; img.src = `data:image/jpeg;base64,${data}`; 

Quality 55 JPEG at viewport resolution was the sweet spot for us. It's sharp enough to read a login form and a 2FA prompt, and small enough to stay smooth.

Step 6: Sending input back (and all the small gotchas)

The client sends coordinates relative to the canvas as displayed, plus the displayed canvas size. Page-render scales them into viewport space:

const x = (clientX / canvasWidth) * viewportWidth; const y = (clientY / canvasHeight) * viewportHeight; 

That lets the overlay switch between a modal, full screen and a resized window without breaking clicks.

Here are the things that bit us:

Input reordering. Socket messages arrive faster than Playwright dispatches them. Without a queue, a mousemove can overtake the mousedown it belongs to, and drags and text selection break. We fixed it by chaining every input onto a per-session promise:

const dispatch = session.inputQueue.then(() => dispatchInteractiveInput(session, message)); session.inputQueue = dispatch.then(() => undefined, () => undefined); 

Double-click. We send clickCount on mousedown, which we compute on the client: same spot (< 6 px) within 400 ms counts as a repeat, capped at 3 for triple-click.

Scroll. React registers wheel listeners as passive, so preventDefault() is silently ignored and the host page scrolls instead of the remote one. We attach a native listener with { passive: false }.

Key names. KeyboardEvent.key mostly matches Playwright's key names, with exceptions: " " → Space, Esc → Escape, OS → Meta, the legacy Left/Up arrow names, and Dead/Unidentified get dropped.

Clipboard. The remote browser has its own clipboard, so forwarding Ctrl+V does nothing useful.

• Paste: we let the local browser fire its paste event, read the text, and send { type: "text" }. Page-render then calls keyboard.insertText(), capped at 10k chars.
• Copy: we ask page-render for window.getSelection(). Clipboard writes must start during the user gesture, but the text arrives later over the socket. The fix is to pass a promise into ClipboardItem synchronously and resolve it when the reply lands:

const text = new Promise<string>((resolve) => (pendingCopy.current = resolve)); navigator.clipboard.write([new ClipboardItem({ "text/plain": text.then((t) => new Blob([t], { type: "text/plain" })) })]); send({ type: "copy" }); 

Native dialogs. alert/confirm/prompt block the page, and headless Chromium has no UI to show them. We catch Playwright's dialog event, forward it as a message, and render a small in-app dialog. The page-side handler awaits a gate promise until the user answers.

Popups. The screencast only covers one tab. If a "Log in" button opens a popup, we read its URL, close it, and navigate the streamed tab there. Otherwise the user would be looking at a tab that no longer changes.

Navigation guard. The user is now driving a browser that runs inside our cluster. Every framenavigated event is checked by the same public-URL validator render_page uses for SSRF protection, and a blocked navigation is sent back with goBack(). Without this, someone could type an internal service address into the remote browser.

Step 7: Know when they're done, without making them click "Done"

People forget to click "Done". So while the session is paused we poll for success:

// Facebook only issues c_user + xs after the *final* login step (incl. 2FA), // and a checkpoint URL means we're not there yet. async function detectFacebookLoginCompleted(page) { if (urlLooksLikeLogin(page.url())) return false; const cookies = await page.context().cookies("https://www.facebook.com"); return has(cookies, "c_user") && has(cookies, "xs"); } 

The probe needs two consecutive positive hits, about 1.5 s apart, before it auto-releases. Login flows redirect several times, and one lucky sample during a redirect chain isn't enough.

After a verified login, we save context.storageState() to a per-user, per-workspace vault in Redis with a sliding 7-day TTL. That user's next render_page call starts logged in, and they don't see the overlay again until the session expires.

The security model, in one table

Two rules we had to make explicit in code:

• Never expose someone else's session. We also keep an operator-managed, platform-wide Facebook session as a fallback. If a render using that session hits a login wall, we clear all cookies and storage and navigate to a clean login page before offering control, so the customer never gets an operator's logged-in browser. That logic lives in a tiny policy function with its own tests.
• Scope interactivity narrowly. It's only enabled for the scenario that needs it, only with a trace id, workspace and user attached, and behind a feature flag.

What we'd do differently / known limits

• One live session per page-render process. A paused browser holds real memory and a CPU share, and right now page-render runs as a single replica, so the WebSocket always lands on the pod that owns the browser. Scaling out means routing the socket by trace id, for example with a sticky hash or by recording the pod id next to the Redis status.
• Polling Redis every 750 ms for the pause status is simple and good enough for one event per render. Redis pub/sub would cut the latency, but polling works across restarts without extra plumbing.
• JPEG over JSON over WebSocket is not the most efficient encoding. Binary frames would save about 33% on base64. JSON was easier to debug, and the bandwidth is small at quality 55.

Takeaways

If your agent uses a real browser, it will eventually hit something only a human should handle: logins, CAPTCHAs, consent screens, "are you sure?" dialogs. You don't need a commercial remote-browser product to get a solid handoff:

• Pause inside the tool call with a promise instead of failing and retrying. The agent's control flow stays simple.
• CDP Page.startScreencast with ack-after-send gives you streaming with built-in backpressure.
• Single-use tickets consumed atomically, plus cookie auth, origin checks and access checks on upgrade, keep the most sensitive socket in your system locked down.
• Keep the human channel separate from the model channel. The LLM gets results, not pixels or passwords.

We're building ClickClarity.ai, an AI analyst for marketing teams that works across Google Ads, Meta, GA4, Shopify and more. If you've built human-in-the-loop handoffs for browser agents, I'd love to hear how you handled it in the comments.

Rodion Shlomo Solomonyk
CTO at Click Clarity AI