Headless Chrome is a process, not a function: lifecycle, memory and failure modes in browser automation

The run died at minute forty with ENOMEM. The task it had been doing was unremarkable — open a page, read a table, fill a form, close. What was remarkable was ps:

$ ps -eo pid,ppid,rss,etime,comm | grep -c headless_shell 37 $ du -sh /tmp/.org.chromium.Chromium.* | tail -3 612M 

Thirty-seven browsers, none of them mine anymore, and four hundred gigabytes of profile directories spread across a disk that had sixty. Somewhere in my agent loop, a finally that was supposed to close a browser had been skipped by a task-level timeout, the parent dove off, and the children kept running with nobody to reap them. The next run launched more. The failure looked like a memory bug in my code. It was a process-lifecycle bug in my environment.

That is what this post is about. Not "how to automate a browser" — there are ten thousand tutorials for page.click. This is about the engineering that decides whether a browser automation loop survives a week in production: what the browser's object model actually costs, where the leaks are, why your navigations never "finish", what a selector really guarantees after a re-render, what a trace file buys you at 3 a.m., and which failures you are allowed to retry and which ones will double-charge someone.

Everything below is from an assistant that operates a browser on someone's behalf: filling forms, submitting paperwork, reading account pages, cancelling things. It runs in containers, it is limited to a few hundred megabytes of RAM per worker, and it fails in public.

Three objects, three different costs

Headless Chrome is usually introduced as a function you call. It is a process tree with a handle, and Playwright's object model mirrors that tree. Get the costs right and the rest of the design follows.

Two consequences are easy to miss:

Launching a browser per task is the expensive way to get isolation, and the cheap way to run out of processes. Contexts give you cookie isolation, storage isolation, permission isolation and a separate storage_state for essentially nothing. What a context does not give you is process isolation: a renderer crash takes its page with it, and a pathological page can starve its siblings. So the usual shape is one browser per worker, one context per task, one page per document the task actually needs — and a fresh browser every N tasks to shed accumulated renderer baggage.

The context is your security boundary, so make it one. If your assistant ever acts for two different people, or for one person against two different services, the identity lives in the context. Cookies do not leak upward from a context to a browser — they leak across pages inside one context. Any design that reuses a single context "to save time" is a design where a stale session from task A is the session task B acts with. storage_state is a credential file; treat it exactly like the password it stands for.

The lifecycle must not depend on your code finishing

Here is the shape I use in Python. It is longer than async with async_playwright(), and every extra line is a failure I have already had.

import asyncio, os, signal, shutil, tempfile from contextlib import asynccontextmanager from playwright.async_api import async_playwright @asynccontextmanager async def browser_session(): profile = tempfile.mkdtemp(prefix="pw-profile-") pw = await async_playwright().start() browser = None try: browser = await pw.chromium.launch( headless=True, args=[ "--disable-dev-shm-usage", # /dev/shm is 64MB in most containers "--disable-gpu", "--no-first-run", "--no-default-browser-check", f"--user-data-dir={profile}", ], ) yield browser finally: if browser is not None: try: await asyncio.wait_for(browser.close(), timeout=10) except Exception: pass await pw.stop() shutil.rmtree(profile, ignore_errors=True) 

Two notes on that, because both are load-bearing.

The browser.close() is wrapped in a timeout because close can hang. A page in a beforeunload loop, a renderer that has stopped answering, a debugging socket that will not drain: any of them can make a graceful close wait forever. A close that never returns is worse than a kill, because it holds the finally block open and every resource behind it.

The second note is the one that actually fixes the orphans, and it is outside this function. Playwright launches Chromium as a child of its driver process, and the renderer, GPU and utility processes are children of that. If your Python process is killed by a task-level timeout — the supervisor's SIGKILL, not yours — none of your finally blocks run at all, and the whole tree is orphaned. So the real fix has two layers: your code cleans up on its own deadline, and whatever supervises your code must kill a process group, not a process. Start each task with start_new_session=True (Python's subprocess) or in its own container, and kill the group on timeout.

When you need the browser in a group you control explicitly — the case I ended up in — launch Chromium yourself and attach to it. This is public API and it gives you the PID:

proc = subprocess.Popen( ["/usr/bin/chromium", "--headless=new", "--remote-debugging-port=0", f"--user-data-dir={profile}", "--disable-dev-shm-usage"], start_new_session=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, ) endpoint = read_ws_endpoint(proc) # parse "DevTools listening on ws://..." browser = await pw.chromium.connect_over_cdp(endpoint) ... os.killpg(os.getpgid(proc.pid), signal.SIGKILL) # reaps the entire tree 

connect_over_cdp is Chromium-only, and it means Playwright does not own the launch flags or the browser version — a trade worth making exactly when lifecycle control is your problem. Either way, the difference between "one run leaked" and "the host is out of file descriptors by Tuesday" is whether the thing that kills your process knows about its children.

Also worth stating plainly: --no-sandbox is what you need when Chromium runs as root inside a container, and it is a real reduction in isolation. If you can run the browser as a non-root user, do; if you cannot, understand that the flag is why a compromised renderer is a compromised container. --single-process looks like a memory saving and is not: it collapses the site-isolation boundary and turns "a tab crashed" into "the browser did".

And clean the profile directories. They are 20–60 MB each and they are created under /tmp, which on many images is memory-backed. That is the second half of my ENOMEM: not RAM the pages used, but RAM the temporary profiles occupied and never gave back.

A page that never loads, and why "networkidle" is a trap

waitUntil looks like a knob and is actually a question about the document, not about the network:

• commit — the response has started arriving. The document exists; almost nothing in it does.
• domcontentloaded — the HTML is parsed. Scripts that run on parse have run. Subresources may still be in flight.
• load — window.onload fired: images, stylesheets, iframes loaded. This is the default, and on a heavy page it can be the slowest of the four.
• networkidle — no network connections for 500 ms. Playwright's own documentation discourages it, and the reason is not style.

networkidle is a guess about a network you do not control. A page with a 30-second heartbeat (setInterval(fetch, 30000)), a websocket, a long-polling support widget, or a "someone is viewing this" beacon will never be idle — so your navigation hangs until the timeout and fails a page that was usable twelve seconds ago. The opposite also happens: a page that is idle for 500 ms while a client-side router then fetches the actual content, so "idle" is reached before the data is.

The reliable pattern is: commit or domcontentloaded, then an explicit readiness condition on the DOM, plus a wall clock. And cut the network you do not need, which both speeds the load and makes "quiet" meaningful:

BLOCK = ("image", "media", "font") async def quiet_network(context): async def handler(route, request): if request.resource_type in BLOCK and "/api/" not in request.url: await route.abort() else: await route.continue_() await context.route("**/*", handler) 

Keep that list conservative. Aborting script or stylesheet changes what the page is — lazy content often loads via script, and a layout that depends on a font metric can move the button you were about to click. Aborting image is usually safe for reading a table; aborting it on a page whose print layout you need is not.

Readiness is a predicate, not an event

The most useful function in my stack is about thirty lines and has saved me more debugging than any other:

async def stable(page, selector: str, quiet_ms: int = 700, timeout_ms: int = 15000): """Return when the element's inner HTML has not changed for quiet_ms.""" last, since = None, None deadline = asyncio.get_event_loop().time() + timeout_ms / 1000 while asyncio.get_event_loop().time() < deadline: try: html = await page.locator(selector).first.inner_html(timeout=2000) except Exception: await asyncio.sleep(0.2) continue now = asyncio.get_event_loop().time() if html != last: last, since = html, now elif now - since >= quiet_ms / 1000: return html await asyncio.sleep(0.2) raise TimeoutError(f"{selector} never settled") 

Why stability rather than presence: a table that has rendered its header and one row passes any "element exists" check, and then you read nine rows instead of two hundred. A list that is being sorted passes it twice. Requiring the same HTML across a window catches the difference between "rendered" and "finished rendering", which is the failure that produces plausible wrong data rather than an error.

Add one more guard for pages that are never quiet: anchor on a completion signal the site itself provides — a row count matching a total in the page, a "no more results" node, a status column that left a pending state. If the site gives you no such signal, that is a fact worth writing down, because the alternative is a timeout that looks like a bug forever.

Selectors: what a locator promises, and what it does not

A Locator is a query, not a handle. It re-resolves on every action, which is why it survives re-renders — and why the same selector can act on a different element two seconds later. That property is the whole design, and it forces a rule:

Never hold a reference across an await, and always scope the query to the record you mean.

Auto-waiting does more than retry. Before acting, Playwright waits for the element to be attached, visible, stable (not animating), enabled, editable where relevant, and to actually receive the event — meaning it is not covered by an overlay. That last check is what saves you from clicking a cookie banner's twin. When it fails, it says so; treat the message as information, not noise:

element is not visible element is outside of the viewport element is not stable element intercepts pointer events strict mode violation: locator resolved to 2 elements 

Ranking of selector strategies by how long they last, from my own churn:

1. Role + accessible name (get_by_role("button", name="Cancel")) — survives CSS refactors and layout rewrites, because if the name changes the site broke for screen readers too.
2. Label / placeholder for form fields (get_by_label("Card number")) — same reason.
3. A test id (get_by_test_id), when the site has them, because they are there on purpose.
4. Structural CSS (div:nth-child(3) > span), which is a promise the site never made.
5. Text — excellent for a button whose text is the meaning, terrible for a cell whose text is data.

The trap is not picking the "wrong" strategy. It is picking a strategy that matches two things, because then the same code either throws (strict mode violation) or, with .first, silently acts on the wrong one. Two "Cancel" buttons on one page is not an edge case; it is what every list view looks like. Scope it:

row = page.get_by_role("row").filter( has=page.get_by_role("cell", name="ORD-4417") ) await row.get_by_role("button", name="Cancel").click() 

And when you cannot express "the row for this order" as a locator, stop and reconsider the step. An automation that cannot name its target per-record is an automation that will eventually cancel someone else's booking.

Detachment is not an exception to catch. It is a signal about your code.

Sooner or later everything throws one of these:

Element is not attached to the DOM Execution context was destroyed, most likely because of a navigation Target page, context or browser has been closed 

The first two mean the node you resolved is gone, because the framework re-rendered the subtree, a virtual list recycled the row, or the document navigated. There is exactly one correct response: re-resolve the locator and act again, from a fresh read of the page. Not "retry the click with the old handle" — that is the same stale node. And not "catch and continue", because the action did not happen.

# WRONG: the handle is a snapshot of one moment handle = await page.query_selector("text=Submit") await page.wait_for_timeout(300) await handle.click() # Element is not attached to the DOM # RIGHT: the locator re-resolves each time it is used await page.wait_for_timeout(300) await page.get_by_role("button", name="Submit").click() 

Execution context was destroyed is the same fact at the document level: a navigation happened somewhere between your two calls, including a same-origin <form> submit you triggered. If it appears after a click, do not retry the click — check whether the click actually worked. This is the cheap version of a much larger problem: retrying an action whose effect you have not established is how a submit becomes two submissions.

Frames, shadow roots, and the places a document hides

An iframe is a document with its own origin, its own JS context, and — commonly — the thing that holds your control. A payment widget, a captcha, a date picker, an embedded form. Playwright's frame_locator waits for the frame to appear and scopes everything after it:

frame = page.frame_locator("iframe[title='Payment']") await frame.get_by_label("Card number").fill(card_number) 

Three things people get wrong:

• Matching the iframe by URL. Frame URLs change per session (tokens, tenant ids). Match by title, name, or position scoped under a container — and if there is exactly one frame, say so explicitly rather than using the first one you find.
• Nested frames. A card form inside a 3-D Secure flow inside the payment iframe is three documents. frame_locator composes, but your readiness predicate must run inside the innermost one.
• Shadow DOM. Playwright's CSS engine pierces open shadow roots, so a get_by_role inside a web component usually works. A closed shadow root is genuinely unreachable from script, by design. When a control is invisible for that reason, the order form is telling you it wants a human, and that is a handover, not a puzzle to solve.

The hard line I keep in code and in the docs of my own tool: no stealth patches, no user-agent or canvas or WebGL spoofing, no CAPTCHA solving, no "human typing" emulation. Every one of those is a decision about someone else's system made without their consent, and they also fail quietly in ways that cost more than they save: a solver that half-works turns a clear "stop and ask the user" into a corrupted submission. When the browser hits a bot check, a one-time code, a passkey prompt, or a 3-D Secure step, the correct action is to stop, persist the page state, and hand back an action a person can finish in ten seconds.

Dialogs, downloads, uploads, popups

These are the four places where a click does something outside the document, and each has a one-line mistake that costs an hour.

Dialogs. If no listener is attached, Playwright auto-dismisses alert, confirm and prompt. So a confirm("Delete this record?") returns false and the delete silently does not happen — you will swear the click worked, because it did. Register the handler before the click, and log every dialog with its message, because the message is the only place the page tells you what it just asked:

page.on("dialog", lambda d: (print("DIALOG:", d.type, d.message), asyncio.create_task(d.accept()))) 

Downloads. A click that starts a download produces no navigation and no DOM change, so an agent that waits for a page transition waits forever. Wrap the click:

async with page.expect_download() as info: await page.get_by_role("button", name="Download invoice").click() download = await info.value await download.save_as("/data/out/invoice.pdf") print(download.suggested_filename, await download.path()) 

Beware the variant that opens a PDF in a viewer tab instead of downloading: expect_popup then, and a PDF viewer is a plugin document with no useful DOM. If you need the bytes, take them from response.body() on the request that produced the document.

Uploads. set_input_files on the input[type=file] is reliable; clicking a "Choose file" button and driving an OS file picker is not available in headless mode at all. If the site hides the input, set_input_files still works on a hidden input — that is not a bypass, it is the normal path.

Popups. expect_popup and then work in the new page, remembering that the opener's context is what holds the cookies. Close popups you do not need; each is a renderer.

Record what the page said, or debug by guesswork

The single highest-value thing I added after the orphans incident is a ring buffer of the browser's own complaints, dumped into the failure record:

class PageLog: def __init__(self, page, limit=200): self.events, self.limit = [], limit page.on("console", lambda m: self.add("console", m.type, m.text)) page.on("pageerror", lambda e: self.add("pageerror", str(e))) page.on("requestfailed", lambda r: self.add("requestfailed", r.url, r.failure)) page.on("response", lambda r: r.status >= 400 and self.add("http", r.status, r.url)) page.on("crash", lambda p: self.add("crash", "page crashed")) def add(self, *parts): self.events.append(" ".join(str(p) for p in parts)[:400]) del self.events[:-self.limit] 

Two rules make it useful. Capture console messages of type error and warning only at first — a modern app logs hundreds of lines and the signal drowns. And capture response status ≥ 400, because the most common silent failure in browser automation is not a broken selector: it is a 401 or a 422 on the XHR your click triggered, in a page that rendered an optimistic UI and moved on.

Traces, HAR, and the artifact that answers "what did it see?"

When a run fails in a container at 3 a.m., the only question that matters is what the page looked like at the moment of failure. Console logs do not answer it. A trace does.

await context.tracing.start(screenshots=True, snapshots=True, sources=True) try: ...task... finally: path = f"/data/traces/{task_id}.zip" await context.tracing.stop(path=path) 

A trace is a timeline of DOM snapshots, screenshots, network, console and source, openable in a viewer, and it is the difference between "the button was not there" and "the button was there, disabled, behind a modal with a 3-second fade". It is also large — hundreds of kilobytes to tens of megabytes. So: trace the failure, not the run. Start tracing always, discard on success (stop() without a path), keep on failure. If your Playwright version supports chunking, keep the last few chunks instead of one long recording.

HAR is the complement for network shape: record_har_path with record_har_content="omit" gives you every request, status, timing and size in a portable file, without the bodies that make a HAR unusable (and without the customer data that makes it a liability). "embed" is for the case where a body is the evidence — and then it is a document you must store like one.

Screenshots are the last resort, and they have a specific use: the picture the user needs when you ask them to take over. A full-page screenshot at the moment of handover plus one sentence of what is needed beats any JSON.

Retry semantics: three classes, one table

Retries are where browser automation turns a flaky test into a duplicate charge. The classification I use is by observable state, not by exception type:

The third class is the one that matters. "Started, unknown" is not a retry problem, it is a read problem: navigate to the record list, find the row you were creating, and act on what is there. If the product gives you no way to read what happened, then the correct behaviour is to stop and report — a duplicate order you caused is worse than an order you did not place.

Two implementation notes that keep retries honest:

• Retry from a known state. A retry that resumes mid-flow reuses a page whose state you no longer trust. Go back to the URL you started from and run the step again, so the retry is the same experiment.
• Cap and count. Bounded attempts, exponential backoff with full jitter, and two metrics: retries and retries_exhausted. Retries above a couple of percent of tasks is a design smell — usually a readiness predicate that is wrong, or a selector that matches the wrong row.

Memory, concurrency, and the number that decides your worker size

Browser automation looks I/O-bound and is partly CPU-bound: each renderer parses, styles and lays out its own document, and a busy page can hold a core. So parallelism past your core count helps only for the parts that are waiting.

The memory question has an empirical answer, and you should measure it rather than trust anyone's table:

# RSS of everything under one worker, sampled while it works watch -n2 'ps -eo rss,comm | grep -E "headless_shell|chrome" | awk "{s+=\$1} END {print s/1024 \" MB\"}"' 

What I do with the number: pick a per-page ceiling from measurement (a light page and a heavy app differ by a lot, so measure your heaviest), leave 30–40% of the container above the sum of your concurrency, and enforce it with a semaphore rather than by trusting the scheduler.

sem = asyncio.Semaphore(4) # = allowed concurrent pages async def run_task(task): async with sem: async with browser_session() as browser: ctx = await browser.new_context(accept_downloads=True) try: await do(task, ctx) finally: await ctx.close() 

And recycle: a long-lived browser accumulates dead renderers, cached compiled code and fragmented heaps. Restart it every N tasks or after a memory threshold, whichever comes first. Restarting a browser is a second; restarting a container because the OOM killer chose your process is a page.

Boundaries, permissions, and the parts that are not engineering

Headless Chrome will do what you tell it, which is the problem. Four rules I would put in any automation that acts for someone:

One identity per context, and none of them borrowed. No shared storage_state across tasks, no "just use my cookies", no acting for person B with person A's session. The context is the identity; a leak here is a data leak, not a bug.

Rate and volume limits are part of design, not politeness. A loop that pages through a site's search results is a scraper. Enforce a per-host concurrency of one or two, sleep between pages, respect robots.txt and the site's terms, and never let a retry loop turn into a traffic spike — a hundred retries per minute is what a macro looks like to the site, and being blocked is the gentle outcome.

Log every navigation with what it was for. A URL list with timestamps, task ids and purposes is the only way to answer "did it go where I said it would" after the fact, including for the pages it visited on its own.

Decide the irreversible ones in advance. Which clicks in your flow cannot be undone, what the amount is before it goes out, and whether the person agreed to it. An assistant that discovers the total at the confirmation step and proceeds anyway has made a decision that was never its own. The engineering version of that rule: never retry the irreversible step without reading state first, and never let a timeout on it be treated as a failure.

What headless cannot do

Being honest about this saves weeks:

• It is not the browser you...