"How much AI traffic do we get?" sounds like one question, but it's really two. Some automated visitors only download your HTML: search and training crawlers, and the fetchers an assistant uses when a user asks about your page. Others drive a real browser: test scripts, scrapers, and the newer agents that click around a site on someone's behalf. The first kind shows up in one place, the second kind in another, and some of them look exactly like people.

I wanted to see this directly instead of guessing. So on 2026-11-20 I set up a one-page test site on my own machine, sent ten kinds of automated client to it three times each, and compared two records: the server log (every HTTP request) and the analytics beacon (a small JavaScript snippet that reports a page view, the way a JS analytics tracker does).

The setup

The test page was plain HTML with an inline script. On load, the script sent a beacon with navigator.sendBeacon() (MDN) containing the user agent, navigator.webdriver, the client-hint brands, screen size and focus state. It counted pointer, scroll, key and touch events, and sent a second beacon when the page was left. A 34-line Python server logged every request and every beacon.

The clients, all on Linux with Chrome 156:

  • A–D: AI bot user agents. I don't control when real crawlers visit, so I replayed the example user-agent strings that the vendors publish for GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI's crawler page) and PerplexityBot (Perplexity's crawler page), using curl or Python's urllib. That shows how your stack treats those strings. It doesn't prove how the real crawlers behave.
  • E: plain curl, default user agent.
  • F: Chrome's own headless mode (--headless=new --dump-dom), no automation library.
  • G and H: Playwright driving Chrome, headless and in a real window.
  • I and J: a normal Chrome window started with a debugging port and steered over the DevTools protocol, without the --enable-automation flag. J also sent 20 synthetic mouse moves and a scroll.

Each client visited three times. The scripts and raw logs are small, and nothing here depends on a particular analytics product.

What each visitor left behind

Client Server log (HTML requests) Analytics beacons User agent says navigator.webdriver
A: GPTBot UA (curl) 3 0 GPTBot n/a (no JS)
B: OAI-SearchBot UA (curl) 3 0 Chrome 131 on macOS + "OAI-SearchBot" n/a
C: ChatGPT-User UA (urllib) 3 0 ChatGPT-User n/a
D: PerplexityBot UA (curl) 3 0 PerplexityBot n/a
E: curl 3 0 curl/8.14.1 n/a
F: Chrome --headless 3 3 HeadlessChrome/156 false
G: Playwright, headless 3 3 (+3 leave) HeadlessChrome/156 true
H: Playwright, window 3 3 (+3 leave) Chrome/156 true
I: Chrome over DevTools 3 3 (+3 leave) Chrome/156 false
J: same + mouse moves 3 3 (+3 leave) Chrome/156 false

Three things stand out.

1. HTML-only fetchers never reach a JS tracker. Fifteen requests from A–E, fifteen lines in the server log, zero beacons. No script ran, so the analytics snippet never ran. If you've ever wondered why your analytics shows no GPTBot at all while your hosting bill shows plenty of bot traffic, this is the reason. Crawlers that do run JavaScript exist (some search engines render pages in a browser engine), so don't read this as "bots never appear". But a client that only downloads HTML is invisible to a JS tracker by design.

2. Headless Chrome is easy to spot, unless you only read client hints. F and G put HeadlessChrome/156 in the user agent. But the Sec-CH-UA header and navigator.userAgentData.brands said "Google Chrome";v="156" for all five browser clients, headless or not. A filter that relies on client hints alone would count headless Chrome as a normal Chrome visit.

3. A real Chrome window steered by a program looked like a person. I and J sent the same user agent, the same client hints and navigator.webdriver === false, just like a human in Chrome. navigator.webdriver is the flag that browsers set when they are under automation (MDN). Playwright set it (G, H). Steering Chrome over the DevTools protocol without the automation switch did not. And J's 20 synthetic mouse moves arrived as 20 ordinary pointer events, so an "engaged visitor" filter based on mouse movement would have counted it as engaged.

Two details I'd treat as noise from my setup rather than signals: the headed clients reported one pointer event even without any input (probably the cursor sitting over the new window), and I and J reported document.hasFocus() === false, likely because my virtual display had no window manager. The time on page told me more: the scripted visits left after 1.50–1.52 seconds every time, and J after 2.85 seconds. Real people aren't that consistent.

What user-agent libraries make of AI bots

Most analytics and log tools sort traffic with a user-agent parser. I ran the documented strings through two popular Python libraries, ua-parser 1.0.2 and user-agents 2.2.0:

String ua-parser (family / OS / device) user-agents is_bot
GPTBot GPTBot / Other / Spider True
OAI-SearchBot OAI-SearchBot / Mac OS X / Mac False
ChatGPT-User "com/bot" / Other / Other False
PerplexityBot PerplexityBot / Other / Spider True
HeadlessChrome HeadlessChrome / Linux / Other False

OAI-SearchBot's documented string starts like an ordinary Chrome-on-Mac user agent and adds compatible; OAI-SearchBot/1.4 at the end. In these versions, one library called it a Mac and the other didn't flag it as a bot. ChatGPT-User, which OpenAI describes as a user-triggered fetcher rather than a crawler, wasn't flagged either. Rule lists get updated, so your version may behave differently. That's exactly why it's worth testing the strings you care about.

A practical way to keep the two kinds apart

Count crawlers and fetchers from the server side. Your access log (or your CDN's log) is the only place HTML-only fetchers appear. Match the bot tokens before you match browser names, because some bot strings contain "Chrome". A minimal classifier:

// Order matters: bot tokens first, because some bot UAs also contain "Chrome". const AI_TOKENS = ["GPTBot", "OAI-SearchBot", "ChatGPT-User", "ClaudeBot", "Claude-User", "Claude-SearchBot", "PerplexityBot", "Perplexity-User"]; function classify(ua = "") { const ai = AI_TOKENS.find((t) => ua.includes(t)); if (ai) return { kind: "ai-bot", name: ai }; if (/bot|crawler|spider|curl|wget|python-requests|HeadlessChrome/i.test(ua)) return { kind: "other-bot" }; return { kind: "browser" }; } 

Keep the token list in one place and check it against the vendors' pages now and then. Anthropic, for example, lists ClaudeBot, Claude-User and Claude-SearchBot and what each one is for in its help article. A user agent is only a claim, though. OpenAI and Perplexity also publish IP ranges for their bots, and our earlier piece on counting AI agents in your access logs walks through checking a request against them.

Tag automation in the tracker, but don't promise yourself it catches everything. In the browser, two cheap checks would have caught G and H (and F's user agent):

const automated = navigator.webdriver === true || /HeadlessChrome/.test(navigator.userAgent); // send it along with the page view, e.g. { ..., automated } 

They would not have caught I or J. Nothing in the user agent, the client hints or the webdriver flag set them apart. For that kind of traffic, look at patterns instead of single visits: identical visit lengths, a burst of visits from one network, many pages in a few seconds, entry pages nobody links to. Annotate the day you add or change a filter, so that a sudden drop in "visitors" isn't mistaken for a real one.

Segment, don't silently delete. If you throw away everything that looks automated, you can't check the filter later. Keep a "suspected automation" segment, or at least a count of what was excluded, and compare it with your human numbers when something looks odd.

Check what your bots are allowed to fetch. Classifying bots is half the job. The other half is deciding which ones may crawl what. The free FindUtils Bot Crawl Checker tells you whether your robots.txt allows or blocks a URL for 24 well-known search, AI and social bots and shows the rule that decides it. The robots.txt Tester lets you try paths against a draft robots.txt, in your browser, before you deploy it.

Where a privacy-friendly analytics tool fits

None of this needs cookies or personal data. User agent, a webdriver flag and a page path are enough to separate most automation, and the aggregate patterns above work on counts.

That's also how DataVisitors, the cookieless analytics product from the team behind FindUtils, describes its approach. Its web analytics page says bot and referrer-spam filtering "removes known unwanted activity before storage", that refusal counts show what the filters exclude, and, honestly, that filtering "cannot guarantee that every unwanted request is detected". The same page describes an AI assistants channel for visits that arrive from ChatGPT, Claude, Perplexity, Gemini and Copilot with usable referrer information, which is the human side of AI traffic: people clicking through from an answer, not crawlers. Accounts are invitation-only for now, so the way in is the waitlist on the homepage.

What I didn't test

  • Real crawler visits. I replayed documented user-agent strings; I didn't wait for the actual bots.
  • Firefox, Safari and mobile browsers, and stealth plugins that hide automation on purpose.
  • Commercial bot-detection services or any specific analytics product's filters.
  • Whether any particular AI agent product uses a real browser, a headless one or plain HTTP. Vendors change this, so measure your own logs.

Checklist

  1. Count HTML-only bots from server or CDN logs, never from a JS tracker.
  2. Match bot tokens before browser names; OAI-SearchBot's string contains "Chrome".
  3. Test your user-agent library against the bot strings you care about.
  4. In the tracker, flag navigator.webdriver === true and HeadlessChrome; don't rely on client hints alone.
  5. Expect some automation to look exactly like people. Watch patterns, not single visits.
  6. Keep excluded traffic countable, and annotate filter changes.

Need help with a project? The team behind FindUtils takes on freelance work on Upwork, including PHP bug fixes and server-side bot filtering for a PHP site.