Most small-business sites have never thought about AI crawlers. That's fine — the default is
permissive, so they get cited by default.
The problem is the ones who did think about it and got it half right.
I fetched robots.txt from 99 small-business domains across 20 industries (dental, HVAC, law,
physio, moving, and so on), collected through industry-and-city searches. 87 responded.
The breakdown:
| Behaviour | Sites | Share |
|---|---|---|
| Allow every AI crawler | 69 | 79.3% |
| Block training, allow AI search | 3 | 3.4% |
| Block training and AI search | 15 | 17.2% |
| Block AI search only | 0 | 0.0% |
The interesting number isn't 17.2%. It's 3.4%.
Of the 18 sites blocking training crawlers, 15 also block the crawlers that build AI citation
indexes. Only three separate the two decisions.
Why that matters: opting out of model training and opting out of being cited are two different
things, controlled by different User-agent strings in the same file. Block GPTBot and you
opt out of training. Block OAI-SearchBot too and you've removed yourself from ChatGPT answers
— which doesn't show up in analytics, because the visit never happens.
For comparison: Scout Radar, which reads robots.txt from the Tranco top 10,000, reports 20.2%
blocking at least one training crawler, with 9.1% blocking training while keeping search open.
So roughly 45% of top-site blockers make the distinction. In my sample it was 17%.
With 18 blocking sites the confidence interval is wide — treat this as a directional signal,
not a measurement. But the mechanism lines up: large sites have someone whose job is to know
the difference, and small sites copy a template or accept a default.
Full method, industry breakdown, limitations and the raw CSV (87 rows, no scores, just
measurements) are here:
https://sacredlantern.online/en/research/ai-crawler-access-2026
Happy to answer questions about the parsing logic — groups per RFC 9309, bot-specific rules
taking precedence over wildcard, Disallow: / unless overridden by Allow: / in the same group.
