More sites now ask a simple question: can ChatGPT, Claude and Perplexity actually read my pages? Most people open robots.txt, see nothing scary, and stop there. That check is not enough.
Two layers can block an AI crawler
- robots.txt. A
Disallowrule for a user agent likeGPTBotorPerplexityBot. Polite crawlers respect it. - The firewall or CDN. Cloudflare's "Block AI bots" toggle, a host's bot protection, or a WordPress security plugin. These return a 403 to the crawler no matter what robots.txt says.
The second layer is invisible if you only read robots.txt. Your file says "allowed", the bot gets blocked anyway, and you never find out.
Training bots vs search bots
Blocking every AI crawler is rarely what people actually want. The user agents fall into three groups:
| Type | Examples | Block it and... |
|---|---|---|
| Training | GPTBot, ClaudeBot, Google-Extended, CCBot | your pages aren't used to train models. You can still appear in AI answers. |
| AI search | OAI-SearchBot, Claude-SearchBot, PerplexityBot | you usually disappear from that tool's answers. |
| User request | ChatGPT-User, Claude-User, Perplexity-User | the AI can't open your page when a user asks it to. |
So a common setup is "search yes, training no": allow the search and user-request bots, disallow the training bots.
User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: PerplexityBot Allow: / User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: CCBot Disallow: / User-agent: * Allow: / I keep the full list of 20 user agents and three ready-made templates in a small open repo: ai-crawlers-robots-txt on GitHub.
Testing the firewall layer yourself
You can fake a crawler's request with curl and look at the status code:
curl -s -o /dev/null -w "%{http_code}\n" \ -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)" \ https://yoursite.com/ A 200 means the request got through. A 403 or a challenge page means something in front of your site is blocking that user agent. Repeat with OAI-SearchBot, PerplexityBot and ClaudeBot.
One caveat: some CDNs verify real crawlers by IP, so a spoofed user agent can be treated differently from the real bot. It's still a quick way to catch the most common problem, a blanket "block AI bots" rule.
What two big sites look like
On 16 Nov 2026 I ran both layers against two well-known sites:
- nytimes.com: only 4 of 20 AI bots allowed in robots.txt, and only 2 of 9 live test requests got through. A deliberate choice for a publisher.
- hubspot.com: all 20 allowed and all 9 live requests got through.
If you're a business that wants to be recommended by AI tools, your result should look like the second one.
A faster check
If you don't want to script it, we built a free checker at CodedSEO that does both layers for 20 bots and also looks for llms.txt: AI Visibility Checker. No sign-up.
Either way, check the firewall, not just the file.

