Look at your robots.txt. Ctrl-F (Search) for GPTBot. See if you have a straight-up Disallow: / with no other info below User-agent: *. It's possible that a little snippet from some 2024 "protect your content" post is still blocking the crawlers that fetch pages for AI answers.

curl -s https://yoursite.com/robots.txt | grep -n -i -A 2 -E "gptbot|user-agent: \*" 

This checklist comes from RankControl, an AI SEO and AEO platform. On November 11, 2026 we reread the crawler pages from OpenAI, Anthropic, Google and Perplexity against it. Three files do the work (robots.txt, a sitemap and an optional llms.txt), and setup takes about 30 minutes plus 15 for checks.

Step 1: Find out who is visiting

AI companies each have a bunch of crawlers for different purposes (answering search questions, getting text for training, when a person opens a single page as a result of querying an assistant). One user agent token per robots.txt group, so you need a separate rule per job. Cloudflare reported that GPTBot requests increased by 305% from May 2024 to May 2025 (July 1, 2025 report).

Token Job What the operator documents
OAI-SearchBot ChatGPT search Opted-out sites "will not be shown in ChatGPT search answers"
Claude-SearchBot Claude search Disabling it stops indexing for search
PerplexityBot Perplexity search Not used for AI foundation models
GPTBot, ClaudeBot OpenAI and Anthropic training Blocking signals no training use
Google-Extended Gemini training and grounding Token only, no user-agent string; no effect on Google Search
CCBot, Bytespider Training (Common Crawl, ByteDance) Bytespider obeys robots.txt only sometimes
ChatGPT-User, Claude-User, Perplexity-User One page, on a user's request OpenAI: rules "may not apply". Perplexity: "generally ignores". Anthropic: disabling stops retrieval

(srcs: OpenAI, Anthropic, Google, Perplexity crawler docs) So no, blocking GPTBot will not prevent a ChatGPT user pulling your page for use in a chat. That's ChatGPT-User.

Step 2: Write the robots.txt

Allow search answer crawlers to crawl & disallow training crawlers; if you block all AI crawlers then you don't participate in AI search. Cloudflare's post on August 29, 2025 (prior 12 months) said 80% training, 18% search, 2% user actions, so that's why the split below.

# Allow AI search crawlers User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / # Opt out of model training User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / 

Remove any group you'd rather allow. Three errors will silently break the file.

  1. Disallow: / under User-agent: *. The wildcard group covers every crawler with no group of its own, including AI bots you have never heard of. RFC 9309, section 2.2.1: a crawler obeys the group matching its token and falls back to * only if none does.
  2. A GPTBot block copied from a 2024 post. OpenAI documents GPTBot as the training crawler and OAI-SearchBot as the search crawler, each setting independent. Blocking GPTBot leaves ChatGPT search alone, while blocking OAI-SearchBot keeps your pages out of its answers (they can still appear as navigational links), and a change can take about 24 hours to reach search.
  3. Checking GoogleOther instead of Googlebot. Google's AI features documentation (last updated December 10, 2025) says robots.txt directives for Googlebot are the control, and a page must be indexed and snippet-eligible to appear as a supporting link in AI Overviews or AI Mode. Google says GoogleOther affects no specific product, so a GoogleOther line changes neither.

Step 3: Decide what llms.txt is for

On November 11, 2026, we queried the crawler pages for OpenAI, Anthropic, Google, and Perplexity for "llms.txt". In Anthropic's and Google's pages, the word doesn't appear. In OpenAI's and Perplexity's pages, the word appears but only to link to the index of documentation for those services. (A November 21, 2026 rerun repeated this result.) Google's generative AI feature guide (last updated July 10, 2026) reports that Google Search "ignores" the file. And SE Ranking's analysis of nearly 300,000 domains finds no measurable correlation between the presence of the file and higher citation rates.

I would definitely publish it still (they have an agent read it on-demand in the llmstxt.org proposal, so if an agent lands on your domain they will have an anchor point):

# YourCompany  > Short one-line description of what your company does.  ## Docs  - [Getting Started](https://yoursite.com/docs/getting-started): Setup guide for new users - [Pricing](https://yoursite.com/pricing): Plans and pricing details 

Put it at yoursite.com/llms.txt (try to keep it under 50 lines). List your 10-20 most important pages. Add a plain English line about what each one does and who it's for. Update every quarter.

Step 4: Walk through the seven settings

There are seven settings that control if your pages are discovered, accessed and cited by AI search engines. RankControl says that sites that do all of the seven right get 3-4x the AI citations of sites that get two or three right.

  1. robots.txt lets in the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot); test the file with a robots.txt tester before it goes live.
  2. No X-Robots-Tag: noindex on key pages. A header can block AI crawlers while robots.txt is clean: curl -s -D - -o /dev/null https://yoursite.com/page | grep -i x-robots-tag should print nothing.
  3. llms.txt at the domain root, as above.
  4. A current sitemap. AI crawlers use it to find pages, so check the dates: curl -s https://yoursite.com/sitemap.xml | grep -m 5 "<lastmod>".
  5. FAQ schema. FAQPage JSON-LD labels each question and answer for machines. The checklist assumes models parse it faster than prose, yet Google's July 10, 2026 guide says its AI features need no special markup.
  6. Answers first. Kevin Indig's Growth Memo analysis of 18,012 ChatGPT citations (February 16, 2026) found 44.2% came from the first 30% of the text.
  7. Server response under 3 seconds, because AI crawlers have timeout limits: curl -s -o /dev/null -w "%{time_total}\n" https://yoursite.com/page.

Step 5: Verify it in 15 minutes

On November 21, 2026, rctrl.com got 200 when checking the robots.txt, homepage (as GPTBot, ClaudeBot and browser), llms.txt. We also have an Allow: / group for GPTBot, ClaudeBot and PerplexityBot in the robots.txt. Do the top grep again for each search crawler name. Then check a key page as each bot to see if you get 200:

for ua in "Mozilla/5.0 (compatible; GPTBot/1.0)" \ "Mozilla/5.0 (compatible; ClaudeBot/1.0)" \ "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)"; do printf "%s -> " "$ua" curl -s -o /dev/null -w "%{http_code}\n" -H "User-Agent: $ua" https://yoursite.com/your-important-page done 

403 or 429 means your server (or CDN) blocks AI bots (regardless of robots.txt). Then check in your logs how many times each bot hit you (adjust the path), if it's 0 times in 30 days, then it is either because of robots.txt or they haven't found you yet.

zgrep -h -o -E "OAI-SearchBot|Claude-SearchBot|PerplexityBot|GPTBot|ClaudeBot|Bytespider" /var/log/nginx/access.log* | sort | uniq -c 

Lastly you should be getting 200 for curl -s -o /dev/null -w "%{http_code}" https://yoursite.com/llms.txt because you'll get a 404 if it's not deployed.

Step 6: Check your CDN and firewall

Seeing 403 for bot strings and 200 for the browser string? That means a CDN or firewall is preventing it from hitting your server. On Cloudflare you can add WAF custom rules for the user agent string of the bots (Perplexity crawler docs say you can't bypass Bot Fight Mode with WAF custom rules. It may still challenge the bots you're allowing with your rule. Match user agents with their published IP ranges). Vercel: Look for bot blocking headers in vercel.json. AWS: update WAF rules to allow the 3 search bots. We lost 2 weeks on a client site to this when robots.txt appeared to be correct.

Step 7: Set up monitoring

Six months after everything is set up, a new robots.txt rule or default CDN bot protection or WordPress plugin update could override your config, and it can take weeks to notice. Manual monitoring requires you to curl your robots.txt every week, check server logs for bot traffic and ask ChatGPT about your brand name, about 2-3 hours a week. RankControl's free AI Crawler Checker tests your robots.txt and firewall against 12 AI crawlers, sending a live request as each of six search and browsing crawlers and reading the rules for six training crawlers, with no signup.

Which crawler was blocked on your site? Let me know in the comments.