Short answer: in my check, dev.to, Substack and Gumroad allow GPTBot, OAI-SearchBot, PerplexityBot and Google-Extended in robots.txt. Medium disallows GPTBot but allows the others. Indie Hackers disallows all four for the whole site. On dev.to the full post text is also in the raw HTML, so a crawler that does not run JavaScript still gets the article.
Small sample: one day, one machine, plain curl with a browser user agent, no bot user agents spoofed. Part of the run was interrupted, so anything I could not finish is left out.
robots.txt, per host
| host | GPTBot | OAI-SearchBot | PerplexityBot | Google-Extended |
|---|---|---|---|---|
| dev.to | allowed | allowed | allowed | allowed |
| aicitationnotes.substack.com | allowed | allowed | allowed | allowed |
| normbrytande.gumroad.com | allowed | allowed | allowed | allowed |
| medium.com | disallowed (/) | allowed (*) | allowed (*) | allowed (*) |
| indiehackers.com | disallowed (/) | disallowed (/) | disallowed (/) | disallowed (/) |
Notes:
- "Allowed" means robots.txt does not forbid it. Whether a CDN or WAF blocks the real bots is not verified.
- GPTBot is OpenAI's training crawler. OAI-SearchBot is the one used for ChatGPT search. Medium only blocks the first.
- dev.to serves a
/llms.txt(200). It asks that AI-assisted articles setai_disclosure_levelaccurately.
Is the article text in raw HTML?
curl -sL --compressed -A "<Chrome desktop UA>" "$URL" - dev.to: all 5 of my posts, full text present, 692 to 1,657 words each.
- Substack and Indie Hackers: full text present.
- Medium: 403 Cloudflare challenge on profile and all 6 posts. The RSS feed returned full content.
- Gumroad product pages: body text only inside an Inertia
data-pageJSON attribute, plus<title>, meta description and Product JSON-LD.
Exact-title lookups
With a web search tool (backend not disclosed, so not tied to any specific engine): all 5 of my dev.to posts were found by exact title. None of my 6 Medium posts were found, and for three cross-posted ones the dev.to copy came up instead.
What the two dev.to posts Perplexity cited had in common
For "How do I check if ChatGPT recommends my product?", Perplexity cited two dev.to posts by other authors, both with 0 reactions. One has the question as its title and answers in the first sentence. The other has a near-question title and answers within 3 short paragraphs, then lists numbered steps. My posts open with a story. Two examples are not proof, so the next test is one page with the question as H1 and the answer first.
