A polite sign on the door stops the polite. Everyone else needs a lock.

Try to read a Reddit thread with an AI agent and you will probably fail. Search tools come back empty, and a browser-driving agent hits a wall. This is deliberate. Reddit spent two years turning a public website into one that machines can read only if they pay or have an agreement.

This article walks through the layers Reddit has publicly described, from the weakest to the strongest, and what you can take from each for your own site.

The Problem

A public page is readable by anyone, and that includes software. Until 2023 that was fine. Then large language model (LLM) builders started treating forums as training data, and AI assistants started fetching pages live to answer questions. For a site whose product is its content, that is a problem: someone else gets the value and the site gets neither traffic nor payment.

There is no single switch for "humans yes, agents no". What exists is a stack of layers, each catching a different kind of visitor.

Request arrives │ ├─ 1. robots.txt asks bots to stay out works on: crawlers that obey it ├─ 2. Terms + lawyers forbids scraping in writing works on: companies that fear a lawsuit ├─ 3. Paid API sells official access works on: buyers who'd rather pay than scrape ├─ 4. Rate limits, │ challenges, │ anomaly detection blocks suspicious traffic works on: most scraping scripts ├─ 5. Side doors restricts copies elsewhere works on: scrapers that skip your site └─ 6. Canaries plants traceable test pages works on: nobody, but proves who copied

Layer 1: robots.txt, the polite sign

robots.txt is a plain text file at the root of a site telling crawlers which paths they may fetch. Reddit announced on 25 June 2024 that it was updating the file and would keep rate-limiting and blocking unknown bots (TechCrunch). By 24 July 2024 it had been changed to disallow all crawlers, with Google still indexing Reddit under its paid data agreement while Bing, DuckDuckGo and others stopped (The Register). A Reddit spokesperson said the condition for access is a commitment not to use crawled data for AI training (9to5Mac). The file even carries a message: "Reddit believes in an open internet, but not the misuse of public content."

What you can copy: block AI-training crawlers by name. Reddit went further and blocked search engines too, which only makes sense if search traffic is not what you depend on. If it is, leave search crawlers such as Googlebot and Bingbot alone. The tokens below are an example; check each against the vendor's docs.

User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: CCBot Disallow: /

Save this as robots.txt at the root of your domain, so it is served at https://yoursite.com/robots.txt. Crawlers look only at that exact path, and each subdomain needs its own file.

Limit: robots.txt is a request, not an enforcement mechanism. Cloudflare documented a crawler that switched to a generic Chrome-on-macOS user agent after its declared crawler was blocked (PPC Land). The vendor denies training models on it. Blocking everything also has a cost: Reddit's blanket rule removed it from every search engine without a deal.

Layer 2: terms and lawyers

Reddit's User Agreement bars "scraping the Services without Reddit's prior written consent", according to the complaint as quoted by PPC Land. Reddit then enforced it: a cease-and-desist letter to Perplexity in May 2024, a suit against Anthropic in June 2025, and in October 2025 a federal suit against Perplexity, SerpApi, Oxylabs and AWMProxy (Reddit Inc. v. SerpApi, S.D.N.Y.) (Search Engine Land). These are allegations, not findings; SerpApi has said it disagrees with them.

What you can copy: put a no-scraping clause in your terms. It is free, and it is the precondition for any later enforcement.

Limit: this only works if you can afford to litigate.

Layer 3: a paid front door

In 2023 Reddit started charging for its API. Apollo, a popular third-party Reddit app built by one developer, would have owed about $20 million a year by its developer's calculation, at a quoted rate of $12,000 per 50 million requests, and it shut down on 30 June 2023 (MacRumors, TechCrunch). Licensing deals, such as the one with Google, sit alongside it. Reddit also backed the Really Simple Licensing (RSL) standard, which puts machine-readable licence terms in robots.txt (Engadget).

Idea to copy: give legitimate users an official, documented way in. Scraping is far easier to justify blocking when a sanctioned alternative exists.

Limit: it also hurt the community. Reddit's own third-party apps died.

Layer 4: block by behaviour

Reddit describes several technical defences in the October 2025 complaint: limits on request rate per Internet Protocol (IP) address, bot challenges (such as CAPTCHAs) and tools that flag unusual activity (PPC Land). Reddit has not published how these work, so treat anything more specific as unconfirmed.

Reddit limits how many requests one IP address (a visitor's network address) can make. The scrapers' answer is to spread requests across thousands of addresses, so no single one hits the limit.

The complaint alleges companies sell exactly this as a service. SerpApi allegedly advertises that customers "don't need to care" about CAPTCHAs and bot detection, and Oxylabs allegedly runs a network of over 62,000 IP addresses in New York. These are allegations, not findings; SerpApi has said it disagrees with them. The result is an arms race: each new defence gets a purchasable workaround.

What you can copy: per-IP rate limits and a managed bot-protection service in front of your site are cheap and catch the lazy majority.

Limit: every behavioural rule misfires on some humans, and a determined actor using real browsers and residential IPs looks like a person. An agent driving the user's own logged-in browser is hard to tell apart from the user by design.

Layer 5: close the side doors

Blocking your own site is not enough if copies of your content are easy to read elsewhere. A scraper can go around the front door.

One example is the Wayback Machine, a service from the Internet Archive that saves snapshots of web pages. Reddit said AI companies were reading its content from those snapshots instead of from Reddit, so in August 2025 it blocked the Wayback Machine from saving most Reddit pages. Only the homepage is still captured (SiliconANGLE, AV Club).

Another is Google's search results. The October 2025 lawsuit alleges that scraping companies collected Reddit content from Google results pages instead of from Reddit, which skips Reddit's own defences.

What you can copy: check where else your content appears (archives, caches, aggregators) and apply the same policy there.

Layer 6: canaries

Reddit says it created a test post "visible only to Google's crawler", and that its text later appeared in Perplexity's answers (allegation, Search Engine Land and PPC Land). Cloudflare ran the same experiment with fresh, unindexed domains whose robots.txt said no, and asked Perplexity about them.

What you can copy: publish a page with a unique, nonsense phrase that only one crawler can reach, then query AI tools for that phrase later. It turns "I think they copied me" into evidence.

What This Means for Your Site

Goal Cheapest step Catches
Opt out of AI training Named entries in robots.txt Honest crawlers
Create a legal basis No-scraping clause in terms Nothing alone; enables enforcement
Stop casual scraping Per-IP rate limit, managed bot filter Most scripts
Stop determined scraping Login wall, paid API, licensing Everything not paying
Prove copying Canary page Nobody, but wins arguments

Limitations

  • No layer stops a determined actor with a real browser. The goal is cost, not a seal.
  • Reddit is unusual. It has a huge data asset, licensing revenue and lawyers. Layers 1, 4 and 6 are cheap for anyone; layers 2 and 3 (litigation, a paid API with licensing deals) need money and a content library worth paying for.
  • Blocking agents blocks assistants. Your own users now ask AI tools to read pages for them. A site that blocks all agents also blocks those users; decide per audience.
  • Search visibility is a real cost. Reddit accepted losing Bing and DuckDuckGo because it sells access. A site that depends on search traffic should not copy that.
  • Facts here come from press and a lawsuit. Allegations are marked as allegations, and Reddit has not published its detection internals.

The Bottom Line

Reddit did not find a trick. It stacked six ordinary layers, from a text file to a lawsuit, and attached a price to the way in. For most sites, the practical version is three steps: name the AI crawlers in robots.txt, put rate limiting or a bot filter in front of the site, and plant a canary so you can tell if it matters.

Resources