This article was created with the help of AI. The numbers come from my own site's Search Console and server logs.
On October 8 I opened my own news site, promtime.net, on my phone and got Cloudflare's error 1015 instead of a page. My own rate-limiting rule had blocked me, and all I had done was read posts on it.
The cause was a Next.js feature that exists to make the site feel fast. Two days later Search Console showed me the same feature from Googlebot's side. Between August 10 and October 22 Googlebot made 25,943 requests to the site, and Search Console filed 16,284 of them under "Other file type": not pages, not images, not scripts.
What the router does
When a <Link> scrolls into view, the Next.js router prefetches the page behind it. It requests the same URL with an _rsc parameter and an RSC: 1 header and gets back the React Server Components payload instead of HTML, so the click that follows feels instant. Googlebot renders pages with JavaScript, and the prefetches fire for it as well.
All 990 example URLs Search Console lists for "Other file type" end in ?_rsc=. One page can have many of them, because Next.js derives the value from request headers. On September 28, within two minutes, Googlebot fetched one post with six different suffixes. Here are four:
/c/ru/post/claude-mythos-solves-the-erdos-discrepancy-problem?_rsc=LIEXrFg38zfWwJsj /c/ru/post/claude-mythos-solves-the-erdos-discrepancy-problem?_rsc=kFwwhQWGfJ28v95A /c/ru/post/claude-mythos-solves-the-erdos-discrepancy-problem?_rsc=QQ8lwV8YUpwFDVCh /c/ru/post/claude-mythos-solves-the-erdos-discrepancy-problem?_rsc=vcDMZA7KM_P7ZtS6 Six days in August
Almost all of it happened between August 18 and 23. On August 19 alone Googlebot fetched 8,984 of these URLs, while on an ordinary day it asked for fewer than a hundred URLs of every other kind. By September the row was down to about ten requests a day.
The timing matches one change. On August 17-19 the site got newsroom pages, an archive with a month navigator and an infinite feed. Why the spike started, and why it then stopped on its own, these numbers can't show.
Who paid for it
Right after that week the site's impressions in Google fell almost to zero, and the crawl is the obvious suspect. The data doesn't back it. Google's guide to crawl budget is written for sites with a million pages or more, or ten thousand if they change every day. Mine has a few thousand. The guide's other warning sign, a pile of URLs stuck in "Discovered - currently not indexed", did show up on my site, but in October, six weeks after the spike was over.
The ones who paid were readers, me included.
The rate-limiting rule on my Cloudflare plan sees only the path, not the query string, so to Cloudflare a prefetch looks exactly like someone opening a page. A post page carried more than forty tag and archive links, and the router prefetched each one as it came on screen. One post view could fire up to 58 requests, enough to trip the limit on its own.
What I changed
- Turned off prefetch on dense link lists: tags, archive months, feed cards. In Next.js that's
prefetch={false}on<Link>. Every other link still prefetches. - Doubled the Cloudflare rate limit.
- Blocked the RSC URLs in robots.txt:
Disallow: /*?_rsc= Disallow: /*&_rsc= The first one is now a standing rule in the project's notes: a dense list of links gets prefetch={false} from the start.
How to check your own site
In Search Console, open Settings, then Crawl stats, and click the "Other file type" row in the "By file type" block. The example URLs are at the bottom. If they end in ?_rsc=, you're looking at the same thing.
In your server logs:
grep Googlebot access.log | grep -c '_rsc=' Since the robots.txt rule went in on October 10, my server logs haven't shown a single _rsc request from Googlebot. Then again, it had almost stopped asking for them long before that. The spike ended on its own, back in August.
The site from this story also publishes AI news in English, as Promtime.
