I run two small Next.js sites on the same host. Google indexes neither of them well, and this week I read the crawl stats for both instead of guessing. They fail in opposite ways.

Site one: a calculator site, robots.txt allows everything

Search Console, crawl stats, last 90 days:

total crawl requests 988 average response time 173 ms by response 200: 97 % 404: 2 % by purpose refresh: 100 % by file type JavaScript 50 % HTML 37 % CSS 7 % image 4 % by Googlebot type page resource load 57 % smartphone 34 %

And the Pages report:

indexed 32 discovered - currently not indexed 176 crawled - currently not indexed 40

Read together: Google made 988 requests in three months, and more than half of them were not pages. They were the scripts and stylesheets it needs to render pages it already knows. About 37 % were HTML, which is roughly 365 page fetches in 90 days, four a day, for a sitemap of 263 URLs. Every request was a refresh of something known. None went to discovering anything new, and 176 URLs sit in "discovered" with no crawl date at all.

Nothing is broken. The server answers in 173 ms, nothing is blocked, the sitemap is read. Google gives this site a very small allowance and spends more than half of it on JavaScript.

Site two: a resume builder, robots.txt blocked /_next/

Same host, same framework, and one line that had been in robots.txt since May: Disallow: /_next/. It was there to keep "infrastructure" out of search. /_next/static is where every stylesheet and script lives.

total crawl requests about 1,680 by file type HTML 80 % JavaScript 2 % image 4 % by Googlebot type smartphone 57 % page resource load 4 % crawled - currently not indexed 411

Here Google fetched far more HTML, because it was not allowed to fetch anything else. It read 411 pages and declined to index them. I cannot prove the blocked resources are the reason: this site also had a week in September when a redirect loop hit every crawler, and it has few links. But a page whose CSS and JS cannot be loaded is not the page a visitor sees, and Google's own guidance is not to block them. I removed the line today.

What I take from the two together

  • The same complaint, "Google does not index my site", had two different shapes: never crawled on one site, crawled and declined on the other. The Pages report tells you which one you have, and the fix is different.
  • Blocking /_next/ does not save crawl budget in any useful way. It trades rendering for raw HTML fetches.
  • Allowing it has a cost on a site with little authority: the rendering requests come out of the same small allowance. My guess, and it is only a guess, is that a deploy makes it worse, because the chunk file names change and Google has to fetch them again.
  • Crawl stats sit under Settings, not in the main reports, and I had never opened them. Ten minutes there replaced weeks of theories.

What is left for site one is the unglamorous part: links from outside, since the allowance seems to follow how much the rest of the web points at you.

The calculator site is AllSmartCalculators; a typical page is the PPF calculator. Written by an AI assistant from my own Search Console data.