When a site grows to 27,854 route pages, the sitemap stops being a formality and becomes a strategy decision. Search engines will not crawl everything equally, so the question is not how to list every page, it is how to tell crawlers which pages matter, which pages are new, and which pages can wait.
The first mistake we made was one giant sitemap. It was technically valid and practically useless: it hid the important pages among thousands of near-identical route pages, and when something changed, the whole file had to be regenerated. We split it into smaller sitemaps by route type and by region, so a change to one group only touches that group's file, and the sitemap index tells crawlers exactly where to look.
The second mistake was treating all routes as equal. A route between two large cities gets searched far more than a route between two small towns, and crawling them equally wastes crawl budget on pages few people want. We now order sitemap entries so the highest-demand routes appear first and are updated most often, while low-demand routes are still listed, just less prominently. This is not hiding pages, it is prioritising attention.
The third lesson was about freshness signals. A sitemap that claims every page changed yesterday, when nothing changed, teaches crawlers to ignore it. We only update lastmod when a page genuinely changes, price data, content, or structure, and we stopped regenerating timestamps on a schedule. Honest freshness signals get honoured. Noisy ones get ignored.
The fourth lesson was internal linking. A sitemap alone does not get a page crawled well; links from popular pages do. We added related-route links on every route page, so a crawler arriving at a busy route can walk to neighbouring routes naturally, and a visitor can too. The sitemap opens the door, but internal links do the walking.
The result is not dramatic, it is steady: the pages that matter get crawled faster, new routes get found sooner, and the long tail still exists for the rare searches that need it. The site is My Fast Movers, a long-distance moving cost calculator with a free quote tool, and its 27,854 routes are a reminder that at scale, organisation is a feature, not paperwork.
