B2B product catalogs love filters. Material, size, thread, voltage, brand, certification. Buyers need them, because nobody scrolls through four thousand fasteners. But every filter combination can become its own URL, and crawlers will happily try them all.

A category with 8 filters and a handful of values each can produce many thousands of URLs that show almost the same products. Google's own guidance on managing crawling of faceted navigation URLs names the two costs: overcrawling, and slower discovery of new pages, because time spent on useless URLs is time not spent on your new products.

Here is how to keep filters for buyers without handing crawlers an endless maze.

Step 1: decide which filter pages deserve to exist

Most filter combinations are not pages anyone searches for. A few are. "Stainless steel M8 bolts" might be a real query with real buyers behind it. "Stainless steel, M8, 40 mm, zinc-free, sold in boxes of 200" is not.

Make a short list of the combinations that match real searches. Those become proper landing pages with a clean URL, their own title and a short introduction:

/bolts/stainless-steel-m8/

Everything else stays a filter for people, not a page for search engines.

Step 2: keep crawlers out of the rest

Google gives two straightforward options for filter URLs you don't need indexed.

Option A: block the filter parameters in robots.txt. This follows the pattern in Google's documentation:

user-agent: * disallow: /*?*material= disallow: /*?*size= disallow: /*?*brand=

If you also want certain filtered views crawled, add specific allow: rules for them, and test the file before you ship it.

Option B: put filters in the URL fragment. Google Search generally doesn't support URL fragments in crawling and indexing, so a filter state like this stays with the visitor:

/bolts/#material=stainless-steel&size=m8

A small amount of client-side code reads the fragment and filters the list:

function readFilters() { const params = new URLSearchParams(location.hash.slice(1)); return Object.fromEntries(params.entries()); } function setFilter(key, value) { const params = new URLSearchParams(location.hash.slice(1)); if (value) params.set(key, value); else params.delete(key); location.hash = params.toString(); } window.addEventListener('hashchange', () => renderProducts(readFilters()));

Fragments suit catalogs where filtering happens in the browser. If filtering needs the server, Option A is usually simpler.

Step 3: if filter URLs must be crawlable, make them predictable

Some sites need filtered URLs indexed. In that case, Google's advice is to keep them tidy:

  • Use the standard & separator between parameters.
  • Keep the filters in the same order every time, so ?material=steel&size=m8 and ?size=m8&material=steel never both exist.
  • Return a real 404 when a combination has no products, instead of an empty page with a 200 status.

Step 4: know what canonical and nofollow can and can't do

Pointing filtered pages to the unfiltered category with rel="canonical" may reduce crawling of those URLs over time, according to Google. It is a hint, not a wall. rel="nofollow" on filter links can help too, but only if every link to that URL carries it, which is hard to guarantee across a whole site.

Use them as support for Step 2, not instead of it.

Step 5: check the result

After the change, watch the Crawl stats report in Search Console for a few weeks. You want to see fewer requests for parameter URLs and more for product and category pages. If you have server logs, filter them by Googlebot and count requests that contain your filter parameters.

Also check that your chosen landing pages from Step 1 are linked from the category page in plain HTML, so crawlers can still find them.

The short version

  • Decide which filter combinations are real landing pages, and build those properly.
  • Keep crawlers out of everything else with robots.txt rules or URL fragments.
  • If filter URLs must be crawlable, keep a fixed parameter order and return 404 for empty results.
  • Treat canonical and nofollow as hints.

My team at VTurnU covers more catalog basics in a guide to SEO for ecommerce stores.