Should your link checker obey robots.txt? Let's argue about it

Every link checker eventually has to answer a question its authors probably didn't expect: when a site's robots.txt says "please don't fetch this path", and your page links to that path, should the checker fetch it anyway to see if the link works?

It sounds like a niche config flag. In practice, it decides whether your tool reports some links as broken, some as "skipped", or quietly doesn't check them at all. And people have strong feelings about it.

Here are both sides, as fairly as I can put them, and then where I land.

A quick refresher on what robots.txt actually is

robots.txt is a plain text file at the root of a site that tells automated clients which paths the site owner would rather they didn't fetch. It's a convention that was eventually written up as an internet standard (the Robots Exclusion Protocol, RFC 9309). Two things are worth remembering:

It's voluntary. Nothing technically stops a client from ignoring it. Well-behaved bots follow it; badly behaved ones don't.
It's about crawling, not access. It doesn't password-protect anything. A disallowed URL is still public. The file says "don't crawl", not "don't look".

The standard talks about "crawlers", meaning automated clients that discover and fetch URLs. Whether a link checker counts is exactly the argument.

The case for obeying robots.txt

"It's an automated client. Automated clients follow robots.txt. End of story."

The purist position is simple: your link checker is software making requests without a human clicking anything. That's the definition of a bot. If you're writing a bot, you follow the rules bots follow.

"Site owners use it to protect fragile endpoints."

Some paths are disallowed for good reasons: search pages that are expensive to render, calendar pages that generate infinite URLs, admin panels, faceted filters. A link checker that fans out across hundreds of documentation pages and hits those paths on every CI run is adding real load to someone else's server, every time you push.

"If you ignore it, you'll get blocked, and then your results are worse anyway."

Sites that notice bots ignoring robots.txt tend to respond with rate limits, 403s or outright IP blocks. If your link checker runs from a shared CI provider, you may be getting the whole IP range blocked for everyone. Politeness isn't just ethics; it's self-interest.

"It's the norm in the ecosystem."

Several well-known crawlers and SEO tools respect robots.txt by default. Following the same norm makes your tool predictable.

The case for ignoring it (for links you already have)

"A link checker isn't discovering URLs. It's verifying ones a human chose."

This is the strongest argument on the other side. A crawler starts somewhere and follows everything it finds, which is exactly what robots.txt was designed to manage. A link checker, at least for external links, does something narrower: it requests specific URLs that an author deliberately put on a page, usually once. That's closer to a reader clicking the link than to a crawler indexing a site.

"robots.txt doesn't say whether a page exists."

If you skip disallowed URLs, you can't tell your users whether those links work. And plenty of important pages live behind disallow rules: login pages, search result links, account pages, product filter URLs, even some documentation sections that sites didn't want indexed. "Skipped" is not the same as "fine", but it often gets read that way.

"Browsers don't read robots.txt, and link previews mostly don't either."

When someone pastes a link into a chat app, the app fetches the page to build a preview. That's automated too. The web already accepts that "fetching a specific URL on behalf of a person" is a different category from crawling.

"The load is tiny."

One HEAD or GET per external link per run, with caching, is negligible for most sites. If you're checking a link to someone's search page once a day, that's not what robots.txt is defending against.

The awkward middle cases

The argument gets messier when you look at what link checkers actually do:

Internal links on your own site. If you're checking your own pages, your own robots.txt is about search engines, not about you. Ignoring it for your own domain seems uncontroversial.
Recursive checking. Some link checkers follow links found on the pages they check, sometimes across domains. At that point it really is crawling, and the "it's just one URL" defence evaporates.
Scale. A docs site with 5,000 pages that links to the same third-party API reference 5,000 times, rechecked on every pull request, can generate meaningful traffic if the tool doesn't deduplicate and cache.
User agent honesty. Some people sidestep the debate by pretending to be a browser. That's the one option I think is clearly wrong: if you're going to fetch something a site asked bots not to fetch, at least say who you are.

Where I land

I think the right default is a split:

Your own domain: ignore robots.txt. You own both the links and the rules. Check everything.
External links you've explicitly written: check them, once, politely, even if robots.txt disallows the path. Identify your tool honestly in the User-Agent, deduplicate URLs across the whole run, cache results for a sensible period, and respect rate limits and Retry-After headers.
Anything that looks like crawling (following links you didn't write, recursing into external sites): obey robots.txt, no exceptions.
Report skipped links as skipped, never as passing. If a user configures the tool to obey robots.txt everywhere, the report should say clearly how many links were not verified and why.
Make it a visible setting, not a hidden behaviour, so teams can pick the stricter option if their policy requires it.

The principle underneath is the one robots.txt itself is built on: it's about managing automated discovery and load. A tool that verifies a short list of author-chosen URLs, honestly and gently, is doing something different from a crawler. A tool that wanders, hammers or hides its identity is not.

That's roughly how we think about checks at linkbruno.com, but I'm genuinely not sure the line is in exactly the right place, and I know some people feel strongly that any automated request should obey the file, full stop.

A quick way to see how this affects you

If you want to know whether this debate matters for your site, it's easy to measure:

Export all the unique external URLs from your site or docs.
For each domain, fetch its robots.txt once.
Count how many of your links fall under a Disallow rule for a generic user agent.

On most content sites the number is small. On docs sites that link heavily to search pages, dashboards or app routes, it can be surprisingly large, and those are exactly the links that break when products change.