Quick tip: a tiny script that prints the title, canonical and robots tags of any URL (so you stop opening view-source)

Three things on a page do a surprising amount of work in search:

the <title>, which usually becomes the clickable headline in results;
the canonical link, which tells search engines which URL is the "real" one;
the robots directives, which can quietly tell search engines not to index the page at all.

They also break quietly. A template change drops the canonical. A staging setting leaks noindex into production. A CMS update starts appending the site name twice to every title. None of it shows on the page itself.

Opening view-source for every URL gets old fast. Here's a small script that checks a list of URLs in one go.

The script

#!/usr/bin/env python3 # seo_tags.py - print title, canonical and robots for each URL import sys, urllib.request from html.parser import HTMLParser class Tags(HTMLParser): def __init__(self): super().__init__() self.title, self.canonical, self.robots, self._in_title = "", [], [], False def handle_starttag(self, tag, attrs): a = dict(attrs) if tag == "title": self._in_title = True elif tag == "link" and "canonical" in (a.get("rel") or "").lower().split(): self.canonical.append(a.get("href", "")) elif tag == "meta" and (a.get("name") or "").lower() in ("robots", "googlebot"): self.robots.append(a.get("content", "")) def handle_endtag(self, tag): if tag == "title": self._in_title = False def handle_data(self, data): if self._in_title: self.title += data for url in (line.strip() for line in sys.stdin if line.strip()): req = urllib.request.Request(url, headers={"User-Agent": "seo-tags-check/1.0"}) with urllib.request.urlopen(req, timeout=15) as r: final, status = r.geturl(), r.status header_robots = r.headers.get("X-Robots-Tag", "") p = Tags(); p.feed(r.read().decode("utf-8", "replace")) print(url) if final != url: print(f" redirected -> {final}") print(f" status: {status}") print(f" title: {p.title.strip()!r} ({len(p.title.strip())} chars)") print(f" canonical: {p.canonical or 'MISSING'}") print(f" robots: meta={p.robots or '-'} header={header_robots or '-'}")

Run it with a list of URLs:

printf '%s\n' https://example.com/ https://example.com/pricing https://example.com/blog/ \ | python3 seo_tags.py

No dependencies beyond the Python standard library.

What to look for

Missing canonical. Not always a disaster, but on sites with URL parameters, filters or tracking codes, a missing canonical makes duplicates more likely.

More than one canonical. The script collects them all. Two conflicting canonicals on one page usually means a theme and a plugin both add one. Search engines may ignore both.

Canonical pointing somewhere unexpected: a staging domain, an http:// URL on an https:// site, or the homepage on every page. All three are common template bugs.

noindex anywhere you didn't put it. Check both the meta tag and the X-Robots-Tag header. The header version is easy to miss because it never appears in the HTML, and it's often set by server or CDN configuration rather than by the people who build pages.

Titles that are empty, duplicated or very long. Long titles get shortened in results. Every page having the same title means search engines have to guess which is which.

Redirects. If a URL you expected to be final redirects, your internal links or sitemap may be pointing at old addresses.

Two small improvements worth making

Handle error responses. As written, urlopen raises an exception on a 404 or 500 and the script stops. Wrap the request in a try/except urllib.error.HTTPError as e: block and print e.code, so one broken URL in your list doesn't hide the results for the rest. A page in your important-URLs file returning an error is, of course, the most urgent finding of all.

Collect a few more tags. The same parser pattern works for other things that drift silently:

the meta description, which often becomes the snippet under your title;
hreflang alternate links on multilingual sites, to check that each language version lists the others;
og:url and og:title, which control how links look when shared, and which sometimes disagree with the canonical;
the lang attribute on the <html> element.

Add each as another branch in handle_starttag. Keep the output to one line per tag so diffs stay readable.

Make it a habit

A few ways to get value from it beyond one-off checks:

Keep a file of your most important URLs, such as the homepage, top products, pricing and key articles, and run the script after every deploy.
Diff the output. Save today's output, run it again after a release, and compare. Any change in a canonical or robots value deserves a look.

python3 seo_tags.py < urls.txt > after.txt && diff before.txt after.txt

Feed it your sitemap to spot-check a sample of URLs from every section of the site.

It's also worth running the script with a crawler's user agent string as well as the default one. Some sites, through bot protection or misconfigured rules, serve crawlers different headers from the ones browsers get, and that difference is exactly what you want to know about.

A limitation to know about

This script reads the HTML your server sends. If your site adds or changes these tags with JavaScript after the page loads, the script won't see that, and search engines may or may not, depending on whether and when they render the page. If the script shows a missing canonical that you can see in your browser's element inspector, that's a finding in itself: the tag only exists after JavaScript runs. Server-rendering these tags is the safer choice.

Why we care

Titles, canonicals and robots directives are small, invisible and decisive, which makes them perfect candidates for automatic checking. Catching changes in exactly these kinds of tags is part of what we do at seotto.com.