Most duplicate pages are not copied on purpose. A CMS keeps serving the old URL after a page is renamed. A newsletter link with a tracking parameter ends up in the sitemap. A print view, a demo copy of the home page, or a set of city pages built from one template quietly becomes a second (or fifth) version of the same content. Each page loads fine on its own. You only see the problem when you compare pages with each other.
This post builds a small duplicate finder in Python using only the standard library (urllib, html.parser, xml.etree, hashlib, difflib, csv, concurrent.futures). It reads a sitemap or sitemap index, fetches every page, strips the shared template, and compares pages three ways: identical HTML, identical main content, and near-identical main content scored with difflib. Matching pages are grouped into clusters, each cluster gets a canonical check and a suggested keeper URL, and the result is a CSV, a console summary and an exit code that works in CI.
Google Search Central's What is canonicalization (showing "Last updated 2026-08-20 UTC"), How to specify a canonical URL with rel="canonical" and other methods (showing "Last updated 2026-07-10 UTC") and Fix canonicalization issues (showing "Last updated 2026-08-21 UTC") were opened on 14 Nov 2026, and every Google statement below comes from those three pages. Appie free-tool descriptions are as of 14 Nov 2026, checked on the tool pages that day. The script was tested on Python 3.13.5 against a local fixture site with planted duplicates; the output below is real, from a run on 14 Nov 2026.
Disclosure: Appie.run publishes this article. Two free tools mentioned near the end are ours. The script does not depend on them.
Where this fits
This is another script in a small standard-library series. An earlier post audited canonical tags and noindex directives across a sitemap, which checks what each page says its canonical is. Another found missing and duplicate H1 tags. This one asks the question underneath both: which pages actually are the same, judged by their content, and do their canonical signals agree about which one should win?
What Google actually says (and what it does not)
The canonicalization overview defines the result of the process: "a canonical URL is the URL of a page that Google chose as the most representative from a set of duplicate pages." It then lists common reasons a site ends up with duplicates, which double as a checklist for what to plant in a test fixture:
- "Region variants: for example, separate USA and UK URLs with hreflang, but essentially the same content in the same language"
- "Device variants: for example, a page with both a mobile and a desktop version"
- "Protocol variants: for example, the HTTP and HTTPS versions of a site"
- "Site functions: for example, the results of sorting and filtering functions of a category page"
- "Accidental variants: for example, the demo version of the site is accidentally left accessible to crawlers"
The same page is clear that duplicates are not a penalty in themselves: "Some duplicate content on a site is normal and it's not a violation of Google's spam policies." The reasons it gives for caring are practical. Having the same content at many URLs "can be a bad user experience" and "may make it harder for you to track how your content performs in search results."
It also describes how Google handles them, and this is the part the script imitates. Google determines "the primary content (or centerpiece) of each page," and "If Google finds multiple pages that seem to be the same or the primary content very similar, it clusters them together." It then picks one page as canonical, and "duplicates are crawled less frequently." Your own signals help but do not decide: "indicating a canonical preference is a hint, not a rule."
The page on specifying a canonical URL ranks the signals "in order of how strongly they can influence canonicalization":
- Redirects: "A strong signal that the target of the redirect should become canonical."
-
rel="canonical"link annotations: "A strong signal that the specified URL should become canonical." - Sitemap inclusion: "A weak signal that helps the URLs that are included in a sitemap become canonical."
Two of its best practices map directly onto checks in the script: "Don't specify different URLs as canonical for the same page using different canonicalization techniques (for example, don't specify one URL in a sitemap, but a different URL for that same page using rel="canonical")," and "When linking within your site, link to the canonical URL rather than a duplicate URL." It also says "We don't recommend using noindex to prevent selection of a canonical page within a single site," and that none of these methods are required: "your site will likely do just fine without specifying a canonical preference."
The troubleshooting page adds the fix for near-duplicates you want kept separate: it "boils down to ensuring that the pages that are clustered together are sufficiently different," and notes that "Google might hold pages in a duplicate cluster for up to two weeks" after you change them.
What none of the three pages give is a similarity percentage. Google does not say how similar two pages must be to be clustered, so the script's --threshold (default 0.90) is a triage line for finding pairs worth a look, not Google's number. URL Inspection in Search Console is where you see which canonical Google actually chose.
How the comparison works
For each page the script keeps only the main content: text inside <main>, <article> or role="main" if the page has one, and everything outside <nav>, <aside>, <form> and top-level <header>/<footer> if it does not. Then, after all pages are fetched, any text block that appears on at least half of them (and on at least three) is dropped as template. That second step removes a "Start your free trial" box that sits inside <main> on every page and would otherwise make unrelated short pages look alike.
Pages are then compared in three passes, cheapest first:
-
exact_html: SHA-256 of the raw response bytes. Identical hashes mean the server returned the same file for two URLs. This is the old URL still serving, or a query-string variant the CMS ignores. -
same_content: SHA-256 of the normalized main-content words (lowercased, whitespace collapsed). Different templates, same body. Print views and AMP-style copies land here. -
near_duplicate:difflib.SequenceMatcher(...).ratio()on the two word lists, at or above--threshold. Template pages with a city name swapped land here.
difflib is accurate but slow, and comparing every pair is O(n²). So the near-duplicate pass first builds an inverted index of 5-word shingles and only sends a pair to difflib if their shingle Jaccard similarity is at least --prefilter (default 0.30). Shingles that appear on more than 50 pages are ignored, because a phrase on 50 pages says nothing about any two of them. Pages under --min-words (default 50) skip this pass, since two short pages can score high on very little text.
The prefilter is deliberately loose. Changing one number in a sentence breaks every shingle that touches it, so Jaccard drops much faster than difflib's ratio. On the fixture below, two pricing pages with a difflib ratio of 0.89 had a shingle Jaccard of 0.35, so a prefilter of 0.5 would have hidden them.
Every match is an edge, and a small union-find groups connected pages into clusters. If A matches B and B matches C, all three are one cluster even if A and C fall just under the threshold.
What the script flags
Within each cluster it reads every page's canonical (<link rel="canonical"> first, then a Link: <...>; rel="canonical" HTTP header) and checks whether the signals agree:
-
competing_self_canonicals: Meaning: Two or more pages in the cluster each declare themselves canonical; What to check: Either they are meant to be different (make them more different) or one should point to the other -
canonicals_disagree: Meaning: Pages in the cluster point to more than one canonical target; What to check: Pick one target for the whole cluster -
no_canonical: Meaning: A clustered page has no canonical at all; What to check: Add one pointing to the keeper, or redirect -
sitemap_lists_non_canonical: Meaning: The sitemap lists a URL whose own canonical points elsewhere; What to check: Google's "don't specify one URL in a sitemap, but a different URL" case; list the canonical instead -
canonical_not_in_sitemap: Meaning: The canonical target is not in the sitemap; What to check: Often the target is missing, renamed or a 404 -
url_variant: Meaning: The URL differs from another member only by query string, trailing slash,index.html,wwwor case; What to check: Usually a tracking or filter parameter; fix the link or the sitemap generator -
noindex: Meaning: A clustered page hasnoindex(informational); What to check: Google recommendsrel="canonical"overnoindexfor duplicates on one site -
http_error/fetch_failed/not_html/redirected: Meaning: The sitemap URL did not load as an HTML page; What to check: Fix it or remove it from the sitemap
Each clustered row also gets a suggested keeper: the URL most members already name as canonical, preferring URLs in the sitemap, then URLs without a query string, then the shortest. It is a starting point for a decision, not the decision.
The script
Save as dup_pages.py. No pip install needed.
#!/usr/bin/env python3 """Find duplicate and near-duplicate pages across a sitemap. Exact HTML hashes, main-content hashes, shingle prefilter + difflib similarity, duplicate clusters, canonical checks per cluster, CSV report. Standard library only. """ import argparse import concurrent.futures as cf import csv import difflib import gzip import hashlib import re import sys import urllib.error import urllib.parse import urllib.request import xml.etree.ElementTree as ET from collections import Counter, defaultdict from html.parser import HTMLParser UA = "dup-pages-audit/1.0 (+https://example.com/bot)" SM_NS = "{http://www.sitemaps.org/schemas/sitemap/0.9}" SKIP = {"script", "style", "noscript", "template", "svg", "iframe", "head"} CHROME = {"nav", "aside", "form", "header", "footer"} MAIN = {"main", "article"} BLOCK = {"p", "li", "h1", "h2", "h3", "h4", "h5", "h6", "td", "th", "blockquote", "pre", "dd", "dt", "figcaption", "div", "section"} VOID = {"area", "base", "br", "col", "embed", "hr", "img", "input", "link", "meta", "source", "track", "wbr"} WORD = re.compile(r"[^\W_]+(?:['’-][^\W_]+)*", re.UNICODE) def fetch(url, timeout=20): req = urllib.request.Request(url, headers={"User-Agent": UA, "Accept-Encoding": "gzip"}) try: with urllib.request.urlopen(req, timeout=timeout) as resp: body = resp.read() if resp.headers.get("Content-Encoding") == "gzip": body = gzip.decompress(body) return (resp.status, resp.geturl(), resp.headers.get("Content-Type", ""), resp.headers.get("Link", ""), body) except urllib.error.HTTPError as e: return e.code, url, "", "", b"" except Exception as e: # DNS, timeout, TLS return None, url, str(e), "", b"" def read_sitemap(url, seen=None): seen = seen if seen is not None else set() if url in seen: return [] seen.add(url) status, _, _, _, body = fetch(url) if status != 200: raise RuntimeError(f"{url} returned {status}") if body[:2] == b"\x1f\x8b": body = gzip.decompress(body) root = ET.fromstring(body) if root.tag == SM_NS + "sitemapindex": urls = [] for loc in root.iter(SM_NS + "loc"): urls += read_sitemap(loc.text.strip(), seen) return urls return [loc.text.strip() for loc in root.iter(SM_NS + "loc") if loc.text] class PageParser(HTMLParser): """Collects title, canonical, robots meta and text blocks by context.""" def __init__(self): super().__init__(convert_charrefs=True) self.stack = [] self.title, self.canonical, self.robots = "", "", "" self.blocks = [] # (text, in_main, in_chrome) self.has_main = False self.buf = [] def flush(self): text = " ".join("".join(self.buf).split()) self.buf = [] if text: in_main = any(m for _, _, m in self.stack) in_chrome = any(c for _, c, _ in self.stack) self.blocks.append((text, in_main, in_chrome)) def handle_starttag(self, tag, attrs): a = {k: (v or "") for k, v in attrs} if tag == "link" and "canonical" in a.get("rel", "").lower().split(): self.canonical = self.canonical or a.get("href", "").strip() if tag == "meta" and a.get("name", "").lower() in ("robots", "googlebot"): self.robots += " " + a.get("content", "").lower() if tag in VOID: return if tag in BLOCK or tag in MAIN or tag in CHROME: self.flush() role = a.get("role", "").lower() is_main = tag in MAIN or role == "main" self.has_main = self.has_main or is_main inside_main = any(m for _, _, m in self.stack) chrome = (tag in {"nav", "aside", "form"} or (tag in {"header", "footer"} and not inside_main) or role in {"navigation", "banner", "contentinfo"}) self.stack.append((tag, chrome, is_main)) def handle_endtag(self, tag): if tag in VOID: return if tag in BLOCK or tag in MAIN or tag in CHROME: self.flush() for i in range(len(self.stack) - 1, -1, -1): if self.stack[i][0] == tag: del self.stack[i:] break def handle_data(self, data): tags = [t for t, _, _ in self.stack] if "title" in tags: self.title += data return if any(t in SKIP for t in tags): return self.buf.append(data) def norm_block(text): return " ".join(text.lower().split()) def analyse(url): status, final, ctype, link_hdr, body = fetch(url) row = {"url": url, "status": status if status else "fetch_failed", "final_url": final, "issues": []} if status != 200: row["issues"].append("http_error" if status else "fetch_failed") return row if final != url: row["issues"].append("redirected") if "html" not in ctype.lower(): row["issues"].append("not_html") return row p = PageParser() p.feed(body.decode("utf-8", "replace")) p.flush() canon = p.canonical m = re.search(r'<([^>]+)>\s*;\s*rel="?canonical"?', link_hdr or "") if m and not canon: canon = m.group(1) row.update({ "html_hash": hashlib.sha256(body).hexdigest()[:12], "title": " ".join(p.title.split()), "canonical": urllib.parse.urljoin(final, canon) if canon else "", "noindex": "noindex" in p.robots or "none" in p.robots.split(","), "blocks": [b for b, in_main, chrome in p.blocks if not chrome and (in_main or not p.has_main)], }) return row def variant_key(u): """Same key for URLs that differ only by protocol, www, case, trailing slash, index.html or query string.""" s = urllib.parse.urlsplit(u) host = s.netloc.lower().removeprefix("www.") path = re.sub(r"/(index\.html?)?$", "", s.path.lower()) or "/" return host + path class DSU: def __init__(self): self.parent = {} def find(self, x): self.parent.setdefault(x, x) while self.parent[x] != x: self.parent[x] = self.parent[self.parent[x]] x = self.parent[x] return x def union(self, a, b): self.parent[self.find(a)] = selfSource：https://dev.to/appierun_66/find-duplicate-and-near-duplicate-pages-across-your-sitemap-with-python-standard-library-only-5dl5Related posts
Thirty plugins and one of them is abandoned: a plugin audit you can actually finish
One Jira Ticket Through an Org of AI Agents: The Architecture
Cramming a New Framework Feels Faster. Your Brain Disagrees.
Appointment reminders in a Telegram bot without a scheduler library
MCP Sampling: When the Server Calls the Client Model
