A broken link checker looks at status codes. A soft 404 hides from it, because the status code says everything is fine. The server answers 200, and the page says "Sorry, this product is no longer available," or shows the site's not-found template, or is nearly blank. Search Console reports these as soft 404s and, per Google, "Such pages are excluded from Search."
This post builds a small auditor in Python using only the standard library (urllib, urllib.robotparser, xml.etree, html.parser, difflib, uuid, csv, concurrent.futures). It reads your sitemap or sitemap index, first requests a URL that cannot exist to see how the server handles missing pages, then fetches every sitemap URL and flags:
- 200 responses whose title, H1 or opening text carry an error message
- 200 responses whose text matches the site's own not-found page
- removed pages that redirect to the home page instead of a replacement
- nearly empty pages
- internal search result URLs listed in the sitemap
- real 404 and 410 URLs still listed in the sitemap, and redirected URLs the sitemap should list by their final address
Google Search Central's Troubleshoot Google Search crawling errors page (showing "Last updated 2026-11-16 UTC") and its HTTP status codes and network errors page (showing "Last updated 2026-02-04 UTC") were opened on 21 Nov 2026, and every statement attributed to Google below comes from those pages. The output in this post comes from a small fictional shop ("Larkspur Outdoor Supply") served on localhost, built for this article with planted problems. Appie free-tool descriptions are as of 21 Nov 2026, checked on each tool's page that day.
Disclosure: Appie.run publishes this article. Two free tools mentioned near the end are ours. The script does not depend on them.
Where this fits
This is another script in a small standard-library series that reads a sitemap and checks one thing per page. An earlier post on finding broken links and redirect chains catches the URLs that fail loudly with a 4xx or 5xx. This one catches the URLs that fail quietly with a 200.
What Google actually says
Definition. "A soft 404 error is when a URL that returns a page telling the user that the page does not exist and also a 200 (success) status code. In some cases, it might be a page with no main content or empty page."
Common causes Google lists. "A missing server-side include file. A broken connection to the database. An empty internal search result page. An unloaded or otherwise missing JavaScript file."
What happens. On the status codes page: "If the content suggests an error for Google Search, an empty page or an error message, Search Console will show a soft 404 error."
How to fix it, by situation.
- Page gone, no replacement: "return a 404 (not found) or 410 (gone) response (status) code for the page."
- Page moved or replaced: "If your page has moved or has a clear replacement on your site, return a 301 (permanent redirect)."
- Page still exists but was flagged: it is "likely it didn't load properly for Googlebot, it was missing critical resources, or it displayed a prominent error message during rendering." Google points you to the URL Inspection tool.
Two things the pages do not say, so treat them as my choices: Google does not publish the phrases or word counts its systems use, and it does not say a redirect to the home page is a soft 404. I flag home page redirects because the home page is rarely "a clear replacement" for a removed product or post.
How the audit works
- Read the sitemap (or every child sitemap in an index) and collect each
<loc>. - Read
robots.txtand skip disallowed URLs. - Request a random URL like
/soft404-probe-3f9a1c2b7d4e. If the server answers 200, every missing URL on the site is a soft 404. Whatever it returns, keep its text as the site's not-found template. - Fetch each sitemap URL with a redirect handler that records the hops.
- Extract the title, the H1 and the main text, skipping
nav,header,footer,scriptandstyle, so menus do not count as content. - Compare the page with the not-found template using
difflib.SequenceMatcher, look for error phrases in the title, H1 and the first 600 characters of text, and count words. - Write a CSV with one row per URL and print the flagged ones by severity.
Only the title, H1 and opening text are searched for error phrases. Searching the whole page would flag a blog post that happens to explain what a 404 is.
What the script flags
| Issue | Severity | Meaning |
|---|---|---|
matches_not_found_page | high | 200 response with 90%+ the same text as the probe's not-found page |
error_message_200 | high | 200 response whose title, H1 or opening text says "not found", "no longer available", "no results found" and similar |
redirect_to_home | high | A non-home URL that ends up on / |
server_error / fetch_error | high | 5xx, or no response at all |
nearly_empty | medium | Under 50 words of main text and no error phrase |
search_url_in_sitemap | medium | A URL with q=, s= or search= in the query string |
gone_in_sitemap | medium | 404 or 410: correct for a removed page, but it should leave the sitemap |
redirected | low | Redirects somewhere other than home; list the final URL instead |
The site-wide probe result is printed at the top rather than per row, since it describes the server, not one page.
The script
Save as soft404_audit.py. Python 3.9 or newer, no packages to install.
#!/usr/bin/env python3 """Find soft 404s in a sitemap, standard library only. Usage: python3 soft404_audit.py https://example.com/sitemap.xml [--out soft404_report.csv] """ import argparse, csv, difflib, gzip, re, sys, uuid import urllib.error, urllib.request, urllib.robotparser import xml.etree.ElementTree as ET from concurrent.futures import ThreadPoolExecutor from html.parser import HTMLParser from urllib.parse import urljoin, urlsplit UA = "soft404-audit/1.0 (+standard library script)" NS = "{http://www.sitemaps.org/schemas/sitemap/0.9}" MIN_WORDS = 50 # main text shorter than this is "nearly empty" TEMPLATE_MATCH = 0.90 # similarity to the site's not-found page ERROR_PHRASES = [ "page not found", "not found", "can't find that page", "cannot find that page", "doesn't exist", "does not exist", "no longer available", "no results found", "0 results", "has been removed", "error establishing a database connection", "something went wrong", "internal server error", ] class Redirects(urllib.request.HTTPRedirectHandler): """Follow redirects but remember each hop.""" def redirect_request(self, req, fp, code, msg, headers, newurl): new = super().redirect_request(req, fp, code, msg, headers, newurl) if new is not None: new.hops = getattr(req, "hops", []) + [(code, newurl)] return new OPENER = urllib.request.build_opener(Redirects) def fetch(url, timeout=20): req = urllib.request.Request(url, headers={"User-Agent": UA}) req.hops = [] try: resp = OPENER.open(req, timeout=timeout) status, final, body = resp.status, resp.geturl(), resp.read(2_000_000) hops = getattr(resp, "hops", None) except urllib.error.HTTPError as e: status, final, body, hops = e.code, e.geturl() or url, e.read(2_000_000), None except Exception as e: # DNS, TLS, timeout return {"status": 0, "final": url, "body": "", "error": str(e)[:120], "redirected": False} if body[:2] == b"\x1f\x8b": body = gzip.decompress(body) return {"status": status, "final": final, "body": body.decode("utf-8", "replace"), "error": "", "redirected": final.rstrip("/") != url.rstrip("/")} class TextParser(HTMLParser): SKIP = {"script", "style", "noscript", "nav", "header", "footer", "template", "svg"} def __init__(self): super().__init__() self.title, self.h1, self.text = "", "", [] self._stack, self._in_title, self._in_h1 = [], False, False def handle_starttag(self, tag, attrs): if tag in self.SKIP: self._stack.append(tag) self._in_title |= tag == "title" self._in_h1 |= tag == "h1" def handle_endtag(self, tag): if self._stack and self._stack[-1] == tag: self._stack.pop() if tag == "title": self._in_title = False if tag == "h1": self._in_h1 = False def handle_data(self, data): if self._in_title: self.title += data elif not self._stack: if self._in_h1: self.h1 += data self.text.append(data) def parse(html): p = TextParser() p.feed(html) text = re.sub(r"\s+", " ", " ".join(p.text)).strip() return p.title.strip(), p.h1.strip(), text def read_sitemap(url, seen=None): seen = seen or set() if url in seen: return [] seen.add(url) r = fetch(url) if r["status"] != 200: sys.exit(f"Could not read sitemap {url}: HTTP {r['status']} {r['error']}") root = ET.fromstring(r["body"].encode()) locs = [e.text.strip() for e in root.iter(NS + "loc") if e.text] if root.tag == NS + "sitemapindex": out = [] for child in locs: out += read_sitemap(child, seen) return out return locs def probe_not_found(site_root): """Request a URL that cannot exist and see what the server does.""" url = urljoin(site_root, f"/soft404-probe-{uuid.uuid4().hex[:12]}") r = fetch(url) _, _, text = parse(r["body"]) if r["body"] else ("", "", "") return url, r["status"], text def check(url, robots, template_text): row = {"url": url, "status": "", "final_url": "", "title": "", "words": 0, "issue": "ok", "severity": "", "detail": ""} if robots and not robots.can_fetch(UA, url): row.update(issue="robots_blocked", severity="info", detail="Disallowed in robots.txt; not fetched") return row r = fetch(url) row.update(status=r["status"], final_url=r["final"]) if r["status"] == 0: row.update(issue="fetch_error", severity="high", detail=r["error"]) return row title, h1, text = parse(r["body"]) if r["body"] else ("", "", "") row.update(title=title[:80], words=len(text.split())) issues = [] if r["status"] in (404, 410): issues.append(("gone_in_sitemap", "medium", f"Returns {r['status']}; remove it from the sitemap")) elif r["status"] >= 500: issues.append(("server_error", "high", f"Returns {r['status']}")) elif r["status"] >= 400: issues.append(("client_error", "medium", f"Returns {r['status']}")) if r["redirected"]: if urlsplit(r["final"]).path in ("", "/") and urlsplit(url).path not in ("", "/"): issues.append(("redirect_to_home", "high", "Redirects to the home page, not to a replacement")) else: issues.append(("redirected", "low", f"Sitemap should list the final URL: {r['final']}")) if r["status"] == 200: haystack = f"{title} {h1} {text[:600]}".lower() hits = [p for p in ERROR_PHRASES if p in haystack] if template_text and text and difflib.SequenceMatcher(None, text, template_text).ratio() >= TEMPLATE_MATCH: issues.append(("matches_not_found_page", "high", "200 response with the same text as the site's not-found page")) elif hits: issues.append(("error_message_200", "high", f"200 response that says: {hits[0]!r}")) if row["words"] < MIN_WORDS and not hits: issues.append(("nearly_empty", "medium", f"Only {row['words']} words of main text")) if urlsplit(url).query and any(k in urlsplit(url).query for k in ("q=", "s=", "search=")): issues.append(("search_url_in_sitemap", "medium", "Internal search result URL listed in the sitemap")) if issues: order = {"high": 0, "medium": 1, "low": 2, "info": 3} issues.sort(key=lambda i: order[i[1]]) row.update(issue="; ".join(i[0] for i in issues), severity=issues[0][1], detail=" | ".join(i[2] for i in issues)) return row def main(): ap = argparse.ArgumentParser() ap.add_argument("sitemap") ap.add_argument("--out", default="soft404_report.csv") ap.add_argument("--workers", type=int, default=8) a = ap.parse_args() parts = urlsplit(a.sitemap) home = f"{parts.scheme}://{parts.netloc}/" robots = urllib.robotparser.RobotFileParser(urljoin(home, "/robots.txt")) try: robots.read() except Exception: robots = None urls = read_sitemap(a.sitemap) probe_url, probe_status, template_text = probe_not_found(home) print(f"Sitemap URLs: {len(urls)}") print(f"Not-found probe: {probe_url.replace(home, '/')} -> HTTP {probe_status}") if probe_status == 200: print(" WARNING: the server answers 200 for a URL that does not exist (site-wide soft 404).") with ThreadPoolExecutor(max_workers=a.workers) as pool: rows = list(pool.map(lambda u: check(u, robots, template_text), urls)) with open(a.out, "w", newline=Source:https://dev.to/appierun_66/find-soft-404-pages-in-your-sitemap-with-python-standard-library-only-error-messages-on-200-3cjg Related posts
Fuzzy-Matching Speech Recognition Output Against a Script with a Lookahead Window
What's Actually Inside an Enshrouded Save File (.ksd)

Short introduction to Rust for data science bachelor/master students

AI is redrawing the line between product and engineering - creating a competitive edge

Europe's biggest AI challenge isn't the technology. It's organizational psychology
