How to audit every URL in your sitemap for on-page SEO issues (no headless browser)

Disclosure: I build and maintain the Apify Actor used in this post (published as Syntrovia Data). It is a paid, pay-per-event tool; the prices are listed below. This post was drafted with AI assistance, and every input, output field and price was checked against the Actor's README.

A sitemap tells search engines which URLs you care about. It does not tell you whether those URLs still answer 200, whether they redirect, or whether half of them ship without a meta description. Desktop crawlers handle one-off audits well. This post covers the other case: a repeatable check that runs in the cloud, reads robots.txt and the sitemap first, and gives you one row per page that you can compare week to week.

The tool is Sitemap & On-Page SEO Checker - HTTP Only, No Browser on Apify Store. It sends plain HTTP GET requests, so runs are cheap, but it does not run JavaScript.

What it checks

Before it requests a page, the Actor:

• reads robots.txt and follows its rules and Crawl-delay;
• finds the sitemaps (the Sitemap: lines in robots.txt, or /sitemap.xml and /sitemap_index.xml), including sitemap indexes up to 3 levels deep, gzip files and plain-text sitemaps;
• sends at most 2 requests at a time and 60 requests per minute by default.

For each page it records the status code, redirect target, title and meta description (with their lengths), H1, canonical, robots meta tag and X-Robots-Tag, word count, link and image counts, and a list of issue codes. There are more than 35 codes, for example missing-meta-description, duplicate-title, canonical-points-elsewhere, redirect-chain, noindex, thin-content and not-in-sitemap.

It stops when a site answers 401, 403, 407 or 429, or shows a CAPTCHA or a bot check instead of the content. A single login page is skipped and not charged; if most pages ask for a login, the site sits behind a login wall and the run stops with stopReason: login-wall. Use it on sites you own or are authorised to audit; the input has a checkbox for exactly that.

Step 1: list the sitemap first (cheap)

Start with sitemap mode. No page is requested; you only get what the sitemaps say.

{ "startUrl": "https://www.example.com/", "authorizedToAudit": true, "mode": "sitemap", "maxSitemapUrls": 20000 } 

Each row has url, sitemapUrl, lastmod, changefreq, priority, allowedByRobots and issues. The sitemap-mode codes catch problems that are easy to miss by eye:

• blocked-by-robots: the sitemap lists a URL that robots.txt disallows;
• other-host and insecure-http: URLs on another host, or plain-http URLs on an https site;
• invalid-lastmod, invalid-changefreq, invalid-priority.

Listing 5,000 URLs costs $0.005 + $1.50 = $1.505.

Step 2: audit the pages

Switch mode to audit and cap the run with maxPages:

{ "startUrl": "https://www.example.com/", "authorizedToAudit": true, "mode": "audit", "maxPages": 500, "discoverFrom": "sitemaps-and-links", "includePageText": true, "maxRequestsPerMinute": 60 } 

A result row looks like this (trimmed from the example in the README):

{ "url": "https://www.example.com/pricing", "statusCode": 200, "foundIn": "sitemap", "inSitemap": true, "title": "Pricing - Example", "titleLength": 17, "metaDescription": null, "h1": "Pricing", "h1Count": 1, "canonicalStatus": "self", "indexable": true, "indexability": "indexable", "wordCount": 481, "images": 9, "imagesWithoutAlt": 2, "issues": ["title-too-short", "missing-meta-description", "images-missing-alt"] } 

A redirect gets its own row, with statusCode 301 and the target in redirectUrl. A same-site target is checked as a separate row, which is how redirect-chain and redirect-loop are found.

One more code worth knowing: not-in-sitemap. It marks an indexable page that the crawl reached through links but that no sitemap lists. It is only set when every sitemap was read to the end, so a partial read does not produce false alarms.

Step 3: pull the results and group them by issue

You can download JSON, CSV or Excel from the Apify Console. For scripts, the Apify API has an endpoint that runs the Actor and returns the dataset in one request. It waits up to 300 seconds, so keep maxPages small for this call: at 60 requests per minute, 100 pages take roughly two minutes.

curl -s -X POST \ "https://api.apify.com/v2/actors/syntrovia~sitemap-seo-check-http/run-sync-get-dataset-items" \ -H "Authorization: Bearer $APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{"startUrl": "https://www.example.com/", "authorizedToAudit": true, "mode": "audit", "maxPages": 100}' \ -o audit.json 

Then count the issues and list the pages to fix first:

import json from collections import Counter with open("audit.json", encoding="utf-8") as f: rows = json.load(f) counts = Counter(code for row in rows for code in row["issues"]) for code, n in counts.most_common(): print(f"{n:5} {code}") print("\nIndexable pages without a meta description:") for row in rows: if row["indexable"] and "missing-meta-description" in row["issues"]: print(" ", row["url"]) 

Each run also writes a RUN_SUMMARY record to its key-value store: totals per issue, the robots.txt rules it applied, the sitemap files it read, and why it stopped (max-pages, all-urls-checked, blocked and so on).

Step 4: make it weekly

Save the input as a Task, add a Schedule (for example every Monday at 07:00 in your time zone), and send the results on with an Apify integration such as Slack, e-mail or Google Sheets. Every Monday you get the same columns, so comparing with last week shows new 404s, new redirects and pages that lost their canonical.

What it costs

Pay-per-event with three events; platform usage is included.

The prefilled 20-page audit costs $0.035, and 1,000 pages cost $1.505. You are not charged for URLs that robots.txt disallows, for requests that got no answer, or for a run that saves nothing. You can also set a maximum cost per run.

When it is the wrong tool

• Pages built with JavaScript. The Actor reads the HTML your server sends. Titles, meta tags, canonicals and status codes are usually in it; body text built in the browser is not, and those pages get the javascript-rendered code.
• Very large sites in one run. One audit run handles up to 5,000 pages. With the default 1-hour timeout and 60 requests per minute, that is about 3,400 pages unless you raise the timeout.
• Sites you do not control. It is meant for your own sites and for clients who asked you to audit theirs.

Try it

Open the Actor page, keep the prefilled example or enter your own site, tick the authorisation box only if it is true, and press Start. If an issue code is unclear or a check you need is missing, open an issue on the Actor's Issues tab. I aim to answer within 72 hours.