There is a quiet set of URLs that the web expects a site to have an opinion about. Some are specs, some are conventions, some are vendor files. Nobody asks you to serve all of them. The web does assume that when it asks, you answer.
We asked pub-trivia.app for 27 of them.
Four exist
| path | status | content type | produced by |
|---|---|---|---|
/robots.txt | 200 | text/plain; charset=utf-8 | app/robots.ts |
/sitemap.xml | 200 | application/xml | app/sitemap.ts |
/manifest.webmanifest | 200 | application/manifest+json | app/manifest.ts |
| the IndexNow key file | 200 | text/plain; charset=utf-8 | a route handler |
All four are things we decided to publish, and three of the four exist because Next has a file convention for them, which is the actual reason they got written. The manifest is nine lines:
{"name":"PubTrivia","short_name":"PubTrivia", "description":"Live pub quiz nights, scan, join, play!", "start_url":"/","display":"standalone", "background_color":"#171717","theme_color":"#171717", "icons":[{"src":"/icon.svg","sizes":"any","type":"image/svg+xml","purpose":"maskable"}]} The sitemap has 72 URLs, 69 with a lastmod taken from a date we wrote down by hand and 3 taken from the build clock, which is why three of them carry milliseconds.
Twenty-three answer a redirect
Everything else on the list behaves identically:
$ curl -s -o /dev/null -w '%{http_code} %{redirect_url}' \ https://pub-trivia.app/.well-known/security.txt 307 https://pub-trivia.app/login?next=%2F.well-known%2Fsecurity.txt The full set that does this: /.well-known/security.txt and /security.txt, /.well-known/change-password, /.well-known/apple-app-site-association, /.well-known/assetlinks.json, /.well-known/openid-configuration, /.well-known/dnt-policy.txt, /.well-known/gpc.json, /.well-known/traffic-advice, /.well-known/host-meta, /.well-known/matrix/server, /humans.txt, /ads.txt, /app-ads.txt, /llms.txt, /llms-full.txt, /feed.xml, /rss.xml, /atom.xml, /feed, /sitemap_index.xml, /browserconfig.xml, /crossdomain.xml, /site.webmanifest, and a search engine verification file name we made up.
The reason is structural rather than interesting: anything not sitting under one of our public path prefixes is sent to the sign-in page by the authentication gate, and a gate working on prefixes has no way to know which paths are merely absent. A missing page under a public prefix 404s properly. A missing file at the root does not.
The redirect even carries cache-control: public, max-age=0, must-revalidate, content-type: text/plain and x-content-type-options: nosniff, which is a lot of ceremony for a response whose body is empty.
Most of those 23 cost nothing. We are not an advertising publisher, so ads.txt is correctly absent. We have no native app, so the two deep-linking files are correctly absent. crossdomain.xml is a Flash artefact. dnt-policy.txt describes a header browsers stopped sending. Answering those with a 404 instead of a 307 would be tidier and would change nothing.
Three of them are things we should probably have, and for the usual reason of never having got round to it: security.txt because there is no documented way to report a problem to us, llms.txt because our content is 67 pages of plain prose about running a quiz night and that is exactly what the crawlers reading it want, and an RSS feed because we publish dated content and currently nobody can subscribe to it.
But one of the 23 is different, and it is the point of this post.
The change-password spec has a trap, and we are in it
Change Password URL is a tiny W3C spec. A site serves /.well-known/change-password, redirecting to wherever its real change-password screen is, and a password manager can send a user straight there instead of making them hunt through a settings menu. Chrome and Safari both use it.
The spec authors knew that a lot of sites answer every unknown path with a 200 or a redirect to something generic, which would make the convention useless. So the spec defines a companion probe with a deliberately absurd name:
/.well-known/resource-that-should-not-exist-whose-status-code-should-not-be-200 A client is required to fetch that first. If it comes back 200, the site is lying about its well-known URLs and the client must ignore them. Here is ours:
$ curl -s -o /dev/null -w '%{http_code}\n' \ "https://pub-trivia.app/.well-known/resource-that-should-not-exist-whose-status-code-should-not-be-200" 307 $ curl -sL -o /dev/null -w '%{http_code} %{url_effective}\n' \ "https://pub-trivia.app/.well-known/resource-that-should-not-exist-whose-status-code-should-not-be-200" 200 https://pub-trivia.app/login?next=%2F.well-known%2Fresource-that-should-not-exist... 307, then 200. A client following redirects, which this one does, gets a 200 from a path that by name cannot exist. So we fail the probe, and any password manager checking us concludes that our well-known URLs are not to be trusted.
The consequence is the part worth sitting with. If we added a correct /.well-known/change-password tomorrow, it would still be ignored, because the probe would still answer 200 first. The feature is not blocked by the missing file. It is blocked by the catch-all redirect two layers below it, and no amount of adding files fixes that.
It is a nice illustration of a general thing: a catch-all that answers something friendly instead of 404 feels harmless, and then one day a spec that depends on 404 meaning 404 quietly stops working, and the symptom shows up nowhere in your logs.
What we are going to do
In order of effort:
- Make unknown root-level paths 404 instead of redirecting. This is the one that actually matters, because it fixes the probe, and it fixes every future convention that depends on absence being legible.
- Then add
security.txt, since once the probe passes it is a four line file. - Then
/.well-known/change-password, pointing at our existing reset flow. -
llms.txtand a feed, when there is a reason beyond tidiness.
Reproduce it
Every command above runs against the public site with no account. The two worth running are the probe with and without -L, which is the whole finding in two lines, and:
curl -s https://pub-trivia.app/robots.txt which is one of the four that does answer, and tells you which parts of the site are meant for crawlers. The pages those crawlers are pointed at, the guides and the comparisons, are all public too; signing up is free and needs no card, but nothing in this post requires it.
