The filler page your sitemap told the crawler about
A rival sitemap lists thirteen pages whose addresses are Latin filler left by the site builder. A sitemap generated from the route table cannot leak one.
Among the 120 addresses in one rival’s sitemap on Thursday were ten under a knowledge-base path and three under an order-and-pay path whose slugs are Latin filler: the kind of placeholder text a site builder drops into a template so the layout has something to show. Somebody published the template pages along with the real ones, and the sitemap dutifully announced them to every crawler.
Nobody meant to. A placeholder page is created in a few seconds and then it is one of a hundred rows in a content tool, indistinguishable from a page that matters until you read its address. The sitemap did not check. It listed everything the tool had.
Where our sitemap gets its list
Our site is generated. The pages are described in a handful of source files, built into static HTML, and the sitemap is written in the same step from the same list. A page that is not in the route table is not on the site, so it cannot be in the sitemap either. There is no content tool with a drafts folder for a template page to hide in.
That is a structural answer, and structural answers are the ones that keep working when nobody is looking. The weak version would have been a rule in a checklist: do not publish placeholder pages. The rival clearly had a version of that rule and it did not help, because the step that produced the sitemap never consulted it.
The check behind the structure
Two tests sit behind it anyway. One asserts every address in the sitemap we wrote answers with a real page on the running worker, and the other asserts every page the worker routes is in the sitemap unless its own head says not to index it. Between them, a page that exists but should not be announced, and an announcement for a page that does not exist, both fail the build by name.
It is a small thing to find in somebody else’s sitemap, and it is not a thing we would correct them on. It is worth writing down because the same mistake was one checkbox away in any content tool, and the reason we cannot make it is not discipline. It is that the list and the site are the same file.