A crawler is not a customer
The first day of our load-time board filled with nine marketing pages at one visit each, all around a second, hours apart. Nobody reads a site like that. A search engine rendering it does, and its numbers would have set every page’s average. How we tell them apart, and the trap our own tests fell into.
A board that ranks pages by how long real visitors waited has one quiet enemy: things that are not visitors. On its first day ours filled with nine different marketing pages, one visit each, spread over ten hours, every one around a second. That is not how a person uses a website. It is exactly how a search engine does — it renders each page it finds, with a real browser, from a render farm that is in no hurry. And a real browser runs our timer like anyone else's, so the render farm's second became the page's average.
Left alone, that number would have been the top row of a board the owner reads to decide what to fix next. So the timer's endpoint now looks at who is sending. Search engines, previewers, link-unfurlers and uptime checkers name themselves in the request, and we already had a classifier for that — the visit counter uses it — deliberately broad, because a robot mistaken for a person flatters every number and a person mistaken for a robot merely goes uncounted. A crawler's timing is still kept, under its own label, so a quiet board is known to be quiet rather than blind. It never reaches the board, and the board says how many it kept out.
The trap in our own tests
The proof of that change ran in a headless browser, the kind automated tests use. Headless browsers say so in their name, and the classifier counts them as robots. So the test loaded the dashboard, read the board, and found its own load filed as a crawler — the number said three renders kept out when the test had caused two. Right for production, and a trap for anyone measuring locally: every local check that reads the board now sets a real phone's browser name on the way in, or its loads vanish and the board looks empty. That is written in the place the next engineer will read before running one.
What the board can say now
Whether the first day's nine visits were a crawler is not knowable — those rows carry no name and age out in a week. But within hours of the change the crawler label had its first rows: a blog post at two seconds, a venue page at under one, both from a render farm, both off the board. The question the pattern raised answered itself, and the board went back to being about people.