Measure from where your customer stands
Our test machine sits beside the database and flatters every route. Today it routed through Europe and its numbers matched the customer for once.
Our database lives in the eastern United States. Our customers are in the south of England. The machine we test from has, for weeks, sat in a data centre beside the database, which means every route we time from it comes back looking quick. A read that costs a customer ninety milliseconds costs that machine ten. Our notes carry a standing warning about this: the shape of a timing transfers, the seconds do not.
This morning the test machine's connection happened to leave through Frankfurt, Paris, Amsterdam and London instead. For the first time its numbers were the customer's numbers, and the route we had just fixed read almost exactly what the arithmetic had predicted.
What the two views show
From beside the database, the way in to every route costs about fifteen milliseconds and the route itself a few tens more. You can see the shape of a problem from there, how many round trips a route makes, whether they run together or in a line, but you cannot feel it. Three trips in series reads as forty milliseconds, and forty milliseconds is nothing to fix.
From Europe, the same three trips read as 285 milliseconds, because each one is a crossing. That is the figure our owner's board had been showing for the permissions screen, and it is the figure that made it a job. Same route, same code, same database. The only thing that changed was where the question was asked from.
Why we did not just fix the test machine
Because the shape view is useful too. Counting round trips is a question you can answer anywhere, and a test that runs the route against a database that records every call does it without a network at all. What we do not do is take a speed figure from beside the database and call it the customer's experience. The board the owner reads is built from real browsers in real venues, and that is the number the 300 millisecond rule is enforced on.
The same mistake on a shop floor
The QR menu is tested on the office wifi, standing next to the router, on a phone that has loaded it fifty times today. It feels instant. The guest is at a table at the far end of the hall, on a mobile signal shared with two hundred other phones, loading it for the first time. Two completely different measurements of the same page, and only one of them is the one that decides whether the guest orders or gives up.
Test from the far table. Test on the first load. If your number comes from beside the thing being measured, you have the shape of the truth and none of its weight.