Eleven waits and a fast database
A screen took four and a half seconds to load and every query behind it ran in under ten milliseconds. The problem was not the database — it was asking eleven questions one at a time across an ocean.
We keep a rule that any page averaging over 300ms gets made faster, and a board that lists every screen against it. For several days the board had nothing to say, because nobody had opened anything since the last release. Then the owner spent a morning in the office and the dashboard came back at 4,561ms across twelve real loads.
The useful part is what happened next, because the obvious first move would have been wrong.
Measuring it locally said nothing was wrong
We run the same code in a test environment that sits physically next to the database. Measured there, the screen took 80 milliseconds. Every query behind it returns in single-digit milliseconds against real production data. By every measurement available on the machine doing the measuring, the screen was fine.
It is not fine, and the reason is geography. The code runs at the edge nearest whoever is using it — in this case London. The database lives in North America. So every single question costs roughly ninety milliseconds of ocean, and the number that matters is not how fast a query runs. It is how many times in a row the screen stops and waits.
Counting the waits
So we counted them, with a stand-in database that tracks how many questions are in flight at once. A question that starts while nothing else is running is a new wait; questions asked together collapse into one.
The dashboard asked twelve questions in eleven separate waits. Two of them were already being asked together. The other nine were asked one after another — and not one of them needed any of the others’ answers. Every piece of information they needed to run was worked out before the first one started.
The same count came back as eleven whether the account could see three kitchens or ten, which is worth noting: it was never a case of doing more work for bigger halls. It was eleven waits in a row regardless.
What changed, and what did not
Eleven waits became two. Same twelve questions, same SQL, same arithmetic afterwards — only the waiting is different. We proved the screen returns exactly the same answer by capturing the whole payload before and after against data that fills every card on it: identical, to the byte.
The honest caveat
We do not yet know what it did for the owner. He has not opened that screen since the release went out, so the before-and-after that matters does not exist yet. The local number moved from 80ms to 52ms, which is about one round trip in a place where round trips are cheap, and therefore not the measurement anybody should quote.
If you are evaluating any system that runs in the cloud: ask where its database is, and ask how many times a screen waits. A vendor who can answer the second question has thought about it. A benchmark run next to the database is not a benchmark of what your staff will experience.