Testing a fix where the problem cannot happen
A change that saves ninety milliseconds in the venue saves five on the bench, and the bench is where most people check. No difference is not the same as no improvement.
There is a particular way of being wrong that feels like being careful. You make a change, you test it, the test shows no difference, and you conclude the change did nothing. Sometimes that is exactly right. Sometimes you tested it somewhere the problem does not exist.
A worked example from this week, and then the general version, because the general version is the useful bit.
Ninety milliseconds that were not there
Our database lives in the eastern United States. Our venues are in England. Every time a page needs something from the database, that question and its answer cross the Atlantic — call it ninety milliseconds, there and back, and that is just physics.
The machines we write and test our code on happen to sit next to the database. Same building, more or less. For them, the same question and answer take about five milliseconds.
So when we removed one of those journeys from a page this week, we measured it before and after on our own machines. Before: twenty-four milliseconds. After: twenty-three. Within the noise. Nothing.
Except we had just removed a ninety-millisecond journey from a page that is opened in Winchester, not in Virginia. The saving is real. It simply cannot show up on a machine for which that journey costs five milliseconds instead of ninety.
The general version
The trap is that "I tested it and saw no difference" sounds like evidence and often is not. What it tells you is that the change made no difference under the conditions of the test, which is only interesting if those conditions resemble the ones you care about.
You will recognise the food-hall version of this immediately.
A new prep layout, trialled on a Tuesday afternoon, shows no improvement. Of course it does not. The layout is meant to help when three people are working in that space at once and there is a queue out the door. On a quiet Tuesday one person has the whole bench, and every layout is a good layout.
A new ordering flow, tested by the owner on their own phone on the venue's wifi, feels fine. The guest it was built for is on a crowded network at a festival with two bars of signal and a cracked screen.
A new kitchen ticket format, judged by reading it at the pass in a calm moment, looks clear. It has to be readable at a glance by somebody who is already behind.
In every case the test was honest and the conclusion was wrong, because the thing being fixed was not present at the time of the test.
What to do when you cannot reproduce the conditions
Sometimes you can just go and test it properly. Run the layout trial on a Saturday. Borrow a colleague's older phone. Judge the ticket during service.
When you genuinely cannot — and we cannot move our office to Winchester for an afternoon — then measure something that does transfer. We could not measure the seconds, so we counted the journeys. Two became one. The count is the same number wherever you stand; only its price changes with distance. Then we said out loud what we had and had not measured, rather than reporting a saving we had not seen.
That is the discipline. Find the part of the improvement that is true everywhere, measure that, and be explicit about the part you are inferring. An honest inference clearly labelled is worth far more than a confident number from the wrong bench.