The time scaled with the distance

The guest ordering page got a row of its own on our speed board this week, and the first morning it had three readings they put it at the top of the list: over four seconds from scan to menu, two and a half of them spent inside the server building one response. That response is the menu. It is the first thing a guest's phone asks for and the one surface in the product that must come up quickly, so this was the job before anything else.

The question was what kind of slow it was, because the fix for a slow query and the fix for a slow sequence are different fixes, and the number on the board does not say which.

Stand in four places

We timed the same call six times, reading the server's own clock in each reply, from wherever our requests happened to leave. Beside the database, in the eastern United States, it took 0.4 seconds. From London, 2.3. From Frankfurt, 2.8. From Madrid, 2.9. Same venue, same six kilobytes, same code. The only thing that changed between the readings was how far the server stood from the database, and the time scaled with that distance by a factor of seven.

A slow query cannot do that. A query takes what it takes wherever you ask from, plus one trip there and back. What scales with distance is the number of trips, and a factor of seven on a round trip of ninety milliseconds pointed at something in the order of twenty-five of them, one after another.

Twenty-seven, in a line

Reading the route confirmed it: twenty-seven reads, each waited for before the next began. The venue row first, then the hall, then what sold this month, then the dishes, the menus, the allergen roll-ups, what is marked off, the sections, the deals, the pools, the modifier groups, the colours, the tips, the notices, the reviews, the table plan, the hall's charity, the hygiene rating. Twenty-five of them needed nothing but the venue's id, which the very first read had answered. Each one was a crossing of the Atlantic for a guest standing in Winchester, made because the previous one had come back, not because it needed what came back.

The route even had a comment beside its one existing batch of parallel reads saying a guest menu must never grow a serial fetch. Four reads later the chain of twenty-seven began. The warning was right and nobody had read it since.

Two waits

Now the venue is read first, because everything is keyed on it, and then everything else is asked for at once. Two reads used to depend on an earlier answer and were moved into the single wave by letting the database do the choosing: all of a venue's menu overrides come back in one statement and the chosen menu's are picked out afterwards, and the published reviews are read only where the venue has switched publishing on, decided inside the query rather than by a first read and an if. The arithmetic after the wave is unchanged, in the same order. Only the waiting moved.

Before touching anything we captured seven menus from a seeded database, including a hall order, a table, a named menu and a code that does not exist. After the change the same seven came back identical to the byte. Then the test that counts how deep the waiting goes was written against the old route, where it failed naming twenty-seven, and passes now at two.

Measured again from the same four cities: London 0.42 seconds, Frankfurt 0.26 to 0.45, Madrid 0.51, and beside the database 0.10. Six times less from Europe. Not yet under our own 300 millisecond rule, and we are not claiming it is; the next step is a different kind of batching and belongs to another release. What a guest's phone reads on the new version will show up on the board over the coming days, and the board, not this post, is the record.

For anyone running a floor

When something is slow, ask it from two distances before you decide what is wrong with it. If it is slow by the same amount from everywhere, the work itself is slow. If it gets worse the further away you stand, it is making too many trips, and no amount of speeding up each trip will fix a route that makes twenty-seven of them.