Engineering · Reliability · Back office

The helper kept every failure, the callers kept one

Save ten rows, two refused. The pool collected both; three screens threw all but the first away and showed one bare message. Now they say how many landed.


Several back-office screens save many rows at once: every printout type for a kitchen, a page of suggested par levels, a batch of products given a new VAT rate. They go through a small pool that keeps a few writes in flight and, crucially, keeps going when one fails. The pool returns every success and every failure, because the alternative is a screen that stops at the first error and leaves nobody knowing how many writes landed. The comment above it says so, in those words.

Then three of its seven callers took the result and threw the first failure, discarding the rest. One failure reported, every other one gone, and the screen's single error handler showing a bare message. The pool was right. The callers reproduced the exact behaviour it exists to avoid, one line after calling it.

Driven, before and after

We forced the same failure both times: a kitchen's printout settings, two writes, one refused because the hall owner had locked that printout, the other allowed through. Before, the counter saw the lock message and nothing else. It was not told that the other printout had in fact saved, although that fact was sitting in the result the whole time. After, the message reads one saved, one refused, names the kitchen, and gives the reason.

Each screen now reports with what it actually has. The printouts save and the par suggestions carry a name per row, so they name the refused ones. The par editor's rows carry a product id and no name, so it reports the count and the reason, which is still far better than one bare message with nine silent successes behind it. These are settings writes, so nothing was corrupted and a retry is harmless. It is still a silent partial, and a silent partial is worse when the information was already in hand.

A test was asserting the bug

The uncomfortable part. A test we wrote five releases earlier pinned the throw at all three sites, with the reasoning that those screens report through one handler and re-enable their button there. They do, with one message. Writing the shortcut down as intended behaviour is what kept it. The test now says the opposite and generalises: no caller may throw away the failures the pool collected, paired with a check that each still says how many landed, so that no throw cannot be satisfied by saying nothing at all.


Try it on tonight’s service.

Nothing to install, no card. Not better by the weekend? Close the tab.