Zero errors, and the job had failed: reading what a report actually says

We ran a long job across our whole product: translate every fixed piece of interface text into every language we offer. It ran for the best part of an hour and finished without a single error.

It had also failed.

The line that was the bug report

The output looked like this, once per language:

de: 1423/2080 done, 657 still missing, 0 errors
zh: 1449/2080 done, 631 still missing, 0 errors
pt: 1998/2080 done,  82 still missing, 0 errors

Read it quickly and it is a success: no errors, thousands of items processed. Read it properly and every language is short, some by a third, and nothing anywhere says why.

The shape of the numbers was the clue. Items were being processed in groups of sixty. 657 is about eleven groups. 120 is exactly two. 60 is exactly one. Whole groups were vanishing, not scattered items.

What was underneath

Each group came back as a list, matched to the list we sent by position. There was a check: if any answer in the group was blank, or had lost a placeholder like a price, refuse it — because a price landing in the wrong half of a sentence is worse than a sentence left untranslated.

That check is right. It was being applied to the group instead of to the item. One bad answer in sixty threw away the other fifty-nine, silently, and the count of what was still missing came back honest because as far as the code was concerned nothing had gone wrong.

We fixed it, re-ran, and it happened again — the same shape, smaller. A second guard had the same disease: if a group came back with the wrong number of answers, the whole group was discarded. That one has to refuse the answer, because a list off by one puts every label under its neighbour. But the group is not wrong; one answer is. It now asks again for each half, and keeps halving until the answers line up.

The operator's version of this

You have jobs like this. A stock import that says "412 rows imported" and does not say that 38 were skipped. A payout run that completes and leaves two kitchens out. An end-of-day that balances because the thing that did not balance was dropped before the sum.

Three habits are worth having:

• Read the second number. "Done" is not the interesting figure. "Attempted minus done" is.
• Look at the shape. Losses in round multiples are structural. Scattered losses are usually data.
• Be suspicious of clean failures. A job that fails loudly gets fixed on the day. A job that fails quietly gets fixed when somebody notices the consequence, months later.

Ours would have looked finished. Twenty-six languages, tens of thousands of rows, no errors — and a third of German missing on every screen.