The line that fenced nothing and broke everything
A single line was added to a purchasing screen to scope its data to the right company. It was never finished: the value it computed was never used, and the thing it read was not available. The screen answered an error to everybody for a day.
Two screens in our back office are where a manager decides what to buy. One lists every supplier with the things you are short of; the other turns that into an order pack you can send. Both are built from the same piece of data.
For a day, both showed nothing at all. Not an empty list — nothing. The page furniture drew, and where the content should have been there was a gap.
What had happened
The day before, somebody — us — had set out to make that data respect company boundaries properly. A hall can hold kitchens belonging to different businesses, and a supplier list is a business's own. The change needed two halves: work out which company is asking, then use that when reading the rows.
Only the first half was written. The line that worked out the company was added at the top of the function; the queries underneath were left exactly as they were. So the calculation was performed and then discarded.
Worse, the thing it needed in order to do the calculation — who is asking — was not something that function had been handed. Every other function of its kind takes it as an argument. This one did not, and nobody noticed, because the line looked exactly like the working ones a few pages up.
The result was that the very first statement in the function failed. Not the query, not the data: the first line. Everything after it never ran, and the screens got an error instead of an answer.
Why it survived a day
Our tests were green the whole time. They are green now. They had no opinion about that screen because nothing had ever called it — not one test in several thousand. It is easy to end up here: the parts of a system that get tested hard are the ones somebody was nervous about, and a purchasing list that has worked for a year is not one of those.
It was found by opening the screen. Not by reading the code, not by a clever check — by loading every back-office page in a browser, one after another, and looking at what came up. Two of a hundred and five were dead. The other hundred and three were fine, including three that looked broken and were telling the truth: a chart with no recipe chosen, and two staff screens that correctly say "you are not on the team here" to an owner who isn't.
The half-written change is the dangerous one
If that scope check had not been started at all, the screens would still work and the data would still be too wide. If it had been finished, the screens would work and the data would be right. The state in between was the worst of both: the protection was absent and the feature was down.
There is a habit worth taking from this, and it has nothing to do with code. When you change a process in your own business — a new sign-off on purchase orders, a new step at close — the risky moment is not the old way or the new way. It is the morning when half the team has the new instruction and the sign- off exists but nothing enforces it. Finish the change or don't start it; the halfway state usually costs more than either end.
What we did about the general case
The fix itself took ten minutes. What took longer was making sure the shape cannot come back quietly: there is now a check that fires every route in the product — all of them, in about a second — and fails if any of them answers an error instead of an answer. Reintroducing the bug on purpose makes it fail by name, which is the only way to know a check is doing anything.
Two screens were down for a day. The cheap thing was never the fix. It was having something that opens every door once in a while.