When a report says nothing, suspect the report

This product runs twelve jobs on a schedule. Some every minute, some once a day, one on a Monday morning, one on a Sunday. They probe whether every service is up, they tell a kitchen an order is waiting when nobody is watching the board, they nudge staff for their availability, they push scheduled orders onto the pass when their collection time approaches.

Two of them have previously turned out to be broken in ways nothing noticed, and both times the discovery came from the same thing: actually running the job in a test, rather than checking that the code mentions it.

So the obvious question was which of the twelve had never been run. A scan was written. It reported that none of them had, including two that had definitely been run, because that work was recent and documented.

Believe the surprising result last

A result that says nothing is covered is exciting. It is also the same result you get from a scan that cannot see anything.

The scan was looking for a job being called by its plain name. Every one of these tests imports the file and calls through it — one extra word in front of the function. The scan was blind to all of them, and reported a confident zero.

That zero is indistinguishable from a real catastrophe. If not a single job had ever been tested, the output would have looked exactly the same. The only reason the mistake got caught is that the answer contradicted something already known to be true.

The second thing it got wrong

It also counted nine jobs, not twelve. Three of them have no name at all — they are written inline, as anonymous blocks hung directly off the schedule. A scan looking for named functions cannot see a job that has no name. It did not report them as uncovered; it did not report them.

Under-reporting is the worse failure of the two. An uncovered job you can go and cover. A job your report has never heard of is invisible.

All twelve were fine

Every single one already had a test that runs it. Somebody had done that work job by job, over weeks, and left a note inside each test saying which number it was — one of them opens with the observation that eleven of the twelve already had one. The seam was closed before anyone came looking.

Which meant the finding was not a bug. It was that nothing stops a thirteenth job being added tomorrow without a test that runs it. Two existing checks watch the schedule: one proves each job gets handed to the scheduler at the right minute, against a database rigged to fail every call — routing, not behaviour. Another proves each job leaves a trace somebody can go and read. Neither asks whether the job works.

So the useful output of an hour that found no bug was a check that makes the current state stay true: every job the schedule dispatches must name a test that runs it. To prove it works, a thirteenth job was inserted into the live schedule and the check failed by name, then removed.

For anyone who runs on checklists

A close-down checklist, a food safety log, a cash-up sheet — all of them are reports, and all of them can quietly stop covering what they were written for. Two questions are worth asking of any of them.

First: if this had gone badly, would this sheet look different? A temperature log where every reading is in range might mean the fridges are fine, or might mean the readings are copied from yesterday.

Second: what does this sheet not have a line for? The item that fell off the checklist when the menu changed is not marked as missing. It is simply absent, and absence does not attract attention the way a failed line does.