The check that locked itself out

We wrote a check this week that does something blunt and useful: it calls every single write operation the product has — four hundred and two of them — with an empty request, and reports any that fall over rather than politely refusing.

The first run came back clean. Four hundred and two fired, nothing broken. That is exactly the result you hope for, and it was very nearly a lie.

What it was actually measuring

The run signs in once at the start. Somewhere near the front of the list is the operation that signs you out, and when the check reached it, it duly logged itself out. Every one of the three hundred and fifty-four operations after that was turned away at the door without ever reaching the code being tested.

So the comforting "nothing broken across four hundred" was a measurement of forty-eight. The other eighty-eight per cent of the run proved only that our sign-in check works.

It was caught because the run printed a tally of what each call answered, and asked one question at the end: am I still signed in? It was not.

This shape is everywhere, and it is not about software

A check that cannot reach the thing it checks returns the same answer as a check that reached it and found nothing wrong. Once you have the shape in your head you see it constantly:

• A stock count that skips the locked storeroom and reports no discrepancies.
• A supplier price review where three of the eight suppliers never replied, and the summary says prices are stable.
• A rota check that only looks at staff who have clocked in, and concludes everybody is where they should be.
• A complaints report that counts the ones logged in the system, in a week when the logging screen was down.

In every case the number looks like evidence and is actually a description of the sample. The question that breaks the spell is the same one every time: how many did this actually look at, and how many did it mean to?

What we changed

The check now signs itself back in before every single call, counts how many were refused, and fails outright if that number climbs — with the lock-out check placed before the others, so a run that locked itself out says so rather than blaming the first odd thing it trips over.

Then we went further in the same direction. With the door open, the run was still shallower than it looked: most of those operations take an identifier, and our test database had almost nothing in it, so they answered "no such thing" and their real work never ran. Seeding one row of each kind — an order, a recipe, a supplier, a staff member — took the number of operations that got properly under way from a few dozen to two hundred and sixty-eight.

Still nothing broken. But now that means something.

The small print that makes it hold

Both improvements can rot silently, so both are guarded. If a seeded row is ever refused — because a table changed shape — the check stops and names it, instead of quietly running shallower. And each half asserts a floor on how many operations got far enough to succeed, so a fixture that collapsed could not leave everything passing on refusals.

Any check worth having should be able to answer two questions: what did you find, and what did you actually look at. The second one is the one that gets skipped.