Engineering · Reliability

The warning that did not say which door

Two comments said an API rejects a schema keyword. A third file still used it. We nearly removed it there before checking which API door the comments meant.


One night this week we read the error log on the live site instead of the code, because the previous two fixes had both been real faults on paths no screen can currently reach, and a fault nobody can hit is a weak use of an evening. The log is the thing our Errors screen shows and the thing nobody reads at four in the morning, which made it the obvious place to look.

Thirty unresolved rows, a hundred and six hits, nothing newer than three weeks old. Every specific one we checked was already fixed: a transient database error that was infrastructure, a count of twenty-two values for twenty-three columns on staff contracts now counted by hand at twenty-three, a null in a required menu column now coalesced, a webhook secret that is correctly absent because no card processor is live. Each is now recorded in a test with its reason, so nobody chases it again.

The one that nearly shipped

One row said the structured-output side of our AI provider had rejected a request because the schema used a keyword it does not support. The two files that had hit that error carried a comment each saying so, and had dropped the keyword. A search of the code found a third file still using it. The obvious fix was to drop it there too, and it was half typed.

The third file does not talk to the structured-output side at all. Its schema describes a tool the model may call, sent through a different field of the same request, and on that side the keyword is ordinary and supported. The error on the live site named the exact field it came from. The two comments did not: they said the API rejects it, without saying which door of the API, and a reader following them to the third file would have removed a working constraint from a working schema and recorded a fix.

Name the surface, not the vendor

Both comments now say which field refuses the keyword and which accepts it, and the test that keeps the fixed errors from being re-chased checks that the third file keeps its constraint. The habit it points at is small and worth having: when you write down that something rejects something, write down the exact place it was rejected. A warning that names the whole vendor reads as true everywhere, and the one place it is wrong is the place the next person will go first.

The same night, a scan we wrote to find unhandled promises in the office code reported seven and six were wrong, because it took each statement to end at the first semicolon and the first semicolon sat inside a callback eight lines above the catch it was looking for. Both mistakes have the same shape: a tool that reads a boundary where there is none and trusts what it finds. Checking the instrument before acting on its reading cost twenty minutes. Acting on it would have cost a working feature and a false entry in the record.


Try it on tonight’s service.

Nothing to install, no card. Not better by the weekend? Close the tab.