Security · Reliability · Engineering

The error that quoted the database back at you

An unexpected fault answered with the raw database statement and its values, on a route open to anybody. A chosen sentence stays; an accident gets one line.


We were testing something else when the answer came back. Ask a route about a record nobody owns and, instead of a refusal, it returned a server error carrying the full database statement and the values bound into it. That was not the route’s doing. It was the one line at the top of the whole application that turns any error into a response, and it treated a sentence somebody wrote for a reader exactly the same as an exception nobody expected.

Not a hypothetical

We read our own production error log before changing anything. The forgotten-password route, which has to be open to anybody, had answered with the mail system’s configuration fault in full. A recipe save had answered with a database constraint by name. The question-answering route had passed its provider’s raw error straight through. A kitchen board had answered with an internal error reference. None of it was secret in the way a password is secret, and all of it was the inside of the building, said to whoever happened to trip over it.

We had fixed this twice before at the call site: a messaging provider’s error that quoted the account token, and a translation provider’s error reaching a guest’s phone. One level up, the same fix covers the ones nobody thought to wrap.

The obvious fix would have been wrong

The tempting rule is to hide the message on anything with a server-error status. The log is what showed that to be backwards. The same route had produced both a deliberate sentence, I could not work out what to look up just now, try asking again, and a raw provider error, and the deliberate one carried the higher status code. Filtering on the number would have hidden the message meant for a reader and kept the one that was not.

What separates them is whether anybody chose the status. A refusal somebody wrote sets one. An accident has none. So a chosen status keeps its sentence, and everything else gets one plain line plus a reference, the same request id that is already in the log row, so a person reporting a fault can name the one to look at. The full message, stack, path and user still go to the log and to the first-occurrence email. The detail is not lost. It just stops being said to the person who tripped it.

Proving it

The tests make the database itself throw, because that is how an unexpected error actually arrives, from underneath the route rather than from a line somebody wrote. Reverting the fix fails one test alone. The deliberate-sentence control passes with and without the fix, which is the property a status-code filter would have broken. Our own first bench measured nothing: it broke a query on a route that refuses you before it runs if you are not signed in, so the refusal arrived before the error could. That mistake is a test now too.


Try it on tonight’s service.

Nothing to install, no card. Not better by the weekend? Close the tab.