The telemetry that never measured a till
We built speed tracking to catch slow tills. Two separate silent failures meant it only ever measured the back office — and nothing anywhere said so.
After a card sale sat on a disabled button for twenty seconds, we built speed telemetry: the register times its own critical calls — recording a sale, the kitchen board's first answer — and reports each one, so a summary can show the slow points worst-first. The summary came back empty. Not slow. Empty.
Failure one: the permission nobody wrote
Access control here is an allowlist: a role can do what a rule says it can, and an unmatched request is refused. Nobody wrote a rule for the new telemetry route, so a till account — the exact device the feature exists to measure — fell through to the default no. Every measurement a till posted was refused with an error no human ever saw, because the reporting call is deliberately fire-and-forget: telemetry must never break a sale, so it never reads the answer either.
Failure two: the browser that throws where others shrug
With the rule fixed, the summary stayed empty. The report was sent as a typed binary blob, and one major mobile engine refuses that exact shape — not with a failed request, but by throwing before the request exists. Our fallback network call sat inside the same error handler, so the throw skipped it. On desktop browsers, where all our automated tests run, the blob is accepted — which is precisely why every test passed while every phone stayed silent.
What we took from it
A measurement system is only proven by a measurement arriving from the worst-placed reporter — the least-privileged account on the least-forgiving device. Fire-and-forget code deserves the opposite of forgetting at build time: because it can never complain in production, it has to be driven end-to-end before it ships, on the engine most likely to differ. The fix was four lines. Finding out it was needed took an empty table and a refusal to explain it away.