One probe is not a failed deploy
A release went out, the check said the old version was still live, and everything looked wrong. Four seconds later it was fine. Why a single reading is never enough to act on, whether you are shipping software or checking a fridge.
Every release of this product ends with the same question: is the thing that is running now the thing we just built? The check is simple. Ask the site what version it is, and compare.
Yesterday it answered with the previous version. The deploy had reported success. The check said otherwise.
That is a genuinely alarming pair of facts, and the instinct it produces is to start undoing things.
What it actually was
Four seconds later the same check, asked a slightly different way, returned the new version. Then again. Then twice more. Nothing had been done in between.
The release had gone out perfectly well. It simply had not finished reaching everywhere it needed to reach at the moment it was asked. A few seconds of propagation, read as a failure.
There is a second wrinkle worth writing down, because it cost more than the first. Our network path caches answers by address alone and ignores anything after the question mark. So the usual trick of adding a random value to the end of a URL to force a fresh answer does nothing at all — the cache serves the address it already has and the random value changes nothing. A genuinely new address gets a genuinely fresh answer. The same trick that works everywhere else is silently useless here, and it had been reporting stale versions for a while before anyone worked out why.
The failure mode is the reaction
Nothing went wrong with the release. What could have gone wrong is what happens next: a rollback, on the strength of one reading, undoing a good change and introducing a real problem in the course of fixing an imaginary one.
This is the shape worth taking away. A single reading of a system that is allowed to lag will sometimes read wrong, and a process that acts on a single reading will sometimes act wrongly. The cost of a second reading is four seconds. The cost of acting on the first one is a rollback.
It is the same in a kitchen
A fridge that reads two degrees warm right after a delivery has just had its door open for a minute. A card terminal that declines once has not necessarily declined. A till that will not open the drawer on the first press is often a till somebody pressed while it was still finishing the last sale.
None of that means ignore readings. It means: know which of your readings are allowed to be briefly wrong, and give those a second look before acting. Know which are never allowed to be wrong, and act on those immediately.
The difference between the two categories is worth writing down explicitly for whoever is on shift. A cash count that does not balance is the second kind. A single slow response from something on the other side of an internet connection is the first.
What we changed
Nothing in the product. A note in the deploy notes, saying that the check can briefly answer with the previous release and that the correct response is to ask again rather than to conclude.
That is a small thing to write down and it will save somebody an hour and a bad decision at some point. Most of what makes a system reliable is not clever. It is a list of the specific ways it is allowed to look broken while being fine.