A filter that throws away good readings
Dropping bad data is easy to overdo. A screen drawn while nobody was looking gives a meaningless number; the same screen if the phone locked afterwards does not. Being stricter would bin real measurements.
We measure how long every screen in the product takes to open, and this week we added a rule that throws some of those measurements away.
The reason is that phones and browsers deliberately stop working on a screen nobody is looking at. Put the app in your pocket mid-load and the phone stops drawing, saves its battery, and finishes the job whenever it comes back. The number that arrives is how long the phone was in your pocket. It is not how long the screen takes.
So: if the screen was out of sight before it finished drawing, bin the reading. Straightforward.
The bit that is easy to get wrong
The obvious next step is to bin anything measured on a screen that went out of sight at all. It is simpler to write, it is simpler to explain, and it is wrong.
A screen that drew in 300 milliseconds and then got locked in a pocket two seconds later produced a perfectly good measurement. The out-of-sight part happened after the measuring finished. Throwing that away costs you a real reading, and on a screen used mostly by staff holding a phone, it would cost you most of them.
Which gives you the worst kind of broken report: one that looks healthy. Few readings, all of them from the handful of occasions nobody pocketed the device, and a total that says quietly on the page that everything is fine.
Match the filter to the window
The rule we settled on asks a narrower question. Not "was this device ever out of use?" but "was it out of use during the stretch of time this particular number covers?" For a screen opened from cold, that stretch runs from the app starting. For a screen opened by tapping inside a running app, it runs from the tap. Two different windows, so two different tests, and a reading only goes in the bin if the interruption lands inside its own window.
The same distinction runs through anything you measure in a venue. A prep time that spans a fire alarm is not a prep time. A prep time on a ticket that was sitting done on the pass while the alarm went is completely valid, and a crude "bin everything from that hour" rule loses it along with the rest of lunch.
Whenever you put a filter on your data, it is worth writing down the readings you are deliberately keeping, not only the ones you are dropping. Filters get tightened over time by people who are only thinking about noise, and that note is the thing that stops a report quietly becoming a report about nothing.