A test that reads the code but never runs it
Two tests covered our card adapter. Both were searches over its source — they asserted what it must not contain and never made it do anything. A green suite, and a bug on the second real test.
There is a kind of test that feels like coverage and is not. It opens the file and searches the text: does this module mention a provider's name where it should not? Does it call the shared helper? Does the till compare a status string it has no business comparing?
Those are real checks and they catch real things. They also never execute a single line of what they are checking.
What we had
Our card-reader adapter — the part that actually talks to the card company — had exactly two tests. Both were searches over its source. The suite was green. Nearly six thousand tests, green.
The adapter had never run. Not once, not in a test, not in production. Its request bodies, its two-call sequence, the way it reads a card's last four digits out of the reply — all of it was code somebody had written carefully from a specification, and nothing had ever made it do anything.
What happened when it ran
We replaced the network with a fake one and called the functions for real. The second test failed. Not a stylistic quibble — a refusal that would have hit the most common sale a counter makes, every time, on the first press with a real key. That one has its own post.
A day later the same treatment on the code that reads a menu from a link found a check that only checked the first address. Also never run. Also invisible to a search over the source, because the bug was in what the function did, not in what it contained.
The three fences, and what each can see
A search over source catches a rule about shape: this file may not name that provider, this bundle must be listed before that one.
Driving the real screens in a real browser catches what source cannot: a number rendered below the fold is present, expanded, complete and unseen. We found three bugs that way in one sitting on a screen the suite had no opinion about.
And running a module with its network faked catches what even the browser cannot. Our browser test of the card flow stubbed the route — so everything below the route stayed dark, which was the entire adapter.
None of the three replaces the others. What they have in common is that each one costs something and each one found something the cheaper ones could not.
The bit that is easy to get wrong
A search over source will happily match your own comments. We have written a test that passed because the phrase it was looking for was in a comment explaining why the phrase should not appear. Strip the comments first, then search — and when a test fails, ask whether the test or the code is wrong before touching either. Five failures in one sitting recently were stale tests. One was a real second database round trip.