On 20 September I broke a website seven times on purpose.
Six times a word that must never appear on it was typed into its copy. Once an entry was moved into a section it must never sit in. After each change the suite ran, one test went red, and the change was reverted. None of the seven was committed. The failures were pasted into the pull request, and the pull request merged with none of the seven in it.
The seven tests had been written the same day. They were green the moment they existed, and that was the problem.
Green on arrival
Each of them asserts an absence. A word one entry may never use. Any figure at all in that entry. Names that belong on one page and nowhere else. A test like that passes the day it is written, because the thing it forbids is not there yet. It would also pass if it were reading the wrong page, or the wrong field, or nothing. On the day it arrives, green cannot tell you which of those you have.
The only way to find out is to give it something to catch. Plant the forbidden word. Watch the test fail, and read the failure, to see that it names the right word in the right place. Take the word out. A test seen red once is evidence. A test that has only ever been green is a decoration with a pass mark.
The file carries this forward as an instruction: anyone loosening one of these assertions should plant it again rather than trust the green. It also says what it cannot do. The pages it checks are listed by hand, so a page added tomorrow is not checked until somebody adds it. That sentence tells the next reader exactly where the green stops meaning anything.
Last time I wrote about a memory system whose most interesting features returned nothing for six months without raising an error, because an empty section reads exactly like a section with nothing to report. That was a feature that could not fire. A test that cannot fail is the same thing, placed inside the one part of the system whose job is to tell me I am wrong.
The gate that reported something else
A test can be sound and the gate around it can still lie.
On 6 September a pull request on an unreleased product merged with six failing tests. The merge was the last link in a chain an agent ran: run the suite, pipe its output into a search, merge if that succeeded. A pipe reports the exit status of its last command. The last command was the search. The search succeeded, so the merge went ahead.
The six failures turned out to be noise. Another session was running its own suite against the same local database at the same moment, and the tests passed when run alone. The merge had no way of knowing that. It was green about a command that was never the suite.
The rule since is plain and a little tedious. Each gate writes its own exit status to a file. The file is read. The merge is a separate step. Four days later a different session reported a clean typecheck, and the status it printed had come through a pipe, so it belonged to the command trimming the output, not to the typecheck. The session found the same trap written in its own notes and ran the check again, capturing the real code.
Loud about the wrong thing
The opposite failure looks healthier.
Office Studio's site repository holds a second, internal app, and two guards in it check that each round of work touches only what it was authorised to touch. They compare the change against an anchor, a commit pinned on 10 August. The anchor never moved. For five weeks the guards reported fifteen reviewed, merged changes to the site as violations. Nobody saw it, because nothing ran them unless somebody chose to, and site work never chose to. Until 18 September the repository had no automated checks at all.
The tests had predicted this themselves, in a note dated the day the anchor was pinned. A stale anchor does not fail quietly; it fails loudly about the wrong thing, which is worse, because the reflex is to make the noise stop.
The fix was two tiers: cheap checks on every push, and the full suite nightly, because its first run on a pull request hit a ceiling of forty-five minutes before it finished. Nightly would catch a stale anchor within a day. It did. The anchor was re-pinned by hand on the 18th, site work kept merging behind it, and every nightly run from the 19th failed on the same two guards. Eight in a row, all red, all about innocent work.
On 26 September the anchor stopped being pinned. The guards now derive it: the point where the branch under test left main. On main that point is the latest commit, so there they check nothing, and the next full run passed. That is a real cost, and the pull request states it under its own heading. The guards bite only when a round's branch runs the suite before it merges. To show that they still bite at all, a one-line edit was made to a page the round had no business touching. Both guards failed and named that one file. The edit was reverted.
A guard that is red every night teaches people to stop reading red. A guard that is green on main because it has nothing to look at teaches them to stop reading green. The question for both is the one the seven plants answered: could this check have gone the other way, and have I watched it go?
What none of this is
None of it is automated. There is no mutation tool in any of these repositories, nothing that seeds small faults by the thousand and reports which ones the suite missed. The practice is manual. Somebody decides a check needs proving, breaks the thing, watches, reverts, and writes down what they saw. It happens when somebody remembers to do it, which is the failure the anchor spent five weeks demonstrating.
It has a blind spot of its own, found in August while building that internal app. Every significant defect in that session was found by mutation rather than by tests passing, several of them inside suites that were fully green. Then mutation reached its edge. Dropping an item from a list turned an assertion on the whole list red, while mutating the items one at a time found nothing, because a dropped item is never visited. Breaking a thing tells you whether a test notices a change. It cannot tell you whether a test notices something missing.
The writing system I built states the other limit in the opening lines of its eval: the eval catches what's mechanically wrong; it cannot catch what's honestly right. In July I tried to push past that. A model was set to judge one kind of failure in drafts, narrator commentary, the line that stops to explain what an image has already shown. Fifteen of its hits were checked by hand. All fifteen were wrong. Recalibrated, it scored revised drafts as worse than the ones they replaced, and on one line it quoted the ruling that excluded the line, then flagged it anyway. The judge was not promoted. It stays in the repository, wired into nothing, with a file beside it saying why. That boundary is taste, and taste is still the part I wrote about in the first of these, the reading that does not delegate.
Where tests cannot see
On 19 September I shipped public calendar feeds in Circuit, one subscribable address per organiser. Twenty new tests, each written to fail first, all passing.
Within minutes of the first feed going live, it was serving an internal rehearsal from August, cancelled, on a public URL.
The code did what the specification said. The specification said a cancelled event is marked, not dropped, and it had been written for a room called off ahead of time, where warning the people who were coming is the whole point. It never asked what a cancellation means once the date has passed and there is nobody left to warn. The tests could fail, and had, and were faithful to a criterion with a hole in it.
Then, on a live subscription, Google Calendar ignored the cancelled marker and drew the event as an ordinary one. No strike through, no grey. The entry was telling subscribers that a called-off room was going ahead. The only way to find that was to subscribe and look.
Four more changes followed the launch that evening, and three of them came from looking at the real feed rather than from any test. Near the end of the night a second organiser's feed was found serving seven internal rows, none of them cancelled, all of them already synced into a subscriber's calendar. Every leak had come through the same window of recent history, and every filter added to close it only waited for the next thing nobody had marked as internal.
So the window came out. The feed now publishes nothing that has finished. A room that has started stays in the feed until it ends, so tonight's room does not disappear from the calendar of the people sitting in it.