- date written
- minutes to read
- 7
- sources cited
- 7
The test suite passed without loading a single page
The automated check loaded 0 of 8 pages and still reported a pass. The accessibility scan covered 0 pages too. How I made it stop hard on missing data, and the test I run before trusting any new check suite.
- check gates
- operations
- ai with a human approver
On 18 August 2026 I handed my fire-door website to Codex for a full review. Finding number 5 is the line I still remember: the layout gate was issuing false passes. This post covers that bug, the fix, and a discipline I already had on another project and forgot to apply here.
The log looked reassuring
The layout suite opens a real browser with Playwright, loads each page at 5 widths, measures horizontal overflow, distorted images and a chat button covering a call to action, then runs an accessibility scanner. Results from that review:
| Width | Pages loaded |
|---|---|
| 320px | 0/8 |
| 390px | 0/8 |
| 768px | 0/8 |
| 1440px | 0/8 |
| 1560px | 0/8 |
| accessibility scan | 0 pages |
After that table the suite still printed "no horizontal overflow, no distorted images" and exited with a pass. The reason is plain. Page load failures were pushed into the warnings list, not the errors list. At the end it counted only the errors list. Empty list, green. A missing browser library also came out as a pass.
Codex rated confidence in accessibility 3/10: a scanner that ran on 0 pages makes the pass stamp meaningless.
What bothers me sits outside the bug itself. I had treated the green check as evidence, while Codex wrote plainly that valid test evidence was still missing. The suite only ever answered one question: did anything get written to the errors list. I was the one asking the wrong question.
The fix: missing data has to turn it red
Codex put the trustworthiness of the process first, ahead of any interface polish. It came with six requirements for the gate, from "a page that does not return a success code is a blocking error" to "a missing browser library must not count as a pass". The exit condition was one sentence: the checks have to actually run.
Claude Code wrote the fix. The layout suite now blocks at these points:
- A dead root page stops the run. Before opening a browser, the suite probes the root page with a 5 second timeout. If the root is silent it stops there, printing how to start the server.
- A page returning an error code is a blocking error. Blocking instead of warning, with the status code in the log.
- A counting gate. Measurements taken must equal pages times widths; any gap raises a "CHECK SUITE HAS INSUFFICIENT DATA" error. Pages scanned for accessibility must equal the pages on the list.
- A missing library exits with an error code. The message prints the words "not a skip".
- The CI runner does the setup before it checks anything. Install the browser, start the server, poll up to 30 times until the server answers, then run the gate.
- Screenshots for a human to look at. The main pages at 390, 768 and 1440 are saved as images.
End of session on 18 August: layout 70/70 real measurements, that is 14 pages times 5 widths, 0 errors. The list is now 15 pages. The accessibility scanner ran 14/14 pages at width 390, 0 violations at serious level or above. Screenshots covered 4 pages times 3 widths, and I opened them myself during the session.
Break it on purpose before you trust it
In the ERP bot framework I had already written a discipline into the overview document: a new check suite must be fed a deliberate fault and go red before anyone trusts it. That suite holds 153 pure logic tests plus 6 structural checks behind a single command, and it caught more than 80 real bugs while the system was being built, 51 of them modules wired together wrong. The discipline existed. The layout gate on the fire-door site was where I forgot to apply it. A suite that has never gone red is a suite nobody has proven runs at all.
The cheapest test uses what is already there. Stop the server, run again, it has to go red. Uninstall the browser library, red. Add a page that does not exist to the list, red, because that page returns an error code and the count comes out short. Removing a page from the list proves nothing, since the expected number is computed from the list itself.
Stopping hard on missing data covers the first kind of false pass, the one where nothing was measured. The second kind, where the measurement is real but misses the thing that matters, only shows up when it hurts. This suite carries two such scars, written into the comments:
- The sticky summary bar on the quote page ran from -20 to 410 at width 390 for a whole cycle while the gate stayed green. At measuring time it was still hidden, width 0, so it was skipped. Now, on multi-step pages, the suite performs a few actions before it measures.
- The gate once stayed green while 6 real content blocks sat at full transparency, because the fade-in effect was attached to an element inside a horizontal scroller that never fired. The log was quiet, the layout was tidy, the images kept their aspect ratio, and the visitor saw no content at all. On 21 August 2026 I added an "invisible content" rule: text that is present but not visible is an error.
The other direction matters as much. False reds kill a gate as fast as false greens. To avoid false reds I dropped the advisory rule set, because it flagged legitimate design choices too. Violations below serious level go to warnings only and the release still ships. The accessibility scanner runs at width 390 only, and that is a time decision: its rules barely change with width, so running it 5 times costs 5 times the minutes for the same result. A gate that cries wolf is a gate somebody is about to switch off.
Four gates, and a fifth one off the line
Before every release the site runs four gates. Whichever one goes red, the run stops there.
| Gate | What it checks | Requires |
|---|---|---|
| Site check | broken links, missing images, animations that never stop | offline |
| Content check | internal jargon, broken string interpolation, blocks on the wrong product type | offline |
| Prefix check | paths missing the prefix when the site sits in a subdirectory | offline |
| Layout check | 15 pages times 5 widths, plus the accessibility scanner | a running server |
The three offline gates also run inside the build step. Wrong content fails the build. That is the last net.
The fifth gate runs outside the automated line: it compares current screenshots against a baseline set. It goes red whenever the interface changes, which is its job. I only sign off a new baseline after seeing with my own eyes that the new version is the one I want to keep. The reason it sits apart: a baseline set breaks on every legitimate design change and then gets switched off, while the hard rules have to stay standing.
What I took from it
- Evidence is a measurement count that matches the expected count, plus screenshots somebody opened. A line of text saying pass is not evidence.
- Feed a new check suite a deliberate fault and watch it go red before trusting it. Stop the server, remove the library, add a bad page: three tests you can run right now without writing anything.
- False reds and false greens end the same way, with somebody switching the gate off. Anything noisy belongs outside the automated line, like the screenshot comparison.
- After the machine says pass, a human approves. I build these systems with Claude Code and Codex, and the last sign-off is my own eyes.
If your company's checks are all green and nobody has tried to make them go red, tell me the context.