Red-Main Normalization: Why Stacked CI Failures Feel Endless
Yesterday my main branch was red. I fixed it. It was still red. I fixed it again. Still red. Three rounds before green — and each round felt like failure, even though every fix was correct.
The cascade
| # | Symptom | Root cause | Fix size |
|---|---|---|---|
| 1 | Every scan: “not on classpath” | Test library lives in a git submodule CI never fetched | 1 line: recursive submodules |
| 2 | Scanner crash: rg missing |
Slim image lacks ripgrep the scanner shells out to | 1 word in apt-get |
| 3 | Resume test red forever | Guard hashed a “frozen” file never committed (gitignored) | Repoint guard at tracked source, hash in-process |
Why it feels endless
Fail-fast pipelines stop at the first error, so stacked failures
present as one immortal bug. Worse: once main is red, every
layer beneath rots silently, and each repair only reveals the next
corpse. I call it red-main normalization — the branch
stays red so long that red starts looking like weather instead of
signal. The fatigue is real but it’s information: a non-converging loop
means the structure is wrong, not your effort.
The two locks that end it
- Guards must point at tracked sources. A test that depends on an uncommitted file is a wish, not a guard. If it isn’t in git, it doesn’t exist.
- Gate before merge, not after. CI that only runs on
maindiscovers breakage after landing. Running the same validation on merge requests moves every future failure left — to the reviewer, before the branch lands, with a name attached.
After both locks: failures arrive one at a time, before merge, green
main as the steady state. The dread goes away on its own —
not because debugging got easier, but because the loop converges
again.