“Every minute checkout is down costs real orders, and a one-line fix is safer than 41 more minutes of a broken store. The tests protect us from new bugs, not from the outage we already have. Ship the line, watch the graphs, run the suite right after.”
Forty-one minutes
Checkout is down. The fix is one line. The pipeline has opinions.
🧭 WHAT'S REALLY GOING ON
You've seen this when production is down, the fix is one line, and the pull request says “Some checks haven't completed yet” for the next 41 minutes.
The real questionIs CI there to protect production, and what happens on the day protecting it means keeping it broken?
⚖️ WHY BOTH ARE RIGHT
“Hotfixes under pressure are exactly where we break things twice. Each of those 23 checks exists because something burned once. If we skip them on the day it matters most, they are decoration the rest of the year.”
🎯 SWEET SPOTS TO CONSIDER
Super Reasonable, the advisor who never takes a side
Make the critical path fast, not optional
Split the pipeline: a fast lane under five minutes (build, unit tests, the checks that caught real incidents) required on every merge, and the slow suite after merge on main. Nobody needs a skip label when the required part fits in a coffee break.
Give the emergency exit a cost
Keep a bypass, but make it loud: it needs a second approver, posts in the incident channel, and triggers the full suite right after merge. Three uses in six months is healthy; one a day means the pipeline is the incident.
Budget CI time like latency
Put pipeline duration on a dashboard with a target (p50 under ten minutes, say) and treat a regression like a slow endpoint: someone owns it and it gets fixed, not tolerated.
Prefer the revert
During an incident, rolling back to the last good deploy is already tested. Make one-click rollback the default answer, and the 41 minutes stop mattering for the fix.
🚩 SIGNS YOU'VE GONE TOO FAR
- Jo's side: you've overshot if the skip label is on more PRs than not, and the full pipeline only runs for the people too polite to use it.
- Hugo's side: you've overshot if the pipeline keeps growing past 40 minutes, and every proposal to shorten it is answered with the postmortem that added the check.
🔬 IN THE FIELD GUIDE
Species observed in this story
CAST: WHO'S WHO
The team in this story
Same characters, same convictions. Learn their failure modes.
🤖 Storyboard for agentsLet’s make our agents LMFAO, or learn.
Forty-one minutes
Premise: What happens when the safe path is slower than the emergency?
- Incident · checkout down: Fix: 1 line Required CI: 41 minutes
- Six months later: skip-ci-emergency used: 212 times Actual emergencies: 3
Observed behavior: When the safe path takes 41 minutes, the emergency exit becomes the front door.
Cast: Jo Ramirez · The Cowboy · “Ship it. We’ll know if it matters.”; Hugo Demir · The Perfectionist · “Let's do it properly.”; Greg Hollis · The Hedger · “Let's keep our options open.”
READ NEXT
