Discovery brief · DISC-012
The queue is the failure boundary; faster detection alone will not fix it
Incident evidence supports changing the architecture, while a one-queue-down test remains the fastest way to validate the proposed split before full build.
Question and method
Is the right investment a queue split, better detection, more consumer capacity, or a producer-side change?
We reviewed six incidents, the inventory snapshot, gateway and queue telemetry, the current runbook, and short interviews with the three producer teams and Reliability.
Evidence converges on one shared boundary
verified- Four of six incidents follow the same backpressure-to-gateway chain.
- Checkout dominates volume, but all three producers fail together.
- Consumer scaling restores service and has not prevented recurrence.
- Producers can retry, but the API does not expose a bounded shed signal today.
Opportunities ranked by learning value
next testsTest first
One-queue-down prototype
Proves whether the healthy partition continues and exposes rebalance behavior before the full quarter.
Do regardless
Earlier alert with blast radius
Reduces response time but does not change how many producers fail.
Insufficient
More default consumers
Adds cost and buffer while retaining one recovery path.
Too broad
Producer-specific queues
Strong isolation at a larger policy and operations cost than evidence supports.
Constraints and unresolved assumptions
open- The producer API and warehouse schema must remain stable.
- Reliability has not approved shed-and-alert behavior.
- Consumer rebalance under one failed queue is not yet observed.
- Ordering is not guaranteed today and should not enter this change indirectly.