Investment proposal
Two engineers for one quarter to remove the shared ingestion failure
Approve RFC 014 now so the next queue failure degrades throughput instead of stopping Checkout, Catalog, and Inventory together.
The ask
Fund two Platform engineers for one quarter
Release only when one failed queue cannot stall the other.
- Decide by
- 2026-09-01
- Decider
- Platform director
- If declined
- Record the single queue as an accepted risk
Why fund this now
Four of the last six incidents ended in the same platform-wide halt. Ingestion represents 41% of the measured footprint, and the observed cadence is roughly one repeat per quarter if the boundary does not change.
Verified from the incident register and 2026-08-12 inventory snapshot.
What the quarter buys
outcome- A stateless dispatcher that partitions accepted events by event ID.
- Two queues and consumer groups that deploy and recover independently.
- Dashboards, alerting, runbooks, and a one-queue-down chaos test.
- A reversible production cutover with the original queue retained through observation.
Cost and opportunity cost
12 weeksTwo engineers for twelve weeks, plus Reliability review during the game day and cutover. The catalog backfill slips one quarter. Ongoing cost is one additional queue, consumer group, and dashboard lane.
The alternatives preserve the failure mode
optionsDo nothing
No delivery cost this quarter; accept the observed halt cadence and recovery coupling.
Increase queue capacity
Lower delivery cost, but only delays saturation and leaves every producer behind one boundary.
Split the path
Best balance of isolation, reversibility, and bounded operational growth.
Queue per producer
Stronger isolation, but too much policy and on-call surface for the current producer set.