Conceptual explanation
Why ingestion fails as a whole even when only one component is full
The important property is not queue size. It is the absence of an independent path: every producer and consumer shares the same place where work can wait.
Learning goal
After reading, you can explain
- why one full queue stops every producer, not only the busiest one;
- why more capacity delays saturation but does not change who stops;
- what RFC 014 defines as the leftover when one queue fails.
One queue creates one fate
modelCheckout, Catalog, and Inventory enter through the same gateway and wait on the same queue. Their traffic volumes differ, but their ability to make progress does not: when the queue cannot accept, the gateway has nowhere else to place any producer’s event.
Shared infrastructure is not automatically shared fate. Shared infrastructure becomes shared fate when it has no independent failure or recovery boundary.
Capacity changes the clock, not the topology
distinctionMore capacity
Moves saturation later
A larger queue or more default consumers can absorb a bigger burst. If saturation still arrives, every producer remains behind it.
More boundaries
Changes who can continue
Independent queues let one partition fail while the other keeps accepting. The outcome changes even if total capacity does not.
Backpressure travels upstream
cause chainConsumers slow
Drain rate falls below arrival rate.
The queue fills
The buffer stops accepting within the gateway wait budget.
Producers fail
The gateway blocks or returns overload to every caller.
The July incident followed this exact chain. Scaling consumers increased drain rate and recovered the queue. It did not change the chain’s ability to recur.
Degradation is a defined leftover
implication“Degrade gracefully” is useful only when the remaining behavior is explicit. Under RFC 014, a failed queue leaves the other partition accepting events, bounds the gateway wait, records a shed outcome, and tells producers to retry the same event ID.
The proposal is valuable because it defines what continues, what stops, and how callers recover. That is a failure boundary; “more resilience” on its own is not.