← Story

Incident review · INC-2026-07-30

One full queue stopped every ingestion producer

For 1 hour 52 minutes, a single saturated queue turned local backpressure into a platform-wide halt. Consumer scaling restored service; it did not remove the failure mode.

Status: Final Window: 2026-07-30 14:12–16:04 UTC Owner: Platform Severity: SEV-1

Impact at a glance

1h 52mproducer disruption
118 minpeak warehouse lag
3producer domains affected

Checkout, Catalog, and Inventory returned 503 or blocked while the gateway waited on the queue. No events were corrupted; freshness and acceptance were unavailable.

Verified from pager timestamps, gateway 5xx counts, queue depth, and warehouse freshness for the incident window.

01

Timeline

UTC
  1. Detected. Lag crossed 120 seconds for five minutes; the on-call page fired.
  2. Scoped. Consumers were alive, but queue depth continued to rise faster than drain rate.
  3. Mitigated. The on-call added one consumer replica. Lag flattened briefly, then resumed climbing.
  4. Escalated. Gateway requests began timing out; every producer entered hard failure.
  5. Recovered. A second scale-up drained the queue and producer acceptance returned.
  6. Validated. Warehouse freshness returned below ten minutes and no replay gap remained.
02

Cause, not symptom

3 links

Trigger

Burst exceeded drain rate

Checkout traffic rose while one consumer was recycling.

Amplifier

One queue held every producer

Healthy producers had no independent path or bounded shed behavior.

Outcome

Backpressure reached the gateway

Requests blocked until clients saw timeouts and 503 responses.

Root cause: ingestion had one queue and no failure boundary. Scaling consumers changed recovery time, not the architecture that allowed one queue to stop all producers.

03

Response review

people + process

Worked

The runbook found the right control

The on-call opened the lag dashboard immediately and the documented scale command was safe to repeat.

Failed

The alert described lag, not blast radius

The first page did not say every producer shared the queue, delaying escalation until hard failures began.

Detection also waited for five minutes above 120 seconds. That protected against noisy pages but consumed most of the buffer before a human could respond.

04

Actions that change the next incident

owned
Corrective action register
ActionOwnerDueProof
Split ingestion into two independently recoverable queuesPlatform2026-09-30Chaos test isolates one failed queue
Page at 60 seconds of lag with producer blast radius in the messageReliability2026-08-21Alert fixture and game-day page
Add bounded shed counts and retry guidance to the gateway dashboardPlatform2026-09-05Dashboard review with producer teams

Closure condition: this review closes when the alert and dashboard actions are verified. The architectural action remains tracked by RFC 014 until its isolation test passes.