← Story

Design document · RFC 014

Split ingestion so one queue cannot stop every producer

Introduce a stateless dispatcher and two independently recoverable queues while keeping the producer API, event payload, retention, and warehouse unchanged.

Status: Proposed Decision by: 2026-09-01 Owner: Platform Reviewers: Reliability, producer teams

The ask

Approve RFC 014: two Platform engineers for twelve weeks

Split the shared queue so one failure degrades throughput instead of stopping every producer.

Decide by
2026-09-01
Decider
Platform director
If declined
Keep the single queue and record it as an accepted risk

The recommendation

Route accepted events through a stateless dispatcher that hashes event ID to ingest-a or ingest-b. Give each queue an independent consumer group and recovery path.

Success means degradation, not invisibility. Losing one queue may reduce throughput and require retries; it must not stop every producer.

01

The shared queue is a platform-wide failure boundary

problem

Ingestion holds 41% of the included platform footprint, or 1.84M of 2.50M units. Four of the last six incidents began as queue backpressure and ended with Checkout, Catalog, and Inventory unable to accept events.

Verified from the inventory snapshot and incidents dated 2026-03-14, 2026-04-02, 2026-06-19, and 2026-07-30.

02

Goals and boundaries

scope
  • Isolate a queue failure to roughly half the event partitions.
  • Recover one path without redeploying producers or the other consumer group.
  • Keep POST /events, authentication, payloads, retention, and warehouse schema stable.

Not in this decision: replacing queue technology, redesigning the warehouse, changing the batch path, or promising global event ordering.

03

Proposed runtime path

design
RFC 014 change view: one queue becomes two Today, producers reach the warehouse through the gateway, the single ingest queue, and one consumer group. RFC 014 adds a stateless dispatcher after the gateway that hashes the event ID to one of two queues, ingest-a and ingest-b, each with its own consumer group. The accent marks the dispatcher. A change ledger lists each change by node id: dispatcher added, ingest-a and ingest-b added, consumer group changed, ingest removed once the shadow week closes; producers, gateway and warehouse are unchanged. TODAY removed RFC 014 · PROPOSED added added changed Producers Gateway Queue ingest Consumers 3 replicas Warehouse Producers Gateway Dispatcher by event ID Queue ingest-a Queue ingest-b Consumers group a Consumers group b Warehouse CHANGE LEDGER dispatcher added Stateless; hashes the event ID to exactly one queue. ingest-a, ingest-b added Independently recoverable; 24-hour retention unchanged. consumer group changed One group per queue; each scales and recovers alone. ingest removed Stays recoverable until the shadow week closes. Unchanged: producer API, authentication, payload, warehouse schema, batch path.

The gateway authenticates and validates as it does today. The dispatcher is stateless, hashes the opaque event ID, and writes to exactly one queue. Consumer groups scale and recover independently.

yamldispatcher:
  partition_key: event.id
  routes: [ingest-a, ingest-b]
  on_queue_unavailable: shed_and_alert
  max_gateway_wait_ms: 200
04

Options considered on the same criteria

trade-offs

Keep one queue

Lowest change cost

Preserves the observed quarterly halt and leaves recovery coupled.

Larger queue

More buffer, same boundary

Changes when saturation arrives, not how much of the platform it stops.

Recommended

Two partitioned queues

Removes the shared recovery path with bounded operational growth.

Queue per producer

Maximum isolation

Multiplies dashboards, policies, and on-call surface by producer count.

05

Rollout and rollback

12 weeks
  1. Build the dispatcher and deterministic partition tests.
  2. Dual-write in staging and shadow-read both new queues for one week.
  3. Run the one-queue-down chaos test and verify the healthy path continues.
  4. Cut production reads behind a flag; keep the original queue through the observation week.

Rollback flips reads and writes back to ingest. It never deletes the new queues or mutates warehouse data.