← Patterns

Current-state architecture · Northwind Ingestion

One producer-facing gateway, one shared queue, one warehouse landing path

This page describes production as it exists on 2026-08-18. Proposed dispatcher and split-queue components are shown only as a clearly marked future state.

As of: 2026-08-18Owner: PlatformScope: HTTP acceptance through warehouse writeChange proposal: RFC 014

System context and boundary

Checkout, Catalog, and Inventory are first-party producers. The gateway authenticates and accepts their events. The warehouse and its readers are downstream. Batch imports are a separate path and outside this architecture page.

Ingestion architecture Producers enter through a gateway inside the trust boundary, buffer in a queue, and are consumed into the warehouse. The queue is the only component without a failover, and it carries the whole ingestion path. TRUST BOUNDARY async Producers Gateway :8443 Queue single instance Warehouse no failover LEGEND dashed: outside Platform accent: single point of failure

The dispatcher and two queues are proposed. Production still follows the single ingest queue path.

01

Containers and responsibilities

current

Gateway

Identity and acceptance

mTLS, audience claim, envelope validation, request correlation, and queue write.

Queue

Shared 24-hour buffer

ingest holds every first-party event and has no failover.

Consumer group

Validate and land

Three normal replicas read the queue and write the warehouse.

Warehouse

Downstream record

Stores accepted events; schema ownership sits outside Platform.

02

Primary runtime path

sequence
  1. Producer sends an event with stable event ID and request ID.
  2. Gateway authenticates, validates, and writes to ingest.
  3. Gateway returns 202 after queue acceptance.
  4. A consumer validates the event and writes the warehouse.
  5. Lookup and freshness expose later state; they are not part of acceptance.
03

Failure and recovery boundaries

risk

Consumer slowdown

Lag and queue depth rise.

Queue saturation

The single buffer stops accepting.

Gateway overload

Every producer blocks or receives 503.

Adding one consumer replica is the safe local recovery control. Because every producer shares the queue, current recovery is also shared.

04

Runtime placement

deployment
Where ingestion runs: prod-ingest and stage-ingest In the prod-ingest cluster, namespace ingest runs the gateway and the ingest-consumer deployment. The gateway enqueues to the queue ingest, the consumers read from it and write to the warehouse; the queue and warehouse are dependencies whose hosting this map does not show. The accent marks the consumer deployment: three replicas run, operators add one at a time, and never more than six without Reliability. The stage-ingest cluster runs the same namespace and is where release candidate sha-8132c3 was tested and where dual-write begins. CLUSTER prod-ingest NAMESPACE ingest CLUSTER stage-ingest DEPENDENCIES Namespace ingest: gateway and ingest-consumer. sha-8132c3 tested 2026-08-12 · dual-write from 2026-10-12 enqueue read Gateway POST /events Consumer group deploy/ingest-consumer +1 · max 6 Queue ingest Warehouse replica running the one replica to add next dependency; hosting not shown

The gateway and ingest-consumer run in namespace ingest on prod-ingest; staging mirrors the namespace on stage-ingest. Replica changes are the only scaling control operators own.

05

Trust and data boundaries

security
  • The gateway terminates producer identity and enforces the audience claim.
  • Event bodies remain opaque to routing; domain validation belongs to gateway schema and consumers.
  • Queue and warehouse credentials are runtime secrets, never document content.
  • The proposed dispatcher inherits gateway identity and adds no body store.