← Story

Operational runbook · RB-INGEST-01

Recover when ingestion queue lag keeps climbing

Use this procedure after ingest_lag_seconds exceeds 120 for five minutes or producers report blocked or failed event acceptance.

Owner: Platform Reviewed: 2026-08-18 Maximum local change: +1 replica Escalate at: 15 minutes

Before changing anything

Where ingestion runs: prod-ingest and stage-ingest In the prod-ingest cluster, namespace ingest runs the gateway and the ingest-consumer deployment. The gateway enqueues to the queue ingest, the consumers read from it and write to the warehouse; the queue and warehouse are dependencies whose hosting this map does not show. The accent marks the consumer deployment: three replicas run, operators add one at a time, and never more than six without Reliability. The stage-ingest cluster runs the same namespace and is where release candidate sha-8132c3 was tested and where dual-write begins. CLUSTER prod-ingest NAMESPACE ingest CLUSTER stage-ingest DEPENDENCIES Namespace ingest: gateway and ingest-consumer. sha-8132c3 tested 2026-08-12 · dual-write from 2026-10-12 enqueue read Gateway POST /events Consumer group deploy/ingest-consumer +1 · max 6 Queue ingest Warehouse replica running the one replica to add next dependency; hosting not shown
Where you are working: one deployment, one replica at a time, six at most without Reliability.
01

Recovery procedure

target: 15 min

  1. Name the failing boundary

    Open the lag dashboard and identify the queue whose lag is rising. On the current system this is ingest; after RFC 014 it is ingest-a or ingest-b.

    Success: the incident note names one queue, current lag slope, and affected producers.

  2. Check consumer health before scaling

    Inspect ready replicas and restart count. If any consumer is crash-looping, do not scale around it; go directly to escalation.

    bashkubectl -n ingest get deploy/ingest-consumer
kubectl -n ingest get pods -l app=ingest-consumer

    Success: all current replicas are ready and restart count is stable.

  3. Add one consumer replica

    Increase the deployment by exactly one replica. Do not exceed six total replicas without Reliability approval.

    bashkubectl -n ingest scale deploy/ingest-consumer \
  --replicas=$((CURRENT_REPLICAS + 1))

    Success: the new replica becomes ready and lag slope flattens within three minutes.

  4. Observe, then choose

    Watch lag, queue depth, gateway p99, and consumer errors for three minutes.

    If lag falls: continue to verification. If lag is flat: observe for two more minutes. If lag rises or a consumer crashes: escalate.

02

Verify, rollback, or escalate

decision

Verify

Close only on sustained recovery

Lag falls for ten minutes and POST /events p99 remains below 200 ms.

Rollback

Remove the added replica

Return to the previous count if the new replica causes errors. Never delete queues.

Escalate

Stop local changes

After 15 minutes, any crash loop, or rising 5xx: page Platform primary and Reliability.

Record the final replica count, recovery time, peak lag, and whether the documented control changed the slope. Link the incident note before handing back.