How-to · producer recovery

Replay a shed event without creating a duplicate

Use the original event ID, confirm it did not already land, and retry only when the response marks the failure as retryable.

Audience: producer engineersTime: 10 minutesApplies after: RFC 014 shed behavior ships

Before you start

You have the event ID and request ID from the original response.
The response code is INGEST_SHED with retryable: true.
You have not changed the payload. A corrected invalid payload is a new operation, not this replay.

Do not invent a new ID. A new ID bypasses the idempotency rule and may create a second warehouse row.

Replay procedure

4 steps

Check whether the event already landed
```
bashcurl -s https://api.example.invalid/events/evt_02
```
Continue only if: the response is 404. A 200 means the original attempt succeeded.
Wait for the retry window
Use bounded exponential backoff from the producer policy. Do not loop immediately against an unavailable partition.

Resend the unchanged event

bashcurl -X POST https://api.example.invalid/events \
  -H 'X-Request-Id: req_replay_01' \
  -d '{"id":"evt_02","type":"checkout.paid"}'

Expected: 202 with status accepted.

Verify by ID
Repeat the lookup until it returns one event. Stop after the producer policy’s maximum attempts and escalate rather than changing identity.

You are done when

success

The lookup returns exactly one event with id=evt_02, the replay response is recorded with its request ID, and the producer’s shed counter does not increase again for that attempt.

Before you start

Replay procedure

Check whether the event already landed

Wait for the retry window

Resend the unchanged event

Verify by ID

You are done when