Retries, dead letters and recovery
The classification that matters
Everything else follows from getting this right:
| Class | Examples | Action |
|---|---|---|
| Transient | 429, 500, 502, 503, timeout, connection reset |
retry, same idempotency key |
| Permanent | 400, 401, 403, 404, 422 |
dead-letter, do not retry |
| Not an error | 200 suppressed, 202 test-key |
record and move on |
A loop that retries the permanent class is the most common failure in a mail integration. It never succeeds, it fills your logs, and it hides the real failures underneath.
Backoff
Exponential with jitter, starting around a second, capped at a minute, with a
total budget rather than unlimited attempts. Honour Retry-After when
present: on the per-minute cap it is 60, and on the monthly allowance it is
the seconds to the next UTC midnight, which is a number to respect rather
than retry through.
Jitter is not decoration. Without it, a worker fleet that all hit the cap at once retries in lockstep and hits it again together.
Retrying is safe, and that is deliberate
A timeout is the hard case: the request may have been accepted and you cannot
tell. Retry with the same Idempotency-Key and the platform resolves it:
- accepted the first time, you get byte-exact the original response, no second message
- never landed, it is processed now
That is why the key is required rather than optional. Generating a fresh one on retry is the bug the requirement exists to prevent.
Dead letters
You need a dead-letter path, because we do not have one for you. A message the API permanently refuses is simply refused; nothing parks it anywhere you can see.
A workable shape:
attempts exhausted, or a permanent class error
-> move the row to `failed`
-> store status, error code, request_id, the attempt count
-> alert on the RATE, not on each one
Store the request_id. It is the only identifier that ties the failure to
our logs, and the first thing support will ask for.
What the platform retries, so you do not
Submission to the receiving server is ours. A temporary rejection, or a
connection dropped before the body went out, is retried five times at 1m,
5m, 15m, 1h and 4h, then lands in deferred with a reason saying it gave
up.
A failure at or after DATA is terminal on purpose: the receiving server may be holding a copy we never heard about, and retrying would deliver twice.
So a message in queued needs nothing from you. Only a terminal outcome,
arriving by webhook, is a decision point.
Recovering a backlog
After an outage, the instinct is to drain the queue as fast as possible. That is the worst moment to do it: a large burst into a cold ramp is exactly the pattern that trips reputation.
Drain at your normal rate, or slower. Check
GET /v1/reputation before and during. If the
backlog is large enough that normal rate will not clear it, that is a
conversation about capacity rather than a throttle to work around.