Handling HTTP 429 in a CRM Import Queue: Retry-After, Dead Letters and Safe Replay
The scenario
Harbor Ridge Supplies is a made-up equipment dealer. Each night a worker takes rows from an import queue and sends them to a CRM API as create or update calls. Around the middle of the batch the API starts answering with HTTP 429. The worker retries immediately, five times, and then drops the row. Support later finds missing contacts and a few doubled ones. Two separate problems are hiding here: the retries ignore what the server asked for, and nothing protects against replaying a write that may already have succeeded.
What Retry-After says
MDN documents Retry-After as a response header that tells the client how long to wait before a follow-up request. For a 429 Too Many Requests response, it indicates how long to wait before making a new request. The header takes one of two forms: an HTTP date after which to retry, or a non-negative integer number of seconds to wait after the response is received. MDN also warns that support on clients and servers is still inconsistent, so a 429 may arrive without the header. Whether your CRM vendor sends it, and what its rate limits are scoped to, is a question for the vendor's API documentation.
Retry-After is about timing only. It says nothing about whether an earlier attempt changed data.
Retry rules
These are engineering recommendations for this queue, not platform rules.
- Parse both forms. Treat the value as a floor and never retry earlier. If the date is invalid or already in the past, fall back to normal backoff instead of retrying instantly.
- Pause only the affected queue. Limits may apply per tenant, per token or per endpoint, so a throttled contacts endpoint should not stop an unrelated worker using a different token. If you cannot tell the scope, pause the narrowest group that shares the credentials and endpoint.
- When the header is absent, use exponential backoff with random jitter and a ceiling.
- Cap attempts per row and total retry time per batch.
- If the requested wait lands beyond your batch window, do not sleep through it. Persist the resume time with the job, or send the job to manual review. Do not shorten the wait to fit the window.
A 429 does not prove nothing was written
It is tempting to assume a 429 means the request was rejected before any work happened. That is not something the header guarantees, and a timeout or dropped connection is even less certain: the CRM may have saved the contact while the reply was lost. Replaying a plain create can then produce a duplicate. Keep timing and duplicate prevention as separate concerns.
- Give each queue row a stable import key derived from the source record. If the CRM supports upsert by external ID, use it.
- If the API documents an idempotency mechanism, send the same key on every retry of the same row. MDN lists an Idempotency-Key header in its reference, but how a particular CRM treats it must come from that vendor.
- Email lookup alone is not a reliable guard. Addresses can be shared, changed or entered in different forms, and a lookup followed by a create leaves a gap in which another process can write the same contact.
- When a write has an uncertain outcome and there is neither an authoritative status endpoint nor idempotency support, do not automate lookup then create. Route the row to manual reconciliation.
Dead letters
After the attempt cap, move the row to a dead-letter queue. Store what is needed to act on it: the row key, a protected reference to the source record, the operation type, the attempt count, the last status, any Retry-After value seen, and timestamps. Avoid copying the full personal payload into logs, and never store tokens or other secrets. Reviewers can fetch the record from the source system through the reference.
Do not replay dead letters automatically. A person, or a separate rate-limited job, should reconcile each one against the CRM and then resubmit. Alert when the dead-letter count passes a threshold so a throttled night does not pass silently. Updates need extra care: replaying an old update after a newer manual edit can undo that edit, so compare a version or timestamp first.
Proposed checks (not performed)
None of these has been run.
- A simulated 429 with a seconds value pauses only the affected queue for at least that long.
- A simulated 429 with an HTTP date behaves the same way.
- A past or malformed date falls back to backoff without an immediate retry.
- A retry time beyond the batch window is persisted or sent to review.
- A timeout after a successful create does not produce a second contact.
- An uncertain write without idempotency support lands in manual reconciliation.
- Dead-letter entries hold a reference and no secrets or full personal payloads.