Two Keys, One Short Window: Rehearsing a Webhook Secret Rotation Without Dropping Events

Changing a webhook signing secret in one instant is risky: the sender and receiver rarely switch at exactly the same moment. A bounded overlap, where the receiver accepts two key versions for a short time, can reduce that risk. This article proposes a rehearsal that uses key version labels only. No secret values belong in the plan, the tickets, or the test notes.

Design

Name versions, for example v1 (current) and v2 (new). The sender signs each request with exactly one version and, where the format allows, states the version label alongside the signature. The receiver holds a list of accepted versions with an end time for each.

Phases:

• P0: receiver accepts v1 only.
• P1: receiver accepts v1 and v2. Nothing is signed with v2 yet.
• P2: sender signs with v2 only.
• P3: receiver retires v1 at a pre-set time.

The order matters: the receiver must accept v2 before the sender uses it, and v1 must stay accepted until the sender has stopped using it and in-flight retries have had time to arrive.

Decision branches

Branch A, sender states a version label: the receiver checks the signature against only that version. Unknown label means reject.

Branch B, no label: the receiver tries the accepted versions, at most two. Keep that count bounded and log which version matched, so you can see when v1 stops appearing.

Branch C, sender cannot sign with a chosen version on request or cannot be sequenced with you: do not promise an overlap you cannot observe. Plan a short maintenance period where failed deliveries are held for later, and say so.

Branch D, retries of events first signed with v1 arrive after P3: decide in advance. Either the sender re-signs on each retry with the current version, or those events fail and are recovered by another route. Choose and write it down.

Security here is layered: a signature check is one constraint alongside transport protection, freshness checks and access limits on where secrets are stored. The overlap deliberately widens one constraint for a bounded time, so keep it short and record its end time.

Worked hypothetical rehearsal

Six labelled test events, each tagged with the phase in which it is sent.

Expected totals: 4 accepted (E1, E3, E4, E5), 2 rejected (E2, E6). The log should show v1 matches for E1, E3, E5 and a v2 match for E4. If E5 appears in real traffic near the end of the overlap, that is a signal to extend the overlap once by a stated amount, not to retire v1 anyway.

Stop conditions

• Do not retire v1 while any production traffic still matches v1 within the last stated observation period.
• Do not extend the overlap more than once without a named owner approving and recording a reason.
• If more than two versions would be accepted at once, stop and finish the earlier rotation first.
• If a secret value appears in a log or ticket, treat it as exposed and start a fresh rotation.

Proposed acceptance tests (to be run by you, none performed here)

1. The six-event table produces 4 accepted and 2 rejected, with the version in each log line.
2. Rejected events do not trigger any business action.
3. After P3, a v1 signature fails even if the timestamp is fresh.
4. A search of the rehearsal notes finds only version labels, never secret material.
5. The chosen Branch D policy is exercised once with a delayed retry.

This is a conditional design proposal. Whether a given sender supports version labels or re-signing on retry must be confirmed from its own documentation.

Sources