You must rotate the HMAC signing secret used for outbound webhooks to about 18,000 subscriber endpoints, after a support engineer pasted it into a ticket. Subscribers verify signatures with the secret they hold; a mismatch means they reject the delivery. Nobody can be made to deploy on your schedule. Sequence the migration.

The sequence

Each step is reversible until step 6, and the whole thing works only because signatures are additive: a delivery can carry more than one.

1. Generate the new secret and expose it in the dashboard and API immediately, marked as pending, alongside the current one. Subscribers can adopt before you change anything, which makes the next step cheap.
2. Sign every delivery with both secrets. Send multiple signature values in one header, each labelled with the key id and the timestamp it covers, as mature providers do: t=...,v1=<sig-old>,v1=<sig-new>. A subscriber that checks any one valid signature passes throughout. This is the step that makes the rotation invisible, and it requires that your signature header was designed to hold a list. If it was not, that is the first fix, and it is itself a compatible change because old clients read the first value.
3. Instrument adoption per subscriber. You cannot see which secret a receiver used, so give them a reason to tell you: a verification endpoint, or a dashboard confirmation step that records the acknowledgement. Absent that, proxy it with endpoint delivery-failure rate per subscriber after the cutover test in step 4.
4. Dark-launch the cutover on one low-volume subscriber cohort by sending only the new signature to accounts that have acknowledged. Watch their delivery success for 48 hours. Roll back by re-adding the old signature, which is a config change.
5. Run the campaign. Email, dashboard banner, Deprecation header on the webhook-management API, and a dated deadline. Expect adoption to stall around 70–85%: the long tail is subscribers whose integration is unowned. Phone the top accounts by volume; they are a small number of the total and a large share of deliveries.
6. Brownout, then remove. Two weeks before the deadline, drop the old signature for one hour a day and publish the window. Failures become support tickets while you are watching, not after. Then delete the old secret. This is the point of no return, because a subscriber discovering the problem afterwards cannot verify anything until they deploy.

Where it can diverge, and how you would know

The silent failure is a subscriber who verifies correctly and also replays-protects against the timestamp in the same header. If your double-signing changes the canonical string being signed rather than just adding a value, every receiver breaks at once. Verify by signing with the old secret over the unchanged canonical form and treating the new signature as a pure addition. Test with a receiver built from your own published sample code, in the oldest language version you document.

The second divergence is quieter: receivers that ignore signature verification entirely. The rotation will not break them, and the exercise reveals how many there are. That number belongs in your next security review rather than in this migration.

How long it really takes

For a communications API in Twilio's mould, with tens of thousands of subscribers and no ability to force deploys, plan 8 to 12 weeks from double-signing to deleting the old secret, of which engineering is maybe two weeks and the rest is adoption. If the secret is genuinely compromised and the risk is active, the honest answer is different: rotate in days, accept broken deliveries for the tail, and lean on the retry queue to deliver once they fix verification. Say which situation you are in before choosing the schedule.

When this is the wrong answer

If signatures were asymmetric from the start, with a published JWKS and key ids, rotation is publishing a new key and waiting for caches to expire, and none of the above is needed. That is the design to migrate towards once this incident is closed, and the argument for it is exactly the eight weeks you are about to spend.