After a routine signing key rotation, roughly 3% of API requests start failing with 401s. The rate decays over about ten minutes and returns on the next rotation. Tokens look valid and clocks are synchronised. What is happening?

The first three things to look at

1. Which resource servers are failing. If 401s cluster on the instances started most recently, or least recently, the problem is per-process state rather than per-token.
2. The kid on the rejected tokens against the key set each failing instance holds. A rejection for an unknown key identifier is a different fault from a signature that does not verify.
3. The decay curve. Ten minutes of decay after a rotation is the shape of a cache expiring, not of a bad token.

The diagnosis

Resource servers fetch the issuer's public key set and cache it, commonly for 10 to 60 minutes, because fetching per request would put the identity provider in the path of every API call, a cost no production system accepts. Rotation publishes a new key and the issuer starts signing with it immediately. Every resource server with a cached key set that predates the rotation has never seen the new key identifier, so it cannot verify the signature and returns 401.

The failures decay as caches expire, and they recur every rotation because nothing about the mechanism has changed. The percentage is set by how many instances happen to hold a stale set when signing switches over.

The misleading signal is the token. It is valid, correctly signed and unexpired. The fault is entirely in the verifier's view of the key material, which is why inspecting tokens leads nowhere.

The fix, in order

1. Publish the new key before signing with it. The key set must contain the new key for at least the maximum cache lifetime across all verifiers before any token is signed with it. If caches last 60 minutes, publish at least 60 minutes ahead. This is the fix; everything else is a safety net.
2. Keep the old key published for at least the maximum token lifetime after rotation, so tokens already issued still verify. The key set therefore holds two keys during every rotation window, which is normal and correct.
3. Refetch on an unknown key identifier, with a rate limit. A verifier that meets an unknown kid should fetch the key set once, bounded to something like one fetch per minute per process, so a rotation self-heals in seconds without letting a flood of bad tokens hammer the issuer.
4. Honour cache headers from the key set endpoint rather than hard-coding a lifetime, so the issuer can shorten the window before a rotation.

The alert that would have caught it earlier

401 rate by failure reason, separating unknown key identifier from bad signature from expired token. Rotation should also be a deployment event on the dashboard, because a spike correlated with a rotation is self-explanatory and a spike with no marked cause takes an hour to attribute.

When this is the wrong diagnosis

If the 401s do not decay, or affect all instances equally, look at audience or issuer validation instead - a new key set often arrives with a configuration change. And if tokens are validated by introspection against the issuer rather than by local signature checking, no key cache exists and the cause is somewhere else entirely, usually the introspection endpoint's own availability.

What the pattern generalises to

Prefer this rule everywhere: any rotation of material that is cached by consumers needs an overlap window at least as long as the longest consumer cache, plus the lifetime of anything already issued. That rule covers signing keys, TLS certificates, API credentials and encryption keys, and the failure when it is skipped always looks like this: a brief, self-healing, perfectly reproducible outage nobody can explain.