At 11:00 a contractor's laptop is found compromised and the API key they had access to is rotated in the secrets manager. By 11:20 the security team has confirmed the old value appears nowhere in configuration. At 02:40 the next morning, provider audit logs show the old key authenticating successfully from an unfamiliar address. What failed, and which design decision made it possible?

The trigger

A rotation that was never a revocation. The secrets manager created a new version of the key and every consumer picked it up. Nothing in that sequence invalidated the old value at the provider that accepts it, and the provider is the only party whose opinion matters.

Why it propagated

Three mechanisms compound:

• Rotation is a write; revocation is a delete. A secrets manager stores values. Whether an old value still authenticates is a property of the issuing system - the cloud provider, the partner API, the database. Writing a new version there is one call; deleting the old one is a second call, and it is the one that is routinely skipped because everything works without it.
• Consumers cache, so the old value appears unused. Long-lived processes read a secret at startup or hold it for a cache lifetime. After rotation, traffic authenticating with the old version decays over minutes to hours and then stops - which looks exactly like successful revocation on a dashboard that counts usage rather than validity.
• "Appears nowhere in configuration" is the wrong test. It proves the value is no longer distributed. It says nothing about whether it is still accepted, and the attacker does not need your configuration.

Why detection lagged fifteen hours

Nobody was watching for authentications by key version. The provider's logs contained the evidence from the first use, and the only alert in place fired on failed authentications - which this was not. A stolen credential that still works generates no errors anywhere, which is the general reason credential compromise is measured in weeks of dwell time rather than minutes.

The structural fix versus the tempting local fix

The tempting fix is a checklist item: "remember to delete the old key". It will be skipped again.

The structural fixes, in order of strength:

1. Make expiry do the revoking. Short-lived, dynamically issued credentials - a 15-minute database lease, a 1-hour cloud role session, a workload identity token - mean a stolen value is worthless by the time it is used, and no human step is required. This is the only fix that holds under pressure.
2. Where a static key is unavoidable, make the rotation pipeline fail unless the old version is confirmed invalid. The last step of rotation is an authentication attempt with the old value that must fail. Rotation is not complete until that assertion passes.
3. Alert on authentication by credential age or version, not only on failures. A key older than its rotation interval authenticating at all is an incident; so is version N-1 authenticating after the rotation window has closed.
4. Publish the dwell window as a number. "After rotation, the old credential remains valid for up to 60 minutes while consumers converge" is a decision somebody should make deliberately, rather than a side effect of a cache lifetime.

The general lesson

Ask what the attacker holds, not what your systems contain. Rotation answers "what do our services use"; revocation answers "what still works". Incident response needs the second, and for every credential class in the estate the answer should be known before an incident: who accepts it, how it is invalidated, and how long invalidation takes to be effective.

When this is the wrong procedure to design

A credential whose issuer has no revocation mechanism at all - some legacy partner integrations - cannot be revoked, so it must be treated as permanently valid once leaked. That changes the response (notify the partner, change the account) rather than the rotation procedure, and it should be recorded as such before an incident. Equally, for credentials that already live for 15 minutes, building a revocation list is usually not worth the lookup on every request: expiry has already bounded the exposure.