A pipeline secret scanner fires on a pull request and the key it found is an active cloud credential committed 11 days ago on a branch that merged to main last week. The repository is private to a 40-engineer organisation. What has to happen first?
Show the full answer Hide the answer
What is being tested
Whether you understand that a detector tells you when you found out, not when the exposure started. The instinct of most engineers seeing this alert is to make the key disappear from the repository. That is the second task. The exposure began 11 days ago and the only action that ends it is rotation.
The mechanism
A git commit is content-addressed and immutable. Once that object exists, every clone holds a copy, and in a 40-engineer organisation there are at least 40 clones, plus CI caches, plus build artefacts, plus whatever a laptop backup swept up. Deleting the object from the server removes none of those.
Private does not mean unexposed either. The credential has for 11 days been readable by every engineer, every contractor with read access, every CI job's environment, and any log that echoed a diff. The honest assumption is that the key is known, because you cannot enumerate who read it. The 2019 NDSS study of secret leakage in public repositories by Meli, McNiece and Reaves found leaks across more than 100,000 repositories with thousands of new unique secrets appearing each day - evidence that this is a routine event with routine handling, not an emergency to be reasoned about from first principles each time.
So the order is: rotate, then audit the credential's use over the exposure window, then purge the history, then move detection earlier. The first two bound the damage. The last two prevent the next one.
Why the other options fail
- Rewrite history and force-push. The right follow-up, done third, and useless as a first move: it makes the key invisible while leaving it valid. It also rewrites every commit hash downstream of the change, which breaks every open branch and every pinned reference, so it is disruptive work bought for no security gain if rotation has not happened.
- Allowlist it and fix it next sprint. This is what happens in practice when the scanner has already produced noise, and it is how a live credential survives in a repository for a year. Allowlisting is defensible for a test fixture or a documented dummy value; it is never defensible for a credential the issuer would honour.
- Make the scanner a blocking pre-receive hook. A genuinely good change, and the only option here that reduces future risk - which is why it is tempting. It does nothing about this key. Prevention and incident response are different queues and conflating them means the live credential waits on a platform change.
When this is the wrong answer
When rotation itself is the dangerous act. A long-lived key embedded in deployed devices, a database credential shared by 30 services with no central configuration, a signing key whose rotation invalidates published artefacts: there, rotating first can cause a bigger outage than the exposure. The move then is to restrict before rotating - scope the credential down to the source addresses and operations actually in use, alert on anything else, and schedule the rotation with a rollback. That is a deliberate decision with a named owner and a deadline, not a reason to skip the step.
The rule that survives: the number that matters is the time from commit to rotation, and nothing you do to the repository reduces it.