Four supply-chain incidents from December 2024 to August 2025 left their primary records on GitHub: advisory texts, live incident threads, remediation commits, and the deleted tags where the attacker's releases used to be. Read together with the revocation machinery that PyPI, AWS and GitHub run in code and policy, they answer a question most security programmes never test: once a machine credential is public, who is allowed to kill it, how fast, and what breaks when they do.
A string that grants access has escaped your control. Everything that happens next is an argument between three clocks: the attacker's, the detector's, and the revoker's.
State the problem without naming a product: a machine credential, a CI token, a publishing key, a bot account's password, has been seen by someone who should not hold it. It is now a race. The attacker needs to use the credential before it dies; you need to find it, kill it, and re-issue it before the use matters. Every mechanism in this guide, scanners, push protection, quarantine policies, honeytokens, short-lived tokens, is a move in that race, and each one was built by an organisation that lost the race at least once first.
The surprise in this corpus is where the race is actually lost. In all four incidents the revocation machinery, once pointed at the right credential, worked in minutes to hours: npm had every Nx publishing token dead five and a half hours after the first malicious publish [Nx advisory]; PyPI deletes a disclosed token in code before a human reads the email [warehouse source]. What failed, repeatedly, was enumeration: knowing which credentials existed and where they were still trusted. Ultralytics fixed its workflow on December 5, 2024, and was republished by the attacker on December 7 through a registry token the first response had not rotated; the poisoned versions 8.3.45 and 8.3.46 exist on PyPI and nowhere in the repository's tag history [tag record]. The reviewdog attacker's opening move was a contributor PAT that, in the maintainer's words, "originated from an attack on another repository" [reviewdog #2079]. You do not lose in the hour after exposure. You lose months earlier, when a credential was created that nobody wrote down.
This guide covers machine credentials, tokens, keys and bot passwords, leaking through
code, CI and package registries, and what production systems do in response. It does not
cover human phishing, session hijacking, password managers, or the internals of vault
products. One constraint shaped the evidence: this research session's network reached
github.com and nothing else, so the corpus is the GitHub record: advisories,
incident threads, remediation commits, KEPs, policy trackers and platform documentation
read from its source repository. The Heroku (2022), CircleCI (2023) and Cloudflare
(2023–24) credential-theft postmortems exist and are directly relevant, but could not be
fetched and are cited nowhere in this page; treat their absence as a hole in this corpus,
not a judgement of relevance. No engineering blog, conference talk or paper is cited, for
the same reason.
Reconstructed across GitHub, PyPI, AWS and the scanner ecosystem: the same pipeline everywhere, with one fork that each platform resolves differently.
Strip away the product names and every leaked-credential defence in this corpus is the
same five-stage pipeline. First, make the secret findable: GitHub's tokens and
PyPI's tokens carry distinctive prefixes precisely so a regex can find them in a haystack;
PyPI's disclosure matcher is one pattern, pypi-[A-Za-z0-9-_=]+
[warehouse source],
and TruffleHog ships more than 800 of them
[trufflehog README]. Second,
detect, at three possible points: before the leak lands (push protection "blocks
pushes that contain secrets before they reach your repository"
[GitHub docs]),
after it lands (scanning of repositories, and also issues, pull request comments, wikis and
gists, which is where secrets actually travel
[GitHub docs]),
or at runtime, which is how the tj-actions compromise was caught: an outside monitor
noticed a tag move and an anomalous outbound fetch, three hours in
[issue #2463].
Third, verify: decide whether the finding is live. GitHub does this by testing the secret against the issuer's own API [validity checks], and TruffleHog's design premise is the same: "for every secret TruffleHog can classify, it can also log in to confirm if that secret is live or not." Fourth, dispose, and here the pipeline forks three ways; section 3 treats the fork as the central decision of the whole topic. Fifth, close the class: the durable fixes in this corpus are not better rotation but credential removal. Nx's final remediation line is that its packages "CANNOT be published with npm tokens any longer" [Nx advisory]; GitHub's OIDC issues a deployment credential that "is only valid for a single job, and then automatically expires" [GitHub docs]; Kubernetes spent KEP-1205 and KEP-2799 replacing tokens that were "valid for as long as the service account exists" with bound, expiring ones [KEP-1205].
PyPI's receiver verifies GitHub's HMAC signature on a disclosure, checks the token
cryptographically, then calls delete_macaroon and only afterwards emails
the owner. Two disclosure origins are wired in: GitHub and Google's Deps.dev.
Source: warehouse config.py, utils.py
AWS's automated response attaches a deny policy to the exposed key's principal. The
2024 generation (V3) denies 98 actions: IAM escalation, instance launches, Lambda
creation, and, notably, cloudtrail:LookupEvents, so the attacker cannot
watch the investigation. Everything else keeps working.
Source: AWSCompromisedKeyQuarantineV3, tracked by MAMIP
GitHub revokes its own leaked tokens automatically on public exposure. For partners' tokens the same scanner only forwards the finding, and "the service provider validates the string and then decides whether they should revoke the secret". Same detector, two different authorities.
The recurring fork: what may act automatically when a credential is found in public, and what it may do. Three platforms answered it three ways, and the difference is not philosophy, it is blast radius.
| Decision | Chosen | Rejected | Because | Evidence |
|---|---|---|---|---|
| Platform response to own leaked token | Auto-revoke on public exposure | Owner notification first | Worst case of a false kill is a retryable failure | GitHub docs |
| Registry response to disclosed token | Delete, then email | Grace period | Publishing can wait; exploitation cannot | warehouse utils.py |
| Cloud response to exposed access key | Deny-list quarantine, key stays live | Automatic key disable | The key may be load-bearing; the fence blocks escalation and forensics-watching instead | quarantine V3 |
| Nx durable fix after token theft | Trusted publishing, no npm tokens | New tokens plus tighter storage | A credential that does not exist cannot be re-stolen | Nx advisory |
| Kubernetes token model | Bound, audience-scoped, expiring | Keep legacy secret-based tokens | Legacy tokens were "valid for as long as the service account exists"; removal took KEP-2799 three feature gates and until v1.30 | KEP-2799 |
| reviewdog org access model, after | Write access withdrawn from most members | 118-member auto-invited write team | The auto-inviter was the attack surface | reviewdog #2079 |
| History scrubbing after a leak | Revoke first; scrubbing optional | Rewrite git history as the fix | "time-intensive and often unnecessary if you've already revoked" | GitHub docs |
Five failure classes cover the four incidents. None of them is "the scanner missed the secret". All of them are coordination failures around a secret that was already known to exist.
echo "Validating PR title: ${{ github.event.pull_request.title }}" under pull_request_target. The injected title used the elevated token to trigger the publish workflow, which sent the npm token to an attacker webhook. Ultralytics fell to the same class months earlier; its own fix deleted the code reading PR titles into shell variables.Windows, populations and policy sizes from the primary record. Every row links its source; nothing here is an industry-survey figure.
| Metric | Value | At | Context | As of | Source |
|---|---|---|---|---|---|
| Dependent repositories exposed | 23,000+ | tj-actions | One action, all tags rewritten | 2025-03 | GHSA |
| Exposure window | ~1 day | tj-actions | March 14–15, 2025, per advisory | 2025-03 | GHSA |
| Exposure window | 1h 49m | reviewdog | 18:42–20:31 UTC, March 11, 2025 | 2025-03 | GHSA |
| Publish to full token revocation | 5h 25m | Nx / npm | 6:32 pm to 11:57 pm EDT | 2025-08 | advisory |
| Report to confirmed registry removal | ~3h 15m | Ultralytics / PyPI | Issue 06:34 UTC; removal confirmed in-thread 09:50 UTC | 2024-12 | #18027 |
| Days from first fix to last remediation commit | 3 | Ultralytics | Dec 5 fix; Dec 8 "split PyPI publish jobs for security" | 2024-12 | commit log |
| Quarantine policy deny-list growth | 28 → 89 → 98 | AWS | V1 (tracked 2020) to V2 (2021) to V3 (2024, updated 2026-03) | 2026-03 | MAMIP |
| Auto-invited members holding write | 118 | reviewdog | The org team the attacker's PAT belonged to | 2025-03 | #2079 |
| Secret types a scanner verifies | 800+ | TruffleHog | Detectors that can "log in to confirm" | 2026-09 | README |
| Age of the mask-registration race issue | 6+ yrs | actions/runner | Opened May 2020, open, 75 comments | 2026-09 | #475 |
| Ecosystem time to retire one token class | ~5 yrs | Kubernetes | KEP-1205 (2019-08) to KEP-2799 stable in v1.30 (2024) | 2024 | kep.yaml |
All figures above are measured or stated by the primary record; none are vendor marketing. What is missing is as important: this corpus contains no fetchable number for how fast a leaked cloud key is first abused (the widely cited minutes-scale honeypot studies live on hosts this session could not reach), no total for secrets leaked per year (the annual scanning-vendor reports were unreachable), and no measured false-positive rates for any scanner. If those numbers drive a decision for you, fetch and date them yourself before relying on them.
Every source behind this page, graded. One host: this corpus is the GitHub record, by construction; the ledger shipped beside this page says exactly what that excludes.
The authoritative record: 23,000+ dependent repositories, all tags retroactively moved to one malicious commit, secrets scraped from Runner Worker memory into public logs, patched in v46.0.1.
The detection report ("this tag was just updated 3 hours back"), the payload dissection, the double-base64 confirmation, and the observation that Renovate was updating SHA-pinned repos onto the malicious hash while the thread ran.
"Pretty sure this repo got hacked" was answered in 38 seconds by the project's bot: "don't forget to star this project if you haven't already to help us reach a wider audience."
A one-hour-49-minute window on March 11, 2025; five downstream actions compromised "regardless of version or pinning method" because they resolve the inner action at run time.
Auto-invites had given 118 contributors write access; the stolen PAT "originated from an attack on another repository"; remediation removed write from most members. Includes the maintainer asking GitHub for the API logs to find the tag-mover.
A minute-by-minute timeline from PR-title bash injection under
pull_request_target, through npm token exfiltration to a webhook, to
malware that posted victims' credentials as repos on their own accounts. Ends with
npm tokens abolished in favour of trusted publishing.
A user diffed the published wheel against the tag and found a miner. The triage bot recommended upgrading to the compromised latest; the human correction and registry removal followed within about three hours.
Dec 5: stored PAT replaced with the ephemeral job token, PR-title shell interpolation deleted. Dec 7: caches disabled and pruned. Dec 8: publish pipeline split "for security". The staging shows what the first response missed, and the registry-only versions 8.3.45/46 show what it cost.
v8.3.43 and v8.3.44 exist; v8.3.41/42 were deleted after the compromise; v8.3.45/46 never existed in git at all. Inferred, and stated as inference: those two versions were published directly to the registry with a stolen token.
The runner maintainer, 2020: "The mask needs to be added before the output is registered. … The output is streaming, so too late if value is already printed." Literal registration plus streaming output is exactly the gap double-base64 walked through five years later.
HMAC-verified disclosures from GitHub and Deps.dev; verify_signature_only,
then delete_macaroon, then the email. The ordering is the policy:
revocation does not wait for a human.
The managed policy AWS attaches to exposed keys, diffed over time by a tracker repo: 28 denied actions in V1 (tracked 2020), 89 in V2 (2021), 98 in V3 (first seen 2024-08-21, still updated 2026-03). A deny list, not a kill: the fence model in public.
"Over 800 secret types", each mapped to an identity, and for each "it can also log in to confirm if that secret is live or not. This step is critical to know if there's an active present danger."
The gitleaks maintainer built verification himself, 43 commits, opt-in, status enums, then closed the PR unmerged (as with its successor #1519) and later froze the project. The README today: gitleaks is "a tool for detecting secrets". The reason is not recorded; the outcome is.
"Maintained by the folks who made Gitleaks, including the original author", with "Secrets Validation" as a headline feature: validate liveness with HTTP requests from inside the rule definition. The 2024 argument resolved itself in the 2026 tool.
The self-hostable server behind canarytokens.org: mint deliberately plantable credentials whose only legitimate use is never, so any use is a high-confidence alarm. The inverse of everything else on this wall: a leak you want.
The problem statement that still reads as the whole field's thesis: tokens that are not audience bound let "any recipient of a JWT masquerade as the presenter to anyone else", and tokens that are not time bound stay valid "for as long as the service account exists".
The removal plan for the legacy tokens: no-auto-generation, then tracking of remaining use, then cleanup, three feature gates landing stable in v1.30. The tracking gate exists because you cannot delete what you cannot prove unused, which is the enumeration problem again, inside the fix.
The ecosystem's measurable proxy for this failure class: does each workflow token follow least privilege, because "attackers may use a compromised token with write access to … push malicious code into the project". Both March 2025 attacks ran through exactly that.
"If a valid OAuth token, GitHub App token, or personal access token is pushed to a public repository or public gist, the token will be automatically revoked." Also: unused tokens die after a year.
For other issuers' secrets, GitHub forwards the finding and "the service provider validates the string and then decides whether they should revoke the secret, issue a new secret, or contact you directly. Their action will depend on the associated risks to you or them."
"Rotate the affected credential immediately … removing secrets from your Git history … is time-intensive and often unnecessary if you've already revoked the credential." The docs also enumerate the scan surface: issues, PR comments, discussions, wikis, gists.
"Rather than alerting you to credential leaks after the fact, push protection blocks pushes that contain secrets before they reach your repository", at the command line, the UI, the API and file upload.
Opt-in checks that "periodically check the validity of a detected credential by sending the secret to the issuer and testing it against APIs provided by that service", with deliberately least-intrusive endpoints.
The Actions OIDC concept doc: the cloud "issues a short-lived access token that is only valid for a single job, and then automatically expires", and the first listed benefit is "No cloud secrets". This is the per-job exit at the top of Figure 3.
The official action's quick start is titled "OIDC, recommended", and its security recommendations open with "Use temporary credentials when possible. OIDC is recommended because it provides temporary credentials and it's easy to set up."
Seven rungs. The first three fit in a weekend on a sandbox account; the line from toy to real is crossed at rung four, where revocation meets production.
Mint a canary credential (a Canarytokens AWS key or one of your own dead tokens), plant it in a private test repo, a CI variable and a wiki page, then use it once.
Done when: the alert reaches a human, and you can say from which of the three plantings. Teaches: which of your leak surfaces are actually watched.
Run TruffleHog or a gitleaks-lineage scanner across your organisation's repositories, including issues and wikis where supported. Split findings into live and dead.
Done when: you have two counts, and every live finding has an owner. Teaches: why the verify-vs-detect decision in section 3 dominates triage cost.
For one system, enumerate every long-lived machine credential: issuer, holder, storage location, scopes, revocation API, and who may call it. The Ultralytics second wave is the test case: would your list have contained the registry token?
Done when: a named person can produce the list in under an hour, unaided. Teaches: enumeration is the actual hard problem; everything else is tooling.
Stand up a small service in the warehouse mould: accept a signed report of a leaked token of your own format, verify the signature, revoke, then notify. Feed it from your scanner's webhook.
Done when: a planted leak of your own token format is dead within minutes with no human action, and the owner has an email explaining why. Teaches: revocation-before-notification as a code path, not a runbook step.
Pick one production credential and revoke it on purpose during business hours, with the owning team watching. Measure what breaks and for how long. This is the AWS-fence lesson inverted: if you cannot afford to run this rehearsal, you cannot afford the real response either.
Done when: mean time to re-issue is known, and the one credential whose revocation would cause an outage is on a remediation list. Teaches: the blast radius of revocation, which section 3 showed drives every platform's policy.
Move one CI-to-cloud or CI-to-registry path from a stored secret to OIDC or trusted publishing, the Nx endgame. Then delete the old secret and watch what complains.
Done when: the old credential is revoked and the pipeline is green for a week. Teaches: the KEP-2799 sequence in miniature: stop issuing, measure use, delete.
Alert on tag moves and unexpected pin changes in the third-party actions and images your credentialed jobs execute, and gate your dependency-update bot's pin bumps behind review for that set.
Done when: a test tag-move on a fork raises an alert before the next scheduled job runs. Teaches: why tj-actions' consumers had no move to make; the ones who noticed were the ones watching references, not code.
What actually found this material. Half of it is ordinary search; the other half is treating GitHub itself as the archive, which works from any network that can reach it.
site:github.com advisories "supply chain" secrets leaked CI logstj-actions OR reviewdog OR ultralytics compromised issue postmortemrepo:<org>/<repo> in:title compromised OR hacked OR "rotate"https://raw.githubusercontent.com/github/advisory-database/main/advisories/github-reviewed/<YYYY>/<MM>/<GHSA-id>/<GHSA-id>.jsongit clone --filter=blob:none --no-checkout https://github.com/<org>/<repo>git log --since=<incident> --until=+7days -- .github/workflows/publish.ymlgit log -1 v<bad-version> # a missing tag means a registry-only releasegit show <fix-commit> -- .github/workflows/ # what the first response changedis:pr is:closed is:unmerged sort:comments-desc verifyrepo:kubernetes/enhancements path:keps/sig-auth tokenrepo:actions/runner is:issue mask secret sort:comments-descrepo:z0ph/MAMIP AWSCompromisedKeyQuarantine # AWS managed-policy diffs, datedrepo:github/docs path:content/code-security secret-scanningrepo:pypa/warehouse path:warehouse/integrations/secrets