A CI job is the most privileged and least guarded machine most teams operate: it holds every credential you own and, on a public repository, runs code proposed by anyone on the internet. This guide reconstructs, from the postmortems of Codecov, CircleCI, Nx, tj-actions and Salesloft, why one long-lived secret in that job keeps producing total compromise, and why every serious remediation since 2021 converges on the same move: delete the standing secret and mint a short-lived, cryptographically-bound identity per job.
The problem stated without the acronyms, who has been forced to solve it in production, and the one thread that connects incidents that look unrelated.
State it without the technology's name and it sounds absurd. You have one machine that holds a copy of every credential your organisation uses to reach anything: your cloud account, your package registry, your artifact store, your signing keys, your database. That same machine automatically runs code, and on any open-source project the code it runs is proposed by strangers. It is the one system that is simultaneously the most privileged and the most exposed, and for most of the last decade it authenticated to everything downstream with secrets that never expired. That machine is the build pipeline, and it has quietly become the highest-value target in the software supply chain.
The reason it is worth a field guide now is that the incidents stopped being rare. Between 2021 and 2026 the same shape recurred across Codecov, Travis CI, CircleCI, PyTorch, Coinbase, the tj-actions ecosystem, Nx and Salesloft. In every case the loss was not a clever cryptographic break. It was a credential that lived in a build job and should not have, or a build job that ran attacker-controlled code and should not have. The surprise, once you line the postmortems up, is how little the attackers had to do and how completely the remediation trajectory has converged: platform vendors, package registries and cloud providers are all, independently, deleting the long-lived secret and replacing it with a short-lived identity that is bound to a specific workflow and expires in minutes.
Scope. This guide is about the credential and trust boundary of the CI/CD job itself: how build-time secrets leak, how untrusted code gets to run with them, and how short-lived federated identity replaces them. It leans on GitHub Actions because that is where the public evidence is densest, but the failure classes are platform-independent and the GitLab and Sigstore equivalents are drawn in for contrast. It deliberately does not cover dependency and package integrity as such (typosquatting, malicious transitive dependencies, the mechanics of the Shai-Hulud worm), nor the human-endpoint compromise that seeds some of these incidents, nor artifact signing beyond where it touches CI identity. Those are adjacent guides.
Every CI system has the same six parts. What separates a compromised pipeline from a resilient one is how two of them are wired: how the job authenticates downstream, and how untrusted input reaches the runner.
Strip GitHub Actions, GitLab CI, CircleCI and Jenkins down and the same skeleton appears. A trigger fires from a version-control event. A workflow definition says what to run, read from somewhere in the repository. A runner executes the steps, either ephemeral (a fresh VM per job) or self-hosted (a persistent machine you own). A secret store injects credentials into the job. Downstream, a resource such as a cloud account or a package registry decides whether to honour whatever the job presents. That skeleton is not where systems differ. They differ at two joints, and both joints are where every incident in this guide happened.
The first joint is how the job proves who it is to the resource. The old answer, still the majority answer, is a long-lived secret: an AWS access key, an npm token, a registry password, stored in the secret store and injected as an environment variable. The key never expires, is valid from anywhere, and is indistinguishable from the legitimate holder the instant it is copied. The new answer, which every major platform is now pushing, is federated identity via OpenID Connect. The CI platform runs an OIDC identity provider; for each job it mints a signed JWT whose claims describe exactly what is running (which repository, which branch, which workflow, which environment). The downstream resource verifies that JWT against the provider and against a trust policy, and hands back a credential scoped to the life of the job. GitHub, GitLab and CircleCI all now document this path; GitLab sets the token's lifetime to "the job's timeout if specified, or 5 minutes if no timeout is specified" (GitLab docs), and Sigstore's Fulcio uses the identical GitHub identity token to issue short-lived signing certificates (Fulcio docs).
The second joint is where the workflow definition and the checked-out code come
from relative to who can trigger the job. A standard pull-request trigger runs the
fork's proposed code but, by GitHub's design, withholds secrets and write access from it. The
pull_request_target trigger, introduced so that workflows could label or comment
on fork PRs, relaxes that: it runs in the context of the target repository, with its
secrets and a writable token, while still being triggerable by the fork. GitHub's own Security
Lab named the resulting hazard the "pwn request" in 2020. The maintained static analyzer
zizmor is blunt that the common mitigation folklore is wrong: "Many online resources
suggest that pull_request_target and other dangerous triggers can be used securely by ensuring
that the PR's code is not executed, but this is not true"
(zizmor audits).
Ephemeral runners bound the damage of code execution to one job. Self-hosted runners on public repos are the opposite: a persistent machine reachable by any contributor's PR, which is how the PyTorch supply-chain proof of concept reached "full access to the runners" and stole persistent PATs.
Documented at: Stawinski/Khan, DEF CON 32
The automatic GITHUB_TOKEN was write-by-default for years. GitHub changed the
default to read-only in February 2023, but only "for new repositories owned by personal
accounts"; existing repos kept the permissive default, so least privilege here is opt-in
for most of the installed base.
Runs this way at: GitHub, 2023
In the OIDC path, the sub claim is the load-bearing control: it encodes the
repo, branch and workflow, and the downstream trust policy must pin it exactly. Bind it
loosely and the "keyless" win becomes a role assumable by the whole internet.
Detailed at: GitHub docs, Datadog
Five forks in the road, each with the option the evidence favours, the option that keeps losing, and the specific condition under which the loser is actually right.
pull_request trigger, which withholds secrets from fork code by designpull_request_target takes its workflow from the default branch, and checkout v7 refuses fork code under itpull_request_target with a checkout of the PR head@v45 or @v1The remaining two decisions, runner choice and the exact shape of the trust policy, are captured in the table. The pattern across all five is the same: the convenient default trades a small amount of daily friction for an unbounded worst case, and every team that took the loss wrote the same remediation.
| Decision | Chosen | Rejected | Because | Flips when | Evidence |
|---|---|---|---|---|---|
| Downstream auth | Per-job OIDC identity | Long-lived key in env | Stolen token dies with the job | Resource can't federate; use shortest-lived token + rotation | GitHub, 2025 |
| Fork PR privileges | pull_request, no secrets | pull_request_target + checkout | Pwn request runs strangers' code privileged | Must act on fork PRs; label-gate a secretless split | GH Security Lab |
| Action reference | Full commit SHA | Floating tag | Tags are mutable and were moved | Never for security; automate SHA bumps | Datadog, 2026 |
| Runner for public repos | Ephemeral, per-job | Persistent self-hosted | Self-hosted persists attacker access | Need special hardware; require approval + ephemeral | DEF CON 32 |
| Trust policy subject | sub pinned with exact match | Missing or wildcard sub | Loose sub = role assumable by any repo | Many repos need it; enumerate them, don't wildcard the org | Datadog, 2023 |
Seven published incidents, grouped by the assumption that failed. Three failure classes account for all of them, and naming the classes is more useful than the stories.
Lined up by root cause rather than by company, the incidents collapse into three classes. Class A, the standing secret: a long-lived credential lived in the pipeline and one copy was total compromise (Codecov, CircleCI, Salesloft). Class B, the pwn request: untrusted code ran with the target repository's privileges (Nx, the tj-actions chain, Travis CI). Class C, the confused deputy: a federated identity was trusted too broadly, so a role meant for one repository was assumable by any (the Datadog and Tinder findings). Class A is the oldest and the most expensive. Class B is the one platform vendors are now engineering out of the defaults. Class C is the failure mode the "keyless" migration introduces if the subject claim is bound carelessly, which is the trap worth flagging loudest because it hides inside the fix for Class A.
$(env) and POST it to their server, harvesting secrets from every CI run that piped the script.pull_request_target workflow interpolated an unsanitised PR title into a shell step, letting an attacker inject bash that stole the npm publishing token, then publish malicious Nx packages.${{ ... }} expansion of attacker-controllable metadata is a code-injection sink. Pass such values through an intermediate environment variable, never straight into a run step.sub condition. As Datadog put it, "a GitHub Action from any GitHub repository can assume the role." One real case came from a config-parsing bug that silently dropped the subject condition.aud and an exact sub; GitHub later moved to immutable subject claims (owner-id + repo-id) so a recycled name cannot mint a matching token.No incident here required breaking cryptography or finding a zero-day in the platform. Each was a credential that should not have been reachable, or code that should not have run privileged. The academic measurements say why this keeps happening: Koishybayev et al. found 99.8% of workflows overprivileged, and ARGUS found injectable code in 5,298 workflows. The defaults produce the vulnerability at population scale, so the fix has to be a default change, which is exactly what the platforms are now shipping.
The figures that size the problem and the fix. Every row carries its source and the date it was true; the note below separates what was measured from what is a direction of travel.
| Metric | Value | At | Context | As of | Source |
|---|---|---|---|---|---|
| Undetected exfiltration window | ~65 days | Codecov | Tampered uploader read CI env vars; found by a customer's SHA check | 2021 | Codecov |
| Repos using the compromised action | 23,000+ | tj-actions | Pinned mostly by mutable tag | 2025 | Wiz |
| Repos that actually leaked secrets | 218 | tj-actions | From one PAT four repos upstream | 2025 | Unit 42 |
| Accounts / repos exposed | 2,180 / 7,200 | Nx | One stolen npm publish token, three waves | 2025 | Wiz |
| Orgs hit via reused OAuth tokens | 700+ | Salesloft Drift | None breached in their own systems | 2025 | Google TIG |
| IAM roles assumable by any repo | 500+ / 275 accts | Datadog scan | Missing sub condition | 2023 | Datadog |
| Workflows overprivileged (read-write) | 99.8% | 447k workflows | Academic corpus scan | 2022 | USENIX |
| Workflows with code injection | 5,298 | 2.78M scanned | ARGUS taint analysis | 2023 | USENIX |
| Secrets leaked to public GitHub | 23.8M | 2024 calendar year | +25% year on year | 2025 | GitGuardian |
| 2022-leaked secrets still valid | 70% | GitGuardian | Long-lived secrets are rarely revoked | 2025 | GitGuardian |
| GitLab CI ID token lifetime (default) | 5 min | GitLab | Or the job timeout if set | 2026 | GitLab |
| New npm publish token max lifetime | 7 days | npm / GitHub | Classic tokens being deprecated | 2025 | GitHub |
Measured: the Codecov window, the tj-actions counts, the Nx and Salesloft exposure figures, and the two academic corpus numbers are all from primary incident reports or peer-reviewed scans. Direction of travel, not a floor: the 5-minute and 7-day lifetimes are the current defaults or announced plans, and are trending shorter, not longer; treat them as ceilings to design under, not guarantees. Derived: the gap between 23,000 repos exposed and 218 that leaked is roughly a 1% realisation rate for this particular payload, which says the loss was bounded by the attacker's collection method, not by any control the 22,800 other repos had in place. There are no vendor marketing figures in this table; that was deliberate.
Every source behind this page, graded. Filter by kind.
The tampered uploader appended $(env) to an exfiltration POST, reading every
secret in every CI run that piped it, undetected for ~65 days until a customer's SHA check.
A stolen SSO session cookie reached production; the attacker "extracted encryption keys from a running process," making encryption at rest moot. Vendor advice: adopt OIDC.
A pull_request_target workflow interpolated an unsanitised PR title into
bash, stealing the npm token. Remediated with Trusted Publishers and manual approval.
Reconstructs the four-repo pwn-request chain from a leaked SpotBugs PAT through reviewdog to tj-actions, whose tags were moved to a memory-dumping commit. 218 repos leaked secrets.
Access to a source-control account led to an AWS environment and the theft of customers' OAuth tokens, exposing 700+ organisations that were never themselves breached.
CVE-2021-41077: secure env vars of all public projects leaked into fork PR builds. The silent, postmortem-free disclosure drew as much criticism as the bug.
The authoritative record for the tj-actions compromise: secrets discoverable by reading Actions logs, including access keys, PATs, npm tokens and private RSA keys.
Found 500+ IAM roles across 275 accounts assumable by any repo from a missing sub
condition; one case caused by a Terraform duplicate-key parsing bug.
Independent confirmation of the missing-subject class from a second team, with a black-box assessment tool and real case studies of externally assumable roles.
GitHub's own team named and explained the pull_request_target hazard five
years before the incidents it caused: such workflows can "steal or use a GITHUB_TOKEN."
Distils the incident wave into three controls: pin actions to a SHA ("the only way"),
avoid pull_request_target, and restrict token permissions.
23.8M secrets leaked to public GitHub in 2024, up 25% year on year; 70% of secrets leaked in 2022 are still valid, because long-lived credentials are rarely revoked.
Across 447k workflows in 213k repos, 99.8% were overprivileged with read-write repo access and 23.7% were triggerable by a pull request while using the repo's code.
Taint analysis over 2.78M workflows found code injection in 5,298 workflows and 80 actions, a discovery rate 7x higher than pattern-based scanners.
Demonstrated, against PyTorch and others, that "insecure defaults" let a fork PR reach a self-hosted runner and backdoor major projects.
A one-line typo PR made them "contributors" and gave "full access to the runners," used to steal GitHub and AWS tokens and reach PyTorch releases.
New npm publishing moves to 7-day granular tokens and Trusted Publishing (OIDC); classic never-expiring tokens will be deprecated entirely.
actions/checkout v7 "refuses to fetch fork pull request code in pull_request_target and workflow_run workflows," removing the pwn-request default from the most common action.
The maintained static analyzer states plainly that "run it safely" folklore for
pull_request_target is false, and treats all metadata expansions as injection.
The reference OIDC client bakes in an exact-match sub condition and warns
against loose operators: "Avoid ForAllValues: in Allow statements."
The same principle predates CI OIDC: legacy JWTs "are not audience bound" and "not time bound," so a compromised one is valid until deleted. New tokens bind audience and time.
The second major CI vendor implements the identical pattern; the ID token's lifetime is "the job's timeout if specified, or 5 minutes if no timeout is specified."
Do these in a throwaway org against a throwaway cloud account. The middle rungs cross from toy to production-shaped; the ones after are where reading becomes judgement.
Write a workflow with a run: step that echoes ${{ github.event.issue.title }}.
Open an issue titled foo"; whoami; echo " and watch the command execute.
Done when: your injected command appears in the job log. Teaches: ${{ }} is string substitution before the shell sees it, so metadata is code.
Add a static cloud key as a repo secret. Add a pull_request_target workflow that
prints env. From a second account, open a fork PR and read the secret in the log.
Done when: you retrieve the secret from an account with no write access to the repo. Teaches: the pwn request and the blast radius of one env-injected secret.
Delete the static key. Create an AWS role trusting GitHub's OIDC provider with an exact
sub match, and use aws-actions/configure-aws-credentials to assume it.
Done when: the job calls AWS with no stored key and the credential expires after the run. Teaches: per-job identity, and that a stolen credential now dies with the job.
Change the sub condition to a wildcard, or remove it. From an unrelated repo you
control, assume the same role.
Done when: a repo that should have no access mints AWS credentials. Teaches: the confused-deputy failure the OIDC migration introduces if sub is loose.
Replace every @v3-style reference with a full commit SHA. Add zizmor
to CI and set the automatic GITHUB_TOKEN to read-only by default.
Done when: zizmor passes and no action is referenced by a mutable tag. Teaches: supply-chain pinning plus detection that catches the next dangerous trigger before merge.
Set up npm (or PyPI/crates) Trusted Publishing so a release is authenticated by the workflow's OIDC identity, with no publish token anywhere in the repo.
Done when: a tagged release publishes with provenance and zero stored registry credentials. Teaches: the same identity model extends from cloud access to package release.
If you must use a self-hosted runner, run it per-job in a fresh container and require manual approval before workflows run for first-time contributors.
Done when: a fork PR cannot run on your runner without an approval, and each job starts from a clean machine. Teaches: the runner is a trust boundary; persistence and auto-run are what attackers need.
The queries that actually surfaced this material. The domain vocabulary
(pull_request_target, sub claim, pwn request, Trusted Publishing) is
what turns a vague search into the primary source.
<vendor> "incident report" secrets rotate "environment variables"github actions supply chain "pull_request_target" postmortem"we compromised" OR "we executed" supply chain CI/CD self-hosted runnernpm token stolen "trusted publishing" remediation postmortemgithub oidc "sub" claim missing trust policy assume role research"pwn request" pull_request_target site:securitylab.github.comGITHUB_TOKEN default permissions read-only changelogconfigure-aws-credentials README "sub" immutable subject"characterizing the security" github ci workflows usenixARGUS taint analysis github actions code injectionpath:docs zizmor dangerous-triggers template-injectionKEP bound service account tokens audience time boundstate of secrets sprawl leaked github valid still activeoidc "id token" lifetime CI job "5 minutes" OR "expire"threat intelligence OAuth tokens supply chain "700 organizations"trusted publishing rubygems crates.io npm nuget adopted