Untrusted Build Step

A build step that downloads a script and pipes it to a shell is one of the most common lines in continuous integration, and it is a remote-code-execution primitive aimed at your most privileged environment. Between 31 January and 1 April 2021 an altered Codecov Bash Uploader did exactly that: it exfiltrated the environment variables of every CI job that ran it, undetected for roughly two months, and was found by a customer comparing the script's checksum with the published one.

The failure is not the script. It is that a CI job's environment is a flat bag of long-lived credentials that every step in the job can read. A step needing permission to upload coverage could also read the cloud deployment key, the registry password, the signing key and the package-publishing token, because they are all in the same process tree.

Why it matters

Your build system is a production system holding your most powerful credentials, with the weakest review of what executes inside it. Dependencies are pinned, scanned and reviewed; build steps are pasted from documentation. The asymmetry is not defensible once stated, and it is nearly universal.

Damage is also not bounded by the compromise window, because a static long-lived token remains valuable after exfiltration: a key stolen in February works in July. That is why the only remedy after such an incident is mass rotation, which is expensive, disruptive and frequently discovers credentials nobody knew existed.

Implementation patterns

• Short-lived credentials through workload identity federation. The job exchanges a signed identity token for a credential valid for 15 minutes, scoped to one action. An exfiltrated credential is worthless by the time it is used, which is the single highest-value change available here.
• Split the pipeline by privilege. Build and test in a job holding no credentials; publish and deploy in a job that runs no third-party steps. Untrusted code and privileged credentials must not share a process tree, and this separation costs nothing but arrangement.
• No standing secrets in the environment. Fetch a secret at the moment of use, in the step that needs it, at the narrowest scope.
• Pin third-party steps by digest or checksum, not by tag or branch, so the code reviewed is the code that runs. A remote script has no version, no digest and no review unless you supply them.
• Vendor what you can. A copy of the uploader in your repository is reviewable, diffable and updated deliberately.
• Egress control on build networks. A build that can reach any host can exfiltrate to any host; an allowlist turns the attack into a failed connection and a log line.
• Rehearse rotation. Teams that recovered from Codecov in hours were the ones that had rotated before.

Industry example

Codecov is the clearest case because the mechanism was so plain, and it was not isolated: compromises of widely used CI actions and packages have repeatedly used the same primitive, reading GITHUB_TOKEN, cloud keys and publishing tokens out of the environment of jobs that merely built software. SolarWinds in 2020 made the more general point one layer up — the build system is a target precisely because everything downstream trusts its output.

The corresponding defence is now standard in the major CI platforms: identity federation for cloud credentials, permission scoping per job, and digest pinning of actions. The controls exist; the adoption gap is the risk.

Failure scenarios

• A coverage or linting step reading deployment credentials, because they are in the same environment.
• A long-lived token used months after exfiltration, with the breach discovered from the resulting activity rather than from the pipeline.
• A third-party action pinned to a tag that the author repoints, replacing reviewed code silently.
• Secrets printed into logs by a step that dumps its environment for debugging, then retained in a log store with wider access than the secret store.
• A pull-request build from a fork given repository secrets, which hands them to anyone who can open a pull request.
• Rotation that cannot be completed, because nobody has an inventory of what the exposed credentials were.

Trade-offs

When not to use it

A job that holds no credentials needs none of this — a fork's pull-request build, a documentation site, a lint-only pipeline — and that is also the design principle in miniature: arrange most jobs to hold nothing worth stealing, and concentrate the controls on the few that must hold something. Full egress allowlisting is the one item to defer unless you handle regulated data, because it breaks legitimate builds and needs ongoing maintenance. Choose the privilege split first, since it makes every other control cheaper.

Interview question

Q: A widely used CI step you depend on is compromised and exfiltrates job environments for two months. Tell me what an attacker gets in your pipeline today, and the two changes that would most reduce that.

What a strong answer covers: enumerating what is actually in the job environment and noting that every step can read all of it; that static tokens stay valuable long after exfiltration, so the blast radius is not bounded by the window; federated short-lived credentials as the highest-value change and the privilege split as the cheapest; digest pinning and checksum verification of third-party steps; why detection is unlikely without a pinned expectation to violate; and treating rotation as a rehearsed procedure rather than an emergency.

Quick check

Quiz: Why does pinning a third-party action by tag not protect you? Because a tag can be repointed by its author, so the code that runs is not necessarily the code you reviewed; pin by commit digest instead.

Flashcard: What is the cheapest structural defence against a malicious build step? — Split the pipeline by privilege: build and test with no credentials at all, and publish or deploy in a separate job that runs no third-party code.