Architecture Views

Does a build environment get reused between jobs, and if so, what exactly is guaranteed to have been removed?

Context

Reusing a warm build environment is the single largest performance win available to a CI platform. A pool of containers with the dependency layers already pulled and the toolchain already warm turns a forty-second start into a two-second one, and every CI vendor has been tempted by it. The problem is that the guarantee required is not "we cleaned up the working directory". It is "nothing the previous job wrote — a file outside the workspace, a kernel object, a cached credential in an agent socket, a modified binary on the PATH, an entry in a package manager's global store — can influence or be read by this one". That guarantee is impossible to audit and its violations do not fail tests; they fail silently, in one job in ten thousand, and look like a flake.

Decision

A sandbox is created for exactly one job and destroyed after it. No sandbox serves two jobs, and the destruction happens whether the job succeeded, failed, timed out or was killed. Warmth is bought back by pre-booting sandboxes in a pool before they are claimed, never by reusing one after a job has run inside it.

How it is realised on AWS

The sandbox manager maintains a warm pool of pre-booted microVMs on each isolation host. A microVM in the pool has booted a base image and has never executed tenant code; at dispatch, one is claimed, the job's workspace and credentials are injected, and the VM is marked non-reusable. On completion the VM is terminated and its backing storage discarded, and the pool replenishes asynchronously from the host's capacity. A VM that has been claimed is never returned to the pool, even if the job never started.

Options weighed

• ChosenSingle-use sandbox per job, warm pool of never-used sandboxes: Makes residue impossible rather than unlikely, and moves the performance problem to cold start where it is measurable
• RejectedReused container pool with a cleanup step between jobs: Cheapest and fastest; the cleanup guarantee cannot be audited and its failures look like flakes
• RejectedReused for trusted builds, single-use for fork builds: Assumes a trusted build cannot be hostile, which a compromised dependency disproves
• Right elsewhereA whole dedicated instance per job: Right where jobs are long and few, or where regulatory isolation is per-customer; the cold start and cost do not survive 1.9M jobs a day

Consequences

What it buys

• Cross-job contamination stops being a class of bug, which removes an entire category of unreproducible flake
• The teardown path is exercised 1.9M times a day, so it is reliable by the time it matters
• Capacity accounting becomes simple: one job occupies one sandbox for a measurable duration, which is what makes cost per job-minute meaningful

What it costs

• Cold start becomes the platform's hardest performance problem and the warm pool a permanent cost line — up to 8% of compute minutes idle by design
• Dependency warmth must be recovered through the cache and mirror rather than through a reused filesystem, which makes cache hit rate load-bearing
• Cost per job-minute is materially higher than a reused-container platform's, and that gap is visible to anyone comparing vendors

Choose differently when

If every repository in the estate were internal, every contributor employed and vetted, and every dependency vendored and reviewed before entry, the residue risk would be low enough that a reused pool with a cleanup step would be the better economic choice. The decision is justified by roughly 300 fork pull requests a quarter and by a dependency graph the company does not control — remove both and the calculation changes.

Why it holds up over time

The rule is about lifecycle, not about technology. Whatever the isolation primitive becomes — a lighter hypervisor, a hardware partition, something not yet built — "one job, then destroyed" remains expressible and remains the property that makes every other isolation guarantee meaningful. The cost of the rule falls as boot times fall; the cost of abandoning it does not fall at all.

Must the boundary between two concurrently running jobs survive a kernel exploit, and does the answer differ for a build of the company's own code?

Context

Containers isolate by kernel namespace, which means a kernel vulnerability reachable from inside the container is a path out of it. Syscall filtering and user-space kernels narrow that surface considerably but do not remove the shared-kernel premise. The question is not whether such exploits are common — they are rare — but what the consequence is when one lands: on a shared-kernel host, another tenant's job, its credentials and its source are in reach. The second half of the question is more interesting. It is tempting to run fork builds in microVMs and trusted builds in containers, because most jobs are trusted and containers are three times cheaper. That reasoning assumes a trusted build cannot be hostile, which has not been true since dependency compromise became the dominant supply-chain attack: a malicious postinstall script in a transitive dependency of the company's own service executes with exactly the privileges of a trusted build.

Decision

Every job runs in a hardware-virtualised microVM. Isolation strength is identical for a trusted branch build, a fork build, a scheduled run and an interactive session. What varies between trust classes is what the sandbox is given — credentials, cache write authority, publish identity — never how well it is contained.

How it is realised on AWS

Nested-virtualisation-capable EC2 instance families run a lightweight microVM hypervisor, one VM per job, with the per-job agent inside the VM and the sandbox manager outside it. Jobs that declare nested container execution get it inside their own VM without host privileges. Two instance families are qualified at all times so a capacity shortage in one is not a platform-wide throughput event.

Options weighed

• ChosenmicroVM per job for every trust class: One substrate to harden, one cold-start number to optimise, and no capacity stranded in the wrong pool
• RejectedHardened containers with syscall filtering for all jobs: Fastest and cheapest, and adequate until the first kernel escape, at which point it is adequate for nothing
• RejectedmicroVMs for untrusted, containers for trusted: The common design; it prices the fork risk correctly and the dependency-compromise risk at zero
• DeferredSeparate physical fleets per trust class: Worth revisiting if a regulator requires physical separation; it doubles the capacity-planning problem for a marginal gain over microVMs

Consequences

What it buys

• A kernel-level compromise inside one job yields no access to another, which is the only honest answer to "what if a build roots the kernel"
• One substrate means one hardening effort, one cold-start budget and one capacity pool — no stranded capacity in a fleet nobody is using this hour
• Nested container execution can be offered safely, because the nesting happens inside the tenant's own VM

What it costs

• Higher cost per job-minute and a slower start than containers, borne on every job rather than only on risky ones
• Dependence on a narrower instance market, which is a capacity-planning risk with no software mitigation
• Some workloads that assume host features will need explicit support rather than working by accident

Choose differently when

If the estate had no external contributors and every dependency were vendored and reviewed, the dominant residual risk would be a platform bug rather than tenant code, and hardened containers would be the right economic answer. Equally, if a future container runtime offered a genuine hardware-backed boundary at container cost, this decision becomes an implementation detail rather than a trade-off.

Why it holds up over time

The requirement is "the boundary survives a kernel exploit", which is a property, not a product. It has been satisfied by different technologies each decade and will be satisfied by others. Stating it as a property rather than as "we use microVMs" is what lets the substrate be replaced without revisiting anything else in this record set.

At what moment, and from what evidence, does the platform decide whether this run's code is trusted — and can that decision ever be revised upward?

Context

Every credential and cache decision in the platform depends on one bit: is the code about to execute code the company has reviewed? Getting that bit from the wrong place is the root of most CI security incidents. If it is derived from the pipeline definition, the fork's own definition can claim to be trusted. If it is derived at credential-request time, a job that has already started can escalate. If it is recomputed per step, a step can change it. And if it is computed from the pull request's target repository rather than from the head's provenance, a pull-request trigger that runs the head's code with the base's permissions is exactly the well-documented mistake that has leaked deployment credentials from several widely-used CI systems.

Decision

The trust class is computed once, at admission, from the trigger event and the repository's membership records — never from anything inside the pipeline definition or the commit. It is written as an immutable column on the run and attached to every job, credential request and cache operation for the run's life. No path exists to raise it. Lowering it is possible only by cancelling the run.

How it is realised on AWS

The admission service reads the signed webhook payload, resolves the head repository and contributor against the tenant's membership and approval records, and writes trust_class on the run row before the run enters the queue. The job identity minted by identity federation binds tenant, repository, ref and trust class into the token, so the secret broker and cache service evaluate the class from the token rather than by looking it up again. A run's effective definition is recorded alongside the class, so an auditor can see both what ran and under what posture.

Options weighed

• ChosenComputed once at admission from the event, immutable on the run: One place to get right, one column to audit, and no escalation path to reason about
• RejectedEvaluated per credential request against current state: Flexible, and creates a window in which a membership change mid-run changes a running job's privileges
• RejectedDeclared in the pipeline definition with policy validation: Lets the artefact under test participate in deciding how much it is trusted
• Right elsewhereMaintainer-approved promotion of a fork run to trusted: Reasonable for a small open-source project where one person reviews every diff; at 610 tenants it becomes a click that nobody reads

Consequences

What it buys

• The security posture of a run is a single auditable value, set before any capacity is committed
• Credential and cache services need no membership lookups on the hot path, because the class travels in the token
• The fork case becomes ordinary: it is a class, not an exception handled by a special code path

What it costs

• A contributor who becomes trusted mid-run does not benefit until the next run, which occasionally confuses maintainers
• The classifier is a small piece of code with disproportionate consequence, and needs test coverage out of proportion to its size
• A legitimate need to run a fork's code with credentials has no in-platform answer, only the trusted re-run at merge

Choose differently when

If the platform served a single team with no external contributors, the class would always be the same value and the machinery would be pure overhead. And if a future source-control platform offered a cryptographically attested contributor-trust signal at event time, the classifier would become a thin adapter over it rather than a decision the platform makes itself.

Why it holds up over time

"Compute the security-relevant fact once, from the least manipulable input available, and make it immutable" is a design rule older than CI and will outlast this platform. What changes over time is which input is least manipulable; the rule does not.

Who is allowed to write a cache entry that a later build will restore and execute?

Context

A build cache is not passive data. Restored into a workspace, it becomes compiler output, dependency binaries, and sometimes executables on the PATH. If an untrusted run can write an entry that a trusted run later restores, the untrusted run has achieved code execution inside the trusted build — and therefore inside whatever the trusted build is allowed to publish and deploy. This is not hypothetical: it is the most-exploited class of CI vulnerability in practice, because the cache is the one shared mutable surface a fork build legitimately touches. The counter-pressure is real too. Cache hit rate is the single largest determinant of build latency, and the contributors most in need of a fast build are exactly the ones who cannot be allowed to write.

Decision

Any run may read the cache. Only a run classified as trusted may write it. Every entry is integrity-verified on restore, and a verification failure is treated as a cache miss rather than as an error. The slower fork build is accepted as the price, and the platform tells the contributor why.

How it is realised on AWS

Cache write credentials are issued by the broker only against a job identity carrying the trusted class; an untrusted job's request is refused with a distinct, explicit reason rather than silently returning nothing. Entries are content-keyed with declared fallback keys and carry a digest verified on restore. Restore is served from a read-wide path that requires no write credential at all, so a fork build's cache read is not a privilege it could misuse.

Options weighed

• ChosenRead wide, write privileged, integrity-verified on restore: Keeps most of the hit rate and removes the code-execution path entirely
• RejectedShared per-repository cache writable by any branch or fork: Best hit rate, and a direct code-execution path from a stranger into the production build
• RejectedPer-branch caches with read fallback to the default branch: Better than shared-writable, and still lets an untrusted branch's entry be read by a trusted build via fallback
• DeferredContent-addressed cache with attested writers only: The right long-term answer and a natural extension of the attestor; deferred because it needs the remote-execution work in Phase 3

Consequences

What it buys

• The most-exploited CI vulnerability class is removed by credential design rather than by validation
• A poisoned or corrupt entry costs a slow build, never a compromised one, because verification failure is a miss
• Trusted builds keep essentially the full benefit of the cache, which is where the majority of the job volume is

What it costs

• Fork builds are measurably slower, and the contributors affected are the ones the company least wants to frustrate
• A fork build that would have warmed the cache for its own next iteration cannot, so iteration on a fork PR is slow throughout
• Cache hit rate must be reported per trust class, or the aggregate number hides the fork experience

Choose differently when

If the cache held only content-addressed, independently verifiable artefacts whose provenance was attested — so that restoring an entry proved who produced it and from what — then write access would stop being a privilege and any run could write safely. That is the deferred option above, and it is the direction this decision should eventually move in.

Why it holds up over time

The asymmetry between reading shared state and writing it is permanent, and it gets more valuable as caching gets more aggressive. Remote execution and distributed action caches raise the stakes rather than lowering them: the more of a build comes from cache, the more a cache write is worth to an attacker.

What can a running job reach on the network, and who decides?

Context

A job needs the network: it fetches dependencies, pulls base images, clones submodules, sometimes calls a test double. Unrestricted egress makes all of that work and also makes exfiltration trivial — a build that has read something it should not can simply post it somewhere. Unrestricted egress is also how a build defeats its own recorded dependency set: a step that curls a script at build time makes the resolved-dependency record a fiction. The difficulty is that a strict allow-list breaks builds in ways that are tedious to diagnose, so any design that makes restriction opt-in results in nobody opting in.

Decision

Every sandbox's egress passes through a policy-enforcing proxy. Restriction is on by default: an untrusted run reaches only the dependency mirror and source control. A tenant may define an allow-list for its trusted runs. Every allowed and denied destination is logged and attributable to the run.

How it is realised on AWS

Isolation hosts place sandboxes on a network with no default route; the per-job agent's traffic is directed to a proxy that evaluates destination against the tenant's policy and the run's trust class. Package and image pulls are served from the platform's own mirror, which is both the performance path and the audit path. Denials return a distinct, named error so the failure reads as a policy decision rather than as a network fault.

Options weighed

• ChosenMediated egress, restrictive by default, tenant allow-list for trusted runs: The safe posture is the one you get without doing anything, and the mirror makes it fast rather than merely safe
• RejectedOpen egress with monitoring and alerting: Breaks nothing and prevents nothing; the alert arrives after the exfiltration
• RejectedOpt-in allow-list per pipeline: Correct in principle; in practice almost no pipeline opts in, so the control does not exist
• Right elsewhereNo egress at all, everything pre-provisioned into the sandbox: Right for a classified or air-gapped environment; the provisioning burden is not worth it when a mirror will do

Consequences

What it buys

• Exfiltration from a hostile build requires defeating the proxy, not merely running code
• The recorded dependency set becomes trustworthy, because fetching outside it is visible and usually blocked
• Upstream registry outages degrade build latency rather than stopping all builds, because the mirror is the default path

What it costs

• Builds that reached the internet incidentally now fail, and the first month of adoption is spent adding legitimate destinations
• The mirror becomes a critical dependency with its own availability and storage cost
• A determined build can still tunnel over an allowed destination; the control is a barrier, not a proof

Choose differently when

If the platform served only internal code with no untrusted runs and no supply-chain concern, the diagnostic cost of the allow-list would outweigh its benefit and monitored open egress would be defensible. As soon as one fork build runs, it is not.

Why it holds up over time

Mediating egress is independent of the network technology that implements it, and its value rises with every step the industry takes towards supply-chain attacks. The mirror that makes it palatable is also the thing that makes build reproducibility possible, so the investment pays twice.

Who makes the statement "this artefact was built from this commit by this pipeline", and why should anyone believe it?

Context

A provenance attestation is only worth the difficulty of forging it. If the signing key is available inside the build — as an environment variable, a mounted file, or a credential the job can exchange — then the statement is made by the thing being described, and a compromised build can produce an artefact together with a perfectly valid attestation saying it came from somewhere else. That is not a marginal weakness; it makes the entire evidence chain decorative, because the case the chain exists to detect is precisely a compromised build. The counter-argument is practical: the build knows most of the facts, so having it assemble and sign the statement is much simpler than reconstructing them outside.

Decision

The signing key is held by a control-plane attestor and is never present inside a sandbox. The job reports its outcome and uploads its artefact; the attestor composes the attestation from the inputs the platform itself observed — the resolved effective definition, the source commit, the recorded dependency set, the builder identity and the timings — and signs it. The build's own claims are inputs to be recorded, never assertions to be signed.

How it is realised on AWS

The signing key lives in a hardware-backed key store reachable only by the attestor's role; no path exists from an isolation host to it. The attestor is invoked by the sandbox manager after the job reports, binds the request to the job's lease and identity, composes the statement from control-plane records, signs, and appends to the transparency log. The artefact is addressed by the digest of its bytes, so the statement is about specific bits rather than about a name.

Options weighed

• ChosenControl-plane attestor signs from observed inputs: A compromised build can produce a bad artefact but not a credible claim about one
• RejectedThe build signs with a short-lived key issued to the job: Much simpler, and the statement is then made by the code under suspicion
• RejectedThe build composes the statement, the control plane signs it unexamined: Looks like the chosen option and is the rejected one: the content is still the build's
• DeferredA separate rebuilder independently reproduces and signs: The strongest available evidence and roughly doubles build cost; worth it for a small set of release artefacts, not for 1.9M jobs a day

Consequences

What it buys

• An attestation is worth verifying, which is the entire point of producing one
• Verification can be performed outside the platform against the transparency log, so the platform is not the only witness to its own claims
• The gate service has something objective to check before a deployment, rather than a self-report

What it costs

• The attestor must reconstruct facts the build knew directly, which means the control plane has to record more than it otherwise would
• The attestor becomes the highest-consequence code in the platform: a confused-deputy bug there yields valid, false statements
• Signing is on the critical path of every job completion, adding latency and a dependency to the seal step

Choose differently when

If artefacts were never deployed anywhere consequential — a research estate, a throwaway environment — the cost of a control-plane attestor would exceed its value and a build-signed statement would be adequate record-keeping. The decision is justified by the artefact reaching production under a gate that trusts the statement.

Why it holds up over time

"The observer signs, not the observed" predates software supply chains and will outlive every signing format. Sigstore, in-toto, whatever replaces them: each is a way of expressing this separation, and the separation is the part worth committing to.

What credential is inside a running job, how long is it useful, and what happens to it when the job stops?

Context

A long-lived credential inside a build environment is the most valuable thing an attacker can obtain from a CI platform, because it outlives the build, works from anywhere, and usually has more authority than the build needed. Static secrets in a pipeline's configuration are the classic form; a cloud role attached to the runner host is the modern one, and it is worse, because every job on that host inherits it whether it needed it or not. The alternative — minting an identity per job and exchanging it for narrowly scoped, short-lived credentials — puts identity federation on the hot path of every one of 1.9M daily jobs, which is a real availability and latency cost.

Decision

Every job is issued a workload identity binding tenant, repository, ref and trust class, valid for the job's lifetime. All secret and cloud access is scoped to that identity. Nothing long-lived is present in a sandbox. Revocation is pushed at job end rather than left to token expiry, and every secret read is recorded in the audit trail before the value is returned.

How it is realised on AWS

The scheduler asks identity federation to mint the job identity at dispatch; the token is delivered to the per-job agent and never to the job's own environment except as an explicitly requested, scoped exchange. The secret broker evaluates the token's trust class against policy, writes the audit record, then returns the value with redaction registered on the log path. Cloud access is obtained by exchanging the identity with the cloud's token service for a short-lived role. At job end the scheduler revokes the identity rather than waiting for the TTL.

Options weighed

• ChosenPer-job identity, scoped exchange, revoked at job end: A leaked credential's useful life is the job's life, and its authority is the job's authority
• RejectedHost-attached instance role shared by every job on the host: Simplest and requires no federation; every job inherits authority it did not ask for
• RejectedStatic secrets in pipeline configuration: Still the industry's most common design and the source of most credential leaks
• DeferredPer-step identity with per-step scope: Strictly better and multiplies federation traffic by the step count; revisit if step-level blast radius becomes the binding concern

Consequences

What it buys

• A credential exfiltrated from a build is worthless minutes later and was never broadly scoped
• Every secret read has an audit record naming the job, the repository, the ref and the trust class
• Denying an untrusted run is a policy evaluation on a token rather than a special case in the build path

What it costs

• Identity federation is on the hot path of every job and must be more available than the execution plane it serves
• Token minting and exchange add latency at job start, inside an already-tight cold-start budget
• Pipelines that expect a static secret must be migrated, and the migration is visible to every team

Choose differently when

If the platform's jobs needed no external authority at all — pure compile-and-test with no publish and no cloud access — the federation machinery would be unnecessary and a host role with no permissions would do. The moment a job can publish or deploy, it does not.

Why it holds up over time

Short-lived, narrowly scoped, workload-bound credentials are the direction every identity system has moved in for fifteen years and the direction the cloud providers keep extending. Building on federation rather than on stored secrets means the platform inherits those improvements instead of having to migrate to them.

If the platform itself were compromised or simply wrong, how would anybody find out that an attestation had been altered or invented?

Context

An attestation stored in a mutable database beside the run metadata is trustworthy exactly as far as the platform is trustworthy. That is a circular guarantee, and it fails in the two cases evidence is most needed: a compromise of the platform's own control plane, and an honest bug that mis-attributes one job's inputs to another's artefact. Both produce records that are internally consistent and factually wrong, and neither is detectable from inside. The cost of doing better is real: an append-only, tamper-evident log is more operationally awkward than a table, cannot be corrected in place, and forces the platform to keep records it might prefer to tidy.

Decision

Every attestation is appended to a tamper-evident, append-only log, retained seven years, verifiable by a third party without the platform's cooperation. Nothing in the log is ever edited or deleted. An artefact whose provenance fails verification is quarantined rather than removed, because a deleted artefact cannot be investigated.

How it is realised on AWS

Attestations and SBOMs are written to object storage under an immutability policy, with the log's inclusion proofs published so an external verifier can check an entry without asking the platform to vouch for it. The transparency log sits in the custody zone, reachable from the attestor and from nothing in the execution plane. Cross-region replication carries evidence at RPO 0, because losing proof is worse than losing work.

Options weighed

• ChosenAppend-only tamper-evident log, externally verifiable, 7 years: Makes an inconsistency discoverable without the platform's cooperation, which is the only case that matters
• RejectedAttestations as rows beside run metadata: Simple, queryable, and worthless in the two failure cases the evidence exists for
• Right elsewherePublic transparency log shared with the wider ecosystem: Right for open-source artefacts the world consumes; for internal artefacts it leaks the shape of the estate
• RejectedPeriodic signed snapshots of the attestation table: Cheaper, and detects tampering only at snapshot granularity, which an attacker chooses to work inside

Consequences

What it buys

• An auditor can answer "which commit produced this image" without the platform being a trusted intermediary
• A control-plane bug that mis-signs becomes discoverable after the fact rather than invisible
• Quarantine rather than deletion keeps the investigation possible, and the artefact still cannot deploy

What it costs

• Nothing can be corrected in place; an error is fixed by appending a correction, which is operationally unfamiliar
• Seven years of immutable evidence is a storage and legal-hold commitment that grows monotonically
• The immutability policy also protects records the company might later wish to remove, which is the point and is sometimes inconvenient

Choose differently when

If artefacts were short-lived and never subject to audit, retention of any kind would be waste and a table would suffice. The decision is driven by two audit reviews a year and by production images whose origin must be provable years after the engineer who built them has left.

Why it holds up over time

Tamper-evident append-only logs have outlived several generations of signing technology, and the reason is structural: the guarantee comes from the data structure rather than from the operator. Whatever replaces today's log formats will still be a log, and its entries will still need to be verifiable by someone who does not trust the writer.

When 40,000 slots are full and one tenant has just pushed a 900-job monorepo pipeline, whose job runs next?

Context

A global FIFO queue with per-tenant concurrency caps is the usual first design and it is comprehensible, cheap and wrong in one specific way: a cap bounds how much capacity a tenant holds but not how much of the queue it occupies. A monorepo pipeline that admits 900 jobs at once puts 900 entries ahead of the next tenant's five, so the small tenant's wait is set by the large tenant's burst regardless of the cap. Because CI latency is felt personally — an engineer is sitting there — the small tenant experiences this as the platform being broken, and the platform team experiences it as an email asking large teams to be considerate. Weighted fair queuing fixes it, at the cost of needing cross-tenant state in the dispatch decision.

Decision

Each tenant has a concurrency entitlement and a priority weight. Dispatch selects across tenants by weighted fair queuing rather than by arrival order, so queue wait inside an entitlement is bounded independently of any other tenant's burst. Idle entitlement is lent to tenants that are over theirs, and reclaimed when the owner returns. Queue wait inside entitlement is an SLO; a tenant over its entitlement has no wait guarantee.

How it is realised on AWS

The scheduler is sharded, with per-shard fair-queue state and a leader per shard, and entitlements held in the run metadata store. Priority tiers — interactive, scheduled, background — are evaluated before fairness, so an interactive job outranks a background job of any tenant, and background work is preemptible. Queue position and estimated wait are computed from the same state and exposed to the waiting engineer.

Options weighed

• ChosenPer-tenant entitlement with weighted fair queuing and lending: Bounds the small tenant's wait and still lets a big tenant use idle capacity
• RejectedGlobal FIFO with per-tenant concurrency caps: Simple and cheap; the small tenant's wait is set by the large tenant's burst
• RejectedHard per-tenant partitions, no lending: Perfectly fair and strands capacity inside unused entitlements, which at 610 tenants is most of the fleet most of the time
• DeferredA market with priced priority: Aligns incentives elegantly and requires a chargeback culture the organisation does not yet have; revisit alongside showback

Consequences

What it buys

• A two-person team's five-job pipeline has a bounded wait no matter what the monorepo is doing
• Idle entitlement is usable, so fairness does not cost utilisation
• Queue position and estimated wait become computable, which turns an unexplained wait into an explained one

What it costs

• The scheduler needs cross-tenant state in the dispatch decision, which constrains how far it can be sharded
• Entitlements become a thing platform engineers must curate, and a badly-set entitlement is now the cause of a wait
• Fairness violations need their own monitoring, because the failure is silent from any single tenant's point of view

Choose differently when

If tenant sizes were roughly uniform and bursts rare, FIFO with caps would deliver nearly the same outcome for a fraction of the complexity. The decision is justified by a distribution in which a handful of monorepos generate a large share of job volume — and that distribution is getting more extreme, not less.

Why it holds up over time

Weighted fair queuing is fifty-year-old theory from packet scheduling and has survived every change of substrate since. The pressure it addresses — the ratio between the largest and smallest workload sharing a resource — increases with monorepo size, generated code volume and AI-assisted development, so the decision gets stronger with time.

Can a sandbox be pre-warmed with a tenant's image and dependency layers, and if so, may it later be handed to a different tenant?

Context

Single-use sandboxes (ADR-01) make cold start the platform's hardest performance problem, and the obvious mitigation is to pre-warm with tenant content: pull the base image, hydrate the dependency layers, then hand the ready sandbox to that tenant's next job. The saving is large — most of the queued-to-first-step budget is image pull and dependency hydration. The problem is that a pre-warmed sandbox is now tenant-specific, so it can only serve that tenant; at 610 tenants with uneven arrival rates, most pre-warmed sandboxes sit idle waiting for a job that does not come, and the ones that are needed are in the wrong pool. Handing a tenant-warmed sandbox to another tenant is the alternative, and it reintroduces exactly the residue problem ADR-01 exists to remove.

Decision

The warm pool holds tenant-agnostic sandboxes that have booted a base image and have never seen tenant content. One is claimed at dispatch, the workspace and credentials are injected, and it is never returned to the pool. Tenant-specific warmth is recovered through the cache and the dependency mirror, not through a pre-warmed sandbox.

How it is realised on AWS

Each isolation host maintains a small pool of pre-booted microVMs from a common base image, sized from the observed arrival rate for that host's resource class. Dispatch claims one and marks it non-reusable. Tenant warmth comes from the cache restore path and from the mirror serving image and package layers from the same region, which is where the remaining cold-start budget is spent and optimised.

Options weighed

• ChosenTenant-agnostic pre-booted pool, claimed at dispatch, never returned: One pool serves every tenant, so the idle-waste ceiling is a single number to manage
• RejectedPer-tenant pre-warmed pools with tenant image layers: Best cold start; at 610 tenants most warm capacity is idle in the wrong pool
• RejectedTenant-warmed pools that may be reassigned between tenants: Has the cold-start benefit and reintroduces the residue problem ADR-01 removes
• DeferredPer-tenant pools for the largest tenants only, agnostic for the rest: Defensible hybrid worth revisiting once per-tenant arrival rates are measured rather than assumed

Consequences

What it buys

• Idle warm capacity is one fungible pool, so the 8% waste ceiling is a number that can actually be held
• No sandbox ever holds one tenant's content while waiting for another tenant's job, so ADR-01 survives intact
• The cache and mirror become the levers for cold start, and both benefit every tenant rather than the one they were warmed for

What it costs

• Cold start is longer than a tenant-warmed pool would give, and that difference is the most visible performance number the platform publishes
• Cache hit rate and mirror locality carry more weight than they otherwise would, which makes them availability-critical
• A very large tenant with unusual images gets no special treatment, and will ask for it

Choose differently when

If the estate were a handful of large tenants with steady arrival rates and huge images, per-tenant pools would be both affordable and materially faster, and the hybrid above becomes the right answer. The decision is driven by 610 tenants with uneven arrival.

Why it holds up over time

The trade — fungible capacity against specific warmth — is a permanent one in any pooled system, and the right side of it is decided by the number of tenants and the variance of their arrival. Both of those are measurable, so this decision is designed to be revisited with data rather than with argument.

Which jobs may run on capacity the cloud can take back, and what happens when it does?

Context

Interruptible capacity is substantially cheaper and CI is a near-ideal workload for it: jobs are short, stateless and retryable. The temptation is to put everything on it and re-dispatch on reclamation. That works arithmetically and fails experientially, because a reclamation adds the full cold start plus the elapsed work to a job someone is sitting and waiting for, and the distribution of that added latency is exactly the long tail the queue-wait SLO exists to bound. Meanwhile the nightly and backfill work, which nobody is waiting for, is the natural home for the risk.

Decision

Interruptible capacity serves the background tier only: scheduled runs, nightly work, backfills. Interactive and scheduled-but-blocking work runs on on-demand capacity. A reclaimed job is re-dispatched to on-demand capacity, and the drain notice is honoured where the provider gives one. The target is that at least 55% of eligible background job-minutes run on interruptible capacity.

How it is realised on AWS

Resource classes carry an eligibility flag; the scheduler places eligible background jobs on interruptible hosts and everything else on on-demand. The per-job agent handles the drain notice by reporting a distinct platform-failure cause and terminating cleanly, so the re-dispatch is attributable and does not pollute the infrastructure-failure ratio's interpretation. Background work is also preemptible by interactive work on shared hosts.

Options weighed

• ChosenInterruptible for background tier only, re-dispatch to on-demand: Takes most of the saving without putting the waiting engineer's tail latency at risk
• RejectedInterruptible for everything with transparent re-dispatch: Best unit cost and worst p99 for exactly the jobs someone is watching
• RejectedOn-demand only: Simplest capacity story and leaves a large, easily-harvested saving on the table
• Right elsewhereInterruptible with checkpoint and resume mid-job: Right for long ML training jobs; a 3-minute CI job is cheaper to restart than to checkpoint

Consequences

What it buys

• A meaningful share of compute cost is harvested from work nobody is waiting for
• The interactive queue-wait SLO is not exposed to reclamation tail latency
• Reclamation becomes an ordinary, attributable event rather than an incident

What it costs

• Nightly throughput is variable, so a capacity-scarce night lengthens the nightly window
• Two capacity pools to plan and monitor rather than one
• The eligibility flag is a decision someone has to make per resource class, and a wrong one is felt as latency

Choose differently when

If the interactive volume were small and latency-insensitive — a batch-oriented estate, or one where CI results are read the next morning — putting everything on interruptible capacity would be correct. Equally, if reclamation rates in the chosen families fell far enough, the distinction would stop earning its complexity.

Why it holds up over time

The rule generalises past any particular cloud's interruptible product: place cheap-but-unreliable capacity where the latency distribution does not matter. That mapping between workload patience and capacity reliability is permanent even as the products change names.

When the control plane restarts mid-run, what does it read to find out what was happening — and if two sources disagree, which one wins?

Context

Run state has two consumers with incompatible needs. Scheduling needs a strongly consistent, transactional view it can make decisions against right now: is this job leased, is this tenant at its entitlement, may this run proceed. History, recovery and audit need an ordered, immutable account of everything that happened, including the transitions a current-state row has already overwritten. Serving both from one store means either a relational table that loses history, or an event log that cannot answer "how many slots is this tenant holding" without a fold over everything. Serving them from two stores means they can disagree, and a design that does not say which one wins will discover the answer during an incident.

Decision

Run and job current state live in a transactional relational store, which is authoritative for every scheduling decision. Every transition is also appended to an ordered event log, which is authoritative for history, replay and audit. On disagreement, the relational store wins for scheduling and the event log wins for what happened. The log is retained independently, so losing the metadata store does not lose the account.

How it is realised on AWS

Aurora PostgreSQL holds runs, jobs, leases and entitlements with RPO 0 and RTO 15 min; transitions are appended to a Kinesis stream with RPO 0 and RTO 30 min, retained 400 days. The scheduler reads and writes the relational store transactionally and emits to the log as part of the same commit path, so a transition cannot be applied without being recorded. Recovery rebuilds the in-memory scheduler view from the relational store, then reconciles against the log to find transitions that were recorded but not applied.

Options weighed

• ChosenRelational authoritative for scheduling, event log for history, stated precedence: Each consumer reads the store that fits it, and the tie-break is written down before it is needed
• RejectedEvent log as the single source of truth, state folded on read: Elegant and immutable; folding for every dispatch decision at 3,200 jobs a minute is the wrong hot path
• RejectedRelational only, history in an audit table: Simpler; the audit table is mutable and shares the store's failure domain, so it fails when it is needed
• RejectedThe runner's own report as authoritative for its job: Removes a store and makes a partition indistinguishable from a success — see ADR-13

Consequences

What it buys

• Dispatch decisions are transactional and fast, and history is complete and immutable
• Recovery has a defined procedure rather than an argument, because precedence was decided in advance
• Losing the metadata store loses current state, not the account of what happened

What it costs

• Two stores to operate, replicate and reason about, with a reconciliation step in the recovery path
• Every transition costs a write to both, which is a throughput floor on the whole platform
• A bug that writes one and not the other produces a divergence that only the reconciliation finds

Choose differently when

At a fraction of this volume, folding the log on read would be entirely affordable and the second store unnecessary. Conversely, if scheduling state grew large enough that a single relational store became the bottleneck, the answer is sharding it rather than collapsing back to one store.

Why it holds up over time

The separation of current state from the account of how it got there is a distinction, not a technology, and it survives every substitution of database or stream. Naming the precedence rule is the part that pays off, because that is what an on-call engineer needs at 03:00 and cannot derive.

A job has stopped reporting. Is it dead, slow, or finished successfully with a lost response — and what does the platform do about it?

Context

This is the network-partition question in its most concrete form, and CI platforms get it wrong in a characteristic way: they treat the absence of a failure report as evidence of success, or they treat the absence of a heartbeat as evidence of death and re-dispatch a job that is in fact still running. Both are bad, but they are not equally bad. Re-dispatching a live job costs capacity and, for a job with side effects, may duplicate them. Marking an absent job successful puts an unverified green check on a pull request, which is the failure this platform exists to prevent. The asymmetry decides the design.

Decision

Every dispatched job holds a lease that it must renew. Loss is detected by lease expiry, never by silence on the result channel. An expired lease means the job is re-dispatched with the attempt count incremented and the cause recorded. A job is never reported successful on the basis of an absent runner. Automatic retry happens only for platform failure or a declared-retryable infrastructure error, never for user failure, and every retry records its cause.

How it is realised on AWS

The scheduler issues a lease with each job assignment and the per-job agent renews it; expiry re-queues the job. Partial logs already streamed are preserved and attached to the failed attempt, so the engineer sees what happened before the loss. Because the artefact is sealed by the attestor after the report, a job whose report is lost has no attestation and therefore cannot be treated as complete even if its bits were uploaded.

Options weighed

• ChosenLease expiry detects loss; never successful on absence; retry only platform failure: Resolves a partition to re-dispatch, which is the safe side of the asymmetry
• RejectedHeartbeat timeout with optimistic completion on late report: Reduces wasted work and occasionally produces a green check nobody earned
• RejectedTrust the runner's report, no lease: Simplest; a silent runner becomes indistinguishable from a passing build
• RejectedRetry every failure a fixed number of times: Hides real failures, wastes capacity on compile errors, and makes the infrastructure-failure ratio unmeasurable

Consequences

What it buys

• A partition costs duplicated work rather than a false green, which is the correct side to fail on
• Retry cause is recorded, which is what makes the infrastructure-failure ratio meaningful rather than decorative
• Partial logs survive a lost runner, so the engineer has something to read

What it costs

• A live-but-partitioned job is re-dispatched and its work duplicated, occasionally twice
• Jobs with external side effects need idempotency of their own; the platform cannot provide it for them
• Lease renewal is traffic on the hot path of every job, and its own availability matters

Choose differently when

If jobs were long, expensive and side-effecting — a multi-hour deploy rather than a three-minute test — the cost of duplicating one would exceed the cost of waiting longer, and a design with a longer lease plus explicit operator adjudication would be better. At a median job of 3m 10s, re-dispatch is the cheap option.

Why it holds up over time

"Absence of evidence is not evidence of success" is a distributed-systems invariant, and leases have been the mechanism for expressing it for decades. What changes is the timeout tuning; the asymmetry that decides the design does not.

What does a red check actually mean, and can the engineer tell from it whether the problem is theirs?

Context

Most CI systems report two outcomes: pass and fail. Everything that is not a passing test — an image pull timeout, a reclaimed instance, a registry outage, a cancelled run, a policy denial — arrives as fail, and the engineer's only recourse is to read the log and guess. The consequence is cultural rather than technical. An engineer who cannot distinguish a flake or an infrastructure blip from their own regression learns that the cheapest response to red is to press retry, and after that they retry genuine failures too. At that point the platform has trained its users out of reading its output, and no amount of dashboard investment recovers it.

Decision

A job reports exactly one of six outcomes: success, user failure, platform failure, cancelled, timed out, or blocked by policy. A platform failure is never attributed to the user's code. The infrastructure-failure ratio — platform failures as a share of all jobs — is a headline SLO at ≤ 0.5% weekly, alerting at 1.0% hourly. Flake rate, defined as differing outcomes on an identical commit and definition digest, is measured and reported per pipeline but deliberately not bounded: detection is the platform's obligation, fixing the test is the owning team's.

How it is realised on AWS

The per-job agent and the sandbox manager classify at the point of failure, where the cause is known, rather than post-hoc from a log. Retry policy keys off the class. The run console surfaces the class, the cause and a first-failure summary — the failing job, step and the log region around the first error. Flake detection compares outcomes across attempts on an identical commit and definition digest.

Options weighed

• ChosenSix outcomes, retry keyed off class, infra-failure ratio as a headline SLO: Makes trust in a red build measurable, and makes retry a decision rather than a reflex
• RejectedPass/fail with the cause in the log: Universal and cheap; it is how platforms teach engineers to stop reading
• RejectedPass/fail/error, three outcomes: Better, and still conflates cancellation, timeout and policy denial with infrastructure error
• DeferredAutomatic flake quarantine after N differing outcomes: Attractive and hazardous: quarantining a test is how a real regression ships. Report it, do not disable it

Consequences

What it buys

• An engineer can tell from the outcome alone whether the problem is theirs, which is what keeps a red build worth reading
• The infrastructure-failure ratio becomes measurable, and it is the number that decides whether the platform is trusted
• Retry stops being a cultural reflex and becomes a classified, recorded action

What it costs

• Every failure path in agent and manager must classify correctly, which is real work spread across the whole codebase
• Six outcomes are more than a badge can express, so the surfaces that show them need design attention
• A misclassification is worse than no classification, because it is confidently wrong

Choose differently when

If the platform served a single team that reads every log anyway, the taxonomy would be documentation rather than mechanism. It earns its cost at 4,200 engineers who will never read the platform's documentation and will absolutely learn a reflex.

Why it holds up over time

The distinction between "your change is wrong" and "our platform failed you" is about accountability, not technology, and it will matter identically on any future substrate. Designing the outcome taxonomy so the ratio is measurable is a permanent asset that survives every migration.

Does the artefact that reaches production come from the run that passed staging, or from a fresh build of the same commit?

Context

Rebuilding per environment is intuitive and common: the same commit, built again with the environment's settings, gives an artefact tailored to its target. It also breaks the only chain of evidence linking what was tested to what is running, because the production artefact has never been tested — an artefact built from the same source is not the same artefact unless the build is bit-for-bit reproducible, and almost no real build is. Timestamps, dependency resolution drift, base-image tag movement and toolchain patch levels all vary between two builds minutes apart. The counter-pressure is that build-time configuration is genuinely convenient, and teams will have baked environment assumptions into their images for years.

Decision

One artefact digest is produced once and promoted, unchanged, through every environment. Environment-specific configuration is bound at deployment time, not at build time. Promotion moves a pointer; the bits never change. Rollback repoints to a retained previously-deployed digest and is an ordinary operation with a latency target rather than a recovery procedure.

How it is realised on AWS

Artefacts are content-addressed, and an environment holds a pointer to one digest plus its configuration and gates. The promotion path is ordered per pipeline, and skipping a stage requires a recorded break-glass override with actor, reason and expiry. A deployment lock serialises promotions to one environment. Retention keeps prior production digests so rollback has somewhere to point.

Options weighed

• ChosenBuild once, promote by digest, configuration at deploy time: "Did we ship what we tested" gets a cryptographic answer, and rollback becomes a pointer move
• RejectedRebuild per environment from the same commit: Convenient for build-time configuration, and the production artefact has never been tested
• RejectedBuild once, then re-tag and re-sign per environment: Keeps the bits and adds a mutable signing step that weakens the provenance for no gain
• DeferredReproducible builds so a rebuild is provably equivalent: Would make rebuilding safe and is a multi-year effort across every language in the estate; a good ambition, not a plan

Consequences

What it buys

• The production artefact is byte-identical to the tested one, and the attestation covers exactly those bits
• Rollback is a five-minute pointer move rather than a twenty-minute rebuild under pressure
• Audit becomes a query: one digest, one attestation, one chain of gate evaluations

What it costs

• Build-time configuration must be migrated to deployment-time binding, which is visible work for every team
• Prior digests must be retained per environment so rollback has a target, which is storage the platform pays for
• One artefact must be able to run in every environment, which constrains how images may be built

Choose differently when

If builds were genuinely bit-for-bit reproducible across the whole estate, rebuilding per environment would be provably equivalent and the constraint on build-time configuration could be relaxed. That is the deferred option, and it is the only condition that changes this decision.

Why it holds up over time

Content addressing plus a mutable pointer is the pattern every artefact ecosystem has converged on, from container registries to package managers to source control itself. It outlives any particular registry because the identity comes from the bytes rather than from the store.

Can the author of a pipeline weaken the checks that decide whether their own change reaches production?

Context

Gate jobs inside the pipeline are the natural design: the checks live beside the code they check, they are versioned with it, and a team can evolve them. They are also self-certification. The pipeline definition is authored by the same people whose change is being gated, so a gate expressed as a pipeline job can be reordered, made conditional, marked continue-on-error, or removed — usually with an entirely sincere reason, under deadline pressure, in a diff nobody reviews closely because it is 'just CI config'. The second half of the question is what happens when the external decision service is unavailable: failing open keeps delivery moving and silently removes every control, and failing closed stops delivery during an outage of something that is not the application.

Decision

Gates are evaluated by a decision service outside the pipeline, consulted by the deployer. A pipeline author cannot weaken the gates applying to their deployment. The service returns allow, deny, or allow-with-override-required, always with a machine-readable reason naming the failing condition. Production fails closed when the service is unavailable; a tenant may configure fail-open for non-production only. Overrides are recorded with actor, reason and expiry, and override usage is a watched signal.

How it is realised on AWS

Policy bundles are versioned per repository and held in the control plane, cached per tenant for a p99 decision under 500 ms. The deployer consults the service before moving an environment pointer; the verdict and its reason are written to the immutable audit trail before the pointer moves. Gate types include required job outcomes, code-owner and human approval, provenance and signature verification, vulnerability-severity thresholds, change-freeze windows, target SLO health and a required soak. Separation of duties is enforced where tenant policy demands it.

Options weighed

• ChosenExternal decision service, fail closed in production, recorded overrides: The only design in which a gate is not under the control of the change it gates
• RejectedGate jobs inside the pipeline: Simplest and most flexible, and is self-certification wearing an automation costume
• Right elsewhereGates in the target platform's admission layer: Excellent for runtime invariants and blind to build-time evidence; complementary rather than alternative
• RejectedExternal service with fail-open everywhere: Keeps delivery moving during an outage by removing every control at exactly the moment nobody is watching

Consequences

What it buys

• A gate means the same thing on every pipeline, and changing it is a policy change with an owner
• Every verdict, approval and override has an audit record attributable to an identity
• Provenance verification becomes enforceable rather than advisory, because the enforcer is not the build

What it costs

• Every promotion depends on a service that must be more available than the deployments it governs
• Teams lose the ability to express a local, sensible gate as a pipeline job and must go through policy
• Break-glass override is necessary and is the control most likely to be abused

Choose differently when

If every team were both the author and the accountable owner of its production environment, with no separation-of-duties requirement and no shared blast radius, in-pipeline gates would be adequate and much simpler. The decision is driven by nine business units sharing a platform and by audit obligations that require the gate to be outside the gated.

Why it holds up over time

Separating the decision from the thing being decided is a governance invariant, and the specific gate types will churn constantly — vulnerability thresholds, freeze windows, SLO health, whatever comes next — without disturbing the boundary. That is exactly why the boundary rather than the gate list is what this record commits to.

How long is a build artefact, a log and an attestation kept — and is the answer the same for a pull-request build as for a release?

Context

A global retention policy is easy to state and always wrong in one of two directions. Set it long enough for audit and the platform pays to keep 480 TB of pull-request artefacts and 6 TB a day of logs from builds nobody will ever look at again. Set it short enough to be affordable and the evidence an auditor needs is gone. The volumes make this the platform's fastest-growing cost line and the one most easily wasted, because nobody notices over-retention until the bill arrives, and by then the data cannot be un-kept cheaply.

Decision

Retention is a property of what produced the data. Release artefacts are kept 400 days, main-branch artefacts 90 days, pull-request artefacts 14 days. Logs are searchable 14 days, retrievable 90 days, and archived 400 days for release pipelines only. Caches are evicted after 7 days idle. Provenance attestations, the transparency log and the audit trail are kept 7 years and are immutable. Run metadata is kept 400 days. Eviction is auditable, and nothing referenced by a live environment pointer or a retained attestation is ever evicted.

How it is realised on AWS

Retention class is written on the artefact at seal time from the run's trigger and ref, so eviction never has to re-derive intent later. Object lifecycle policies implement the tiers; the immutable classes sit under an object-lock policy in the custody zone. Cost per job-minute and artefact TB by class are reported per tenant so the expensive habits are visible to the teams that have them.

Options weighed

• ChosenRetention by provenance and pipeline class, immutable evidence separate: Spends storage where it is asked for and reclaims it where it never will be
• RejectedOne global retention period for everything: Either unaffordable or non-compliant, and usually both at different layers
• RejectedPer-tenant retention chosen by each team: Every team chooses the maximum, because nobody pays for storage they do not see
• DeferredUsage-driven retention — keep what has been accessed recently: Efficient for artefacts and dangerous for evidence, which is by definition accessed rarely and needed absolutely

Consequences

What it buys

• The largest volumes are the shortest-lived, so the cost line tracks activity rather than accumulating forever
• Audit obligations are met by a small, immutable, clearly-bounded set of data
• Retention class is decided at seal time from facts, so eviction never needs to guess at intent

What it costs

• More classes to reason about, and a wrongly-classed artefact is either wasteful or gone too soon
• Pull-request artefacts vanish in 14 days, which occasionally frustrates a long-running investigation
• The 7-year immutable set cannot be trimmed, so a classification error there is permanent

Choose differently when

If artefact and log volumes were small enough that storage was not a material cost, a single long global retention would be simpler and harmless. At 480 TB growing 40% a year and 6 TB of logs a day, it is the fastest way to waste money on data nobody wants.

Why it holds up over time

Tying retention to the provenance of the data rather than to a global default is a policy shape, not a storage feature, and it transfers to any storage technology. As volumes grow, the argument for it strengthens; it never weakens.