# CI/CD Platform

**Solution Architecture v1.0 · Amazon Web Services · Platform Architecture · 2026-09 · 22 views · 17 architecture decision records**

An engineer pushes a commit, switches to Slack, and ninety seconds later a green check appears next to their pull request. That check is the most-used piece of software in their working day and the one they think about least. Behind it, a stranger's code — because a pull request from a fork is a stranger's code — has just been compiled, tested and thrown away on the company's own machines, and a signed statement now exists saying which commit produced which container image. This package is the platform behind that check — for an assumed product company of 4,200 engineers in 610 teams across 38,000 repositories, running 210,000 pipeline runs and 1.9M jobs a day, peaking at 3,200 jobs a minute with 40,000 concurrent slots, 480 TB of retained artefacts and 6 TB a day of raw logs — built on nested-virtualisation-capable EC2 instances running a microVM hypervisor, SQS and Kinesis for run state, Aurora PostgreSQL for run metadata, S3 for content-addressed artefacts and an Object-Lock transparency log, a hardware-backed KMS key for the attestor, and OIDC workload identity federation for job-scoped credentials.

The product is not the ability to run a build quickly. It is **the ability to run code nobody has reviewed without ever letting it touch anything the platform needs to trust** — and then to prove, for any image in production, which commit produced it.

The design rests on one rule: **tenant code executes only inside single-use, hardware-isolated sandboxes that hold no long-lived credential and cannot write shared state; every trusted act — brokering a secret, signing provenance, deciding a gate, recording an outcome — happens in a control plane that never executes tenant code.**

The decisions that carry the design:

- **Every sandbox is single-use, and that is a security decision rather than a hygiene one.** A sandbox is created for one job and destroyed after it, whatever the outcome. Reuse is the cheapest performance win available to a CI platform and the costliest security decision available, because the guarantee required is not "we cleaned the workspace" but "nothing the previous job wrote can influence this one" — and that cannot be audited. The platform buys cold-start engineering and a warm pool instead (ADR-01).
- **Isolation is hardware-level, and it does not vary by trust class.** Jobs run in microVMs, not hardened containers, because only virtualisation answers "a build rooted the kernel". A trusted branch build is contained exactly as strictly as a fork build, because a compromised dependency inside a trusted build is indistinguishable from a hostile fork — and that is now the more likely attack (ADR-02).
- **Trust is a property of the run, assigned at admission from the event.** The classification is computed once, from the trigger and membership records, before any capacity is committed, and written as an immutable column. Nothing the pipeline declares can raise it. A trust class that can be escalated mid-run is not a boundary (ADR-03).
- **The cache is read wide and written narrow.** Any run may read it; only a trusted run may write it, and a failed integrity check on restore is a miss rather than an error. A cache writable by an untrusted run is a code-execution path into every trusted build that reads it afterwards, and the measurably slower fork build is the accepted price (ADR-04).
- **Restrictive egress is the default, because a control that must be turned on will not be.** All sandbox traffic passes a policy proxy; an untrusted run reaches only the mirror and source control. The mirror is what makes the restriction fast enough to survive contact with a deadline (ADR-05).
- **Provenance asserts what the platform observed, never what the build claimed.** The signing key lives in the control plane and is never present in a sandbox. The job reports; the attestor signs from the inputs the platform recorded. A compromised build can produce a bad artefact but not a credible claim about one (ADR-06).
- **Credentials are minted per job and revoked when the job ends.** Nothing long-lived is inside a sandbox, so a leaked credential's useful life is the job's life and its authority the job's authority. Identity federation therefore sits on the hot path of every job rather than in a configuration file (ADR-07).
- **Evidence goes to an append-only log that can be verified without the platform.** An attestation held only in the platform's own mutable store is trustworthy exactly as far as the platform is, which is a circular guarantee that fails in the two cases evidence is for. Quarantine, never delete: a deleted artefact cannot be investigated (ADR-08).
- **Fairness is enforced in the scheduler, not asked for politely.** A concurrency cap bounds what a tenant holds, not what it blocks, so a 900-job monorepo pipeline still puts 900 entries ahead of a five-job pipeline. Weighted fair queuing with entitlement lending is what makes the small tenant's wait bounded (ADR-09).
- **The warm pool is tenant-agnostic, and a claimed sandbox never returns to it.** Pre-warming with a tenant's own layers is faster and, at 610 tenants with uneven arrival, leaves most warm capacity idle in the wrong pool — and handing a tenant-warmed sandbox to another tenant reintroduces exactly the residue the single-use rule removes (ADR-10).
- **Cheap capacity is paid for in tail latency, so it goes where nobody is waiting.** Interruptible instances serve the background tier only; a reclaimed job re-dispatches to on-demand capacity (ADR-11).
- **The relational store is authoritative for scheduling; the event log is authoritative for history.** Two stores, because dispatch needs a transactional current view and audit needs an ordered immutable account — and the precedence rule is written down before the incident that needs it (ADR-12).
- **The runner's report is evidence; the lease is truth.** Loss is detected by lease expiry rather than by silence, and a job is never reported successful on the basis of an absent runner. A partition therefore resolves to re-dispatch, never to a false green (ADR-13).
- **Six job outcomes, and a platform failure is never reported as the user's.** An engineer who cannot tell a flake from their own regression learns that the cheapest response to red is retry — and after that they retry real failures too. The infrastructure-failure ratio is the number that decides whether a red build is worth reading (ADR-14).
- **Build once, promote the digest, bind configuration at deployment.** One artefact crosses every environment, so "did we ship what we tested" has a cryptographic answer and rollback is a five-minute pointer move rather than a twenty-minute rebuild (ADR-15).
- **Gates are decided outside the pipeline they govern, and fail closed in production.** A gate expressed as a pipeline job is self-certification: the author can reorder it, make it conditional, or mark it continue-on-error, usually with a sincere reason under deadline pressure (ADR-16).
- **Retention is set by what produced the data, not by one global number.** Release artefacts 400 days, pull-request artefacts 14, evidence 7 years and immutable. The class is written at seal time, while the platform still knows why (ADR-17).

The architecture one-pager (including why the design should still hold up in ten years, and the five risks that would change it) and the full decision record appear on the landing page of the diagram set, directly below the index of views. The same content is published as [docs/architecture-one-pager.md](docs/architecture-one-pager.md) (~14 min) and [docs/decision-record.md](docs/decision-record.md) (~63 min).

---

## What is here

| Path | Contents |
|---|---|
| `diagrams/index.html` | The landing page: 22 views in seven acts with every format linked, then the **architecture one-pager** and the **decision record** |
| `diagrams/*.html` | One self-contained page per view: the inlined diagram plus the reasoning cards, with copy / PNG / PDF export |
| `diagrams/svg/*.svg` | The same views as SVG with the diagram XML embedded; they re-open fully editable in diagrams.net |
| `diagrams/drawio/*.drawio` | draw.io source |
| `docs/architecture-one-pager.md` | The one-pager as markdown |
| `docs/decision-record.md` | The 17 decision records, the capability-to-technology table and the package glossary as markdown |
| `specs/part-a..c.json` | Diagram specifications, the source of truth for every view |
| `specs/manifest-a..b.json` | Acts, page titles, subtitles and reasoning cards |
| `specs/adr-onepager.json`, `specs/adr-records-a..b.json` | The one-pager, the decision records, the capability-to-technology table and the glossary |
| `scripts/build.sh` | Rebuilds every deliverable from the specs (Node only, no network) |
| `ask.md` | The requirement |

## The twenty-two views

| # | View | What it answers |
|---|---|---|
| 01 | System Context | Who triggers a build, the systems the platform reads and never writes to, and why the outside contributor is in the context rather than an exception |
| 02 | High-Level Architecture | Six stages from a push to a promoted digest, each one a place a run can legitimately stop, with attestation as a stage rather than an afterthought |
| 03 | Actors and Journeys | Nine actors, three of them machines, and the goals they would state in their own words — including the auditor and the scheduled trigger |
| 04 | Journey — Waiting for a Green Check | The loop run six times a day; the trough is attribution, not speed |
| 05 | Journey — A Pull Request From a Fork | The same substrate, a different credential posture; the trough is silence, not slowness |
| 06 | Journey — Promote a Release, Then Take It Back | Promoting a digest is cheap; deciding whether the new version is the cause is not |
| 07 | Layered Architecture | Seven layers, with admission above the control plane because classification is a one-time act nothing below may revise |
| 08 | Containers and Components | One control plane, one execution plane, three kinds of store — because there are three mutabilities to hold |
| 09 | Integration Surface | Three authenticated inbound surfaces, three outbound, and no public build endpoint |
| 10 | Run and Evidence Data Flow | From commit to audit line, with the sealing step that makes everything after it verifiable |
| 11 | Storage Zones | Four zones ordered by what happens if the data is lost; the largest and hottest stores are the disposable ones |
| 12 | Core Data Model | Ten entities, and the one that carries the architecture is a single immutable column on the run |
| 13 | Critical Flow — Commit to Signed Artefact | Sixteen messages; the job reports and the attestor signs, and the red one is lease expiry |
| 14 | The Build Pipeline Inside One Run | Six stages, signing among them, and cache save as the only stage a fork run skips |
| 15 | Execution by Trust Class | One substrate, four postures; the difference is what the sandbox is given, never how well it is contained |
| 16 | Deployment Architecture | Three AZs of capacity, one region of evidence, and a recovery region that holds proof rather than running jobs |
| 17 | Promotion and Environments | One digest across every environment, configuration bound at deployment, rollback as a pointer move |
| 18 | Observability Coverage | Six signal families across five stages, with the empty cells as statements rather than gaps |
| 19 | The Run Lifecycle Loop | Seven steps, and the loop closes through observation changing the next definition rather than through a dashboard |
| 20 | Trust Zones and Crossings | Four zones; the only upward path out of the untrusted zone is a report the control plane validates |
| 21 | Identity — Getting a Credential and Losing It | Twelve messages, and the two that matter are the audit write and the refusal |
| 22 | Failure Classes and Responses | Nine classes, each with its detection signal and what the engineer is told |

## Rebuilding

```bash
bash scripts/build.sh
```

Node 20+ and nothing else. No draw.io Desktop, no browser, no network. The build generates the
draw.io sources from `specs/part-*.json`, fails on any geometry or routing defect, renders the
editable and plain SVGs, writes the HTML pages and the index, injects the one-pager and the
decision record, and proves every relative link in `diagrams/` resolves.

At v1.0 the geometry gate reports 0 errors and 0 warnings across 22 files, the routing gate
0 errors and 0 warnings across 149 edges, all 353 nodes resolve to an icon with no weak matches,
and the link check passes on 174 relative links.