CI/CD Platform  ·  View 22 of 22  ·  Assurance

Failure Classes and Responses

Nine classes, each with how it is detected and what the engineer is told.

Editable source SVG draw.io All views
What fails How it is detected Response What the user sees Transient infrastructure Sandbox launch, image pull Agent never reports ready Retry on fresh capacity Nothing, below threshold Capacity reclamation Interruptible host taken Drain notice, lease loss Re-dispatch on-demand Longer run, same outcome Runner loss Host death, partition Lease expiry, not silence Re-dispatch, keep part log Attempt 2, cause named Poison job Escape, fork bomb, mining Limit breach, egress denial Kill, destroy, flag tenant Blocked by policy Dependency outage Registry or VCS down Mirror miss rate spike Serve from mirror Dependency unavailable Cache poisoning Hostile or bad entry Integrity check on restore Treat as miss, evict Slower build only Gate or policy failure Decision service down Health probe, timeout Fail closed in prod Promotion blocked Region loss Control plane region Multi-AZ probes all red Second region, new runs 30 min degraded Permanent user failure Compile error, failing test Non-zero exit, no retry Report, never retry First-failure summary CI/CD Platform — Failure Classes and Responses Every row distinguishes platform failure from user failure, because conflating them is what teaches engineers to retry instead of read. v 1.0 · owner Platform Engineering · date 2026-09

The rule every row obeys

  • Platform failure and user failure are different outcomes. Every row's fourth column says what the engineer sees, because conflating the two is what teaches people to retry instead of read.
  • "Flake" is never a root cause in this table. Runner loss is detected by lease expiry, dependency outage by mirror miss rate, cache poisoning by integrity check — each has a named detection signal.
  • Permanent user failure is never retried. A retried compile error wastes capacity and hides the first-failure summary.

Fail-closed choices

  • Gate service unavailable fails closed for production and is configurable only below it.
  • Store failure pauses scheduling rather than making decisions on stale state; a run never completes without its artefacts durable.

Risks

  • Nine classes is the set that is designed for. The class not in the table — a control-plane logic bug that mis-signs or mis-attributes — has no automatic response and relies on the transparency log being independently verifiable.
◀  Identity — Getting a Credential and Losing It All views
Solution Architecture v1.0 · Amazon Web Services · Platform Architecture · 2026-0922-failure-classes