Reliability & operations
Original
Checkout time
How a secrets platform hands a workload a credential that expires on its own, and what happens when it breaks: the whole design, its decisions and its traps, told as a hotel.
Drag to look around · arrow keys step through · T for the text view
A narrated 3D walkthrough of the Secrets Distribution Service design: a workload that proves what it is without holding a secret, a credential that expires at checkout, why the chain of trust ends at attestation, and what each failure does.
Walk an engineer or architect through the Secrets Distribution Service use case: how it works, why it is built this way and what would break it, at the level of the diagram views and decision records, with the stack, the patterns and a hypothetical product interface on screen.
What is real, and what is a scenario
- The requirement: availability, latency, throughput, lifetime, recovery and retention targets, the failure classes, the seven principles and the critical decision (ask.md)
- The six stages, the four stores, the chain of trust, the first-issuance trace, the lease lifecycle, revocation by scope, the restart storm, the site outage and the restore (views 02, 10, 13, 17, 19, 21, 22, 23, 24)
- The 34 decision records with the alternatives that lost, especially ADR-01, 04, 09, 10, 12, 14, 15, 21, 27, 31, 33 and 34 (docs/decision-record.md)
- The 3 millisecond cached read, 120 and 500 millisecond mint, 400 millisecond attestation, 30 second revocation, 180,000 leases per site, 15 minute session, 30 minute lease, 12 hour total, 60 minute continuity (ask.md 14, ADR-14, ADR-15, ADR-21)
- The restart storm arithmetic: 20,000 pods per site, one attestation, nine mints and a custodial read each, about 200 role creations a second per database primary (view 22)
- The reach preview of 212 pods for a role and 40 for an identity (view 21)
- The renewal rate: outstanding leases divided by two thirds of the lifetime, 150 a second at 30 minutes (ADR-14, README)
- The GitHub Actions OpenID Connect and CircleCI statements, taken from search-result summaries of GitHub's documentation and CircleCI's incident report in this run (the pages themselves could not be fetched)
- The hotel metaphor, the people, the screens and every meter are illustrations; none of them is a measurement
- The hypothetical product interface (developer terminal, service levels, request trace, incident console, health dashboard) is illustrative, not the real product, and its timings, hit rate and counts are invented to show the idea
- The per-step timings in the first-issuance trace add up to the 400 and 500 millisecond budgets but are not the use case's own split
- The meters (leak window, workloads running, guarantees true) are illustrative scores for each failure, not computed
- The use case's throughput and lease figures do not reconcile; this piece uses the lease arithmetic for renewals and the stated rates only where it says stated
- The walkthrough shows one site and one environment; the design runs two sites and three environments
- The 'three roads' framing is the walkthrough's own: road two and three are the alternatives the decision record rejects (ADR-01) and the second failure in ask.md 19, played forward
- Planning estimates such as about 30 machines per production site are the use case's own and are replaced by measurements in its proof phase
- The use case is this practice's design; the examples show the same idea elsewhere and are not descriptions of how those companies work
What you leave with
- A secrets platform is two promises: prove identity without a stored secret, and issue something that expires
- Short leases contain dead holders; revocation and detection contain live ones
- The honest cost of fail static is a site whose credentials lapse after 60 minutes, and the design states it
- Ask where your own chain of trust ends and what each credential does when the control plane is gone
Sources
- GitHub Docs: OpenID Connect for GitHub Actions Cited from a web-search result summary in this run; the page itself could not be fetched (proxy block). Used only for: OIDC lets a workflow exchange short-lived tokens with the cloud provider so cloud credentials need not be stored as GitHub secrets, and the token is valid for a single job and then expires.
- CircleCI incident report for January 4, 2023 security incident Cited from a web-search result summary in this run; the page itself could not be fetched (proxy block). Used only for: malware on an engineer's laptop led to theft of a valid, 2FA-backed SSO session, and customers were told to rotate secrets stored on the platform (later widened to all secrets, keys and tokens stored there).
More walkthroughs
Reliability & operations
Original
Product systems
Original
LinkedIn, behind the front page
A feed in 300 milliseconds for 1.2 billion members: the whole design, its decisions and its traps, told in a newspaper printing house.
Life with a product
From request #1