A narrated 3D walkthrough of the Spot Capacity Orchestrator design: a contract for every workload, a node that decides on its own when the whistle blows, pools that are reclaimed together, work that survives, and a saving that is counted net.
The brief it was built from
Walk an engineer or architect through the Spot Capacity Orchestrator use case: how it works, why it is built this way and what would break it, at the level of the diagram views and decision records, with the stack, the patterns and a hypothetical product interface on screen.
What is real, and what is a scenario
Real data
- The requirement: fleet size, interruption behaviour, timing budgets, availability, savings and waste budgets, retention, recovery, the thirteen failure classes, the seven principles and the critical decision (ask.md)
- The two planes, the layers, the node agent, the integration surface and the five provider commitments (views 02, 07, 08, 09, 10, 11, README)
- The interruption contract, the five tiers and the feasibility rules, with the 115 second drain limit and the requeue unit rule of rate times length divided by two (view 12, decision records 01 to 04)
- The hard notice worked example: 1.4 seconds to drain signal, a 1.4 gigabyte save in 31 seconds against 45, the 800 millisecond scheduler cap (view 18, decision records 14, 15, 17)
- Pools, correlation groups and caps of 15 and 30 percent with at least eight pools, the five minute scorer and the sixty second exclusion watcher (views 16, 17, 24, decision records 06, 07, 08)
- The four data classes, tags as the reconstruction key, the bucket per workload, the manifest written last, and the 0.14 percent an hour unprotected loss rate with the four hour Young and Daly interval capped at 60 minutes (views 13, 14, 20, decision records 11, 18 to 21)
- The 800 node hall-wide reclaim: 320 writes of 1.4 gigabytes, about 10 gigabytes a second against 20 (view 21); fail static, 5 percent recovery steps and the 10 calls a second budget (view 23, decision record 12)
- The tier-one floor price of about 4,300 euros a month per ten points, fallback caps and escalation, and the net savings ledger of 335 thousand euros counterfactual, 136 thousand spend and 199 thousand net, 59 percent (views 06, 22, 28, decision records 22 to 26)
- The release canary gates, the daily one percent drill, the eight unproven behaviours and the proof phase (views 26, 27, 29, 33 as ADR-33, README, one-pager)
- The Amazon EC2 Spot notice, the Amazon static stability article and the Kubernetes node shutdown page, each fetched from its own publisher in this run
Scenario & assumptions
- The night market, its people, the screens and every meter are illustrations; none of them is a measurement
- The hypothetical product interface (steward handheld, service levels, contract editor, notice timeline, road comparison, placement view, checkpoint view, failure views, savings ledger, health board) is illustrative, not the real product, and its layout and wording are invented to show the idea
- The central road's numbers are this walkthrough's assumptions: about 20 notice answers a second when the office is healthy and a tenth of that when the same event is hurting it. The decision record gives the argument, not a benchmark
- The cloud interface pressure of the node-decides road, the throughput curve during an outage and the work-saved figures for the no-notice and store failures are illustrative scores, not computed from the use case
- The 40-stall workload and its two placements (capped and cheapest first) are an illustration of the caps; the caps themselves are the design's
- The checkpoint rate of about 45 megabytes a second per writer is derived from the worked example in view 18 (1.4 gigabytes in 31 seconds) and is used only to test whether a save fits a drain budget
- The walkthrough shows one region and one hall wave; the design runs three regions, and the 800 node wave and the 12 hour outage are the use case's own scenarios
- The use case is a design, not a report on a running system: every rate, latency and cost is a stated assumption or planning estimate until its proof phase replaces it
- The use case is this practice's design; the three examples show the same ideas elsewhere and are not descriptions of how those organisations build their systems
What you leave with
- On reclaimable capacity the notice is a bonus, never a dependency: the node acts from a contract it already holds
- Diversify across what is reclaimed together, not across names, and fill a bought floor first
- A saving is real only net of redone work, fallback, headroom, storage and the platform's own cost
- Ask of your own design: which emergency response makes a round trip to a service that might be the thing failing?
Sources
- Amazon EC2 User Guide: Spot Instance interruption notices Fetched in this run. Used for: the notice is a warning issued two minutes before EC2 stops or terminates a Spot Instance; it is available as an EventBridge event and in instance metadata; it is emitted on a best effort basis; AWS recommends checking for it every 5 seconds.
- Amazon Builders' Library: Static stability using Availability Zones (Mike Furr, 2019) Fetched in this run (the page redirects to builder.aws.com). Used for: a statically stable design keeps working when a dependency becomes impaired; a control plane makes changes to a system and a data plane is the daily business of those resources; a control plane tends to have more moving parts than a data plane and so is statistically more likely to become impaired; traffic that worked before the event continues to work.
- Kubernetes documentation: Node shutdown Fetched in this run. Used for: the kubelet detects node shutdown and terminates pods; graceful shutdown depends on systemd inhibitor locks to delay shutdown; shutdownGracePeriod and shutdownGracePeriodCriticalPods are both zero by default, which leaves the feature off until both are set above zero.
Narration voice: Kokoro-82M v1.0, voice af_heart (kokoro-onnx 0.6.1). Apache-2.0 (Kokoro-82M weights).
Security & identity
Original
Reliability & operations
Original
Product systems
Original