Request one Portfolio ↗
Platform & infrastructure 12 chapters about 13 min 22 Oct 2026 An original idea

The timetable on the wall

How a service mesh puts a proxy on every call across two thousand services, and keeps working when its own control plane is gone: the whole design, its decisions and its traps, told as a model railway.

Start full screen ↗ Play it here ↓

Drag to look around · arrow keys step through · T for the text view

A narrated 3D walkthrough of the Service Mesh Platform design: a gatehouse on every platform, a planning office that prints bulletins and is never asked mid-call, an attested day pass, a retry that is paid for from a budget, and a release that undoes itself.

The brief it was built from

Walk an engineer or architect through the Service Mesh Platform use case: how it works, why it is built this way and what would break it, at the level of the diagram views and decision records, with the stack, the patterns and a hypothetical product interface on screen.

What is real, and what is a scenario

Real data
  • The requirement: purpose and scope, the 15 section capabilities, the quantified targets (400,000 calls a second, 99.99 percent mesh-attributable availability, 3 milliseconds per call, 12 hours of control-plane outage, 24 hour passes renewed at 12, 8 and 10 percent resource budgets), the failure classes, the seven principles and the critical decision (ask.md)
  • The architecture one-pager and the 32 decision records, including records 1, 3, 4, 7, 8, 9, 11, 15, 16, 17, 18 to 20, 23, 24, 26, 30, 31 and 32, with the options that lost and what each choice costs (docs/architecture-one-pager.md, docs/decision-record.md)
  • The five stages, the layers, the cluster containers and the trust hierarchy with four lifetimes (views 02, 07, 08, 09); the anatomy of one call with its eight decisions and 3 millisecond budget (view 11)
  • Issuance and rotation, root rotation, the canary that fails with 30 second analysis and about 41 seconds to revert, cross-cluster failover at 70 percent healthy, the retry storm contained by a 20 percent budget, and the total control-plane outage hour by hour (views 16 to 21)
  • Configuration in waves of about 45 minutes across 20 clusters, fleet upgrade by revision, observability pages, the cost arithmetic of 80 and 400 virtual processors and 4.8 terabytes, the roadmap and the failure modes (views 22 to 31)
  • The six actors and their journeys, the five stores, the intent data model and the configuration and telemetry pipelines (views 03 to 06, 10, 12 to 15)
  • The 8 week proof phase and its pass and fail lines, including the 20 millicore tripwire (architecture one-pager)
  • The five public sources behind the examples, each fetched in this run (see sources)
Scenario & assumptions
  • The model railway, its people, the screens and every meter are illustrations; none of them is a measurement
  • The hypothetical product interface (a shopper's phone, service levels, the cast, the five stages, the call anatomy, the identity and pass view, the three roads, the outage clock, the failover and wave gate screens, the retry chart, the rollout console, the cost board and the operations board) is illustrative, not the real product; its layout, wording and the example values on it (such as 0.9 and 1.1 milliseconds for the two proxies, 4.9 percent applied share, 3.1 times errors, 0.004 percent mesh-caused errors and 9 hours 12 minutes to expiry) are invented to show the idea
  • The roads not taken are modelled by this walkthrough. Asking the office on every call is derived arithmetic: if every call depends on a 99.9 percent service, at most 99.9 percent can succeed. Fetching only on a miss assumes that one call in a hundred misses during an outage; the decision record gives the argument, not a benchmark
  • The shopper's app, its items and the 2.0 milliseconds of mesh time on the phone screen are an illustration; the 3 millisecond budget is the requirement's
  • The walkthrough shows three districts of six stations; the design runs 20 clusters, 2,000 services and 40,000 proxies. Trains stand for calls and are not drawn to scale
  • The wave and cluster failures show the mechanism of views 19 and 23 with invented counts on the gate screen (0.04 percent, 312 rejected, 9 percent on the old version)
  • The 12 and 24 hour outage behaviour follows view 21 and decision record 4; the walkthrough does not re-derive the moment each individual certificate expires
  • The use case is a design, not a report on a running system: every rate, latency and cost is the requirement's target or a planning estimate until the proof phase replaces it, and seven product behaviours must be confirmed on the chosen releases
  • The use case is this practice's design on open-source software on owned hardware; the examples show the same ideas elsewhere and are not descriptions of how Istio, Amazon or Google build any system

What you leave with

  • A proxy that forwards on three things it already holds survives the loss of every component that tells it what to do
  • Make credential lifetime, outage budget and renewal point one written decision, because shortening one quietly shortens the others
  • Bound retries by a fraction of real traffic and by the caller's deadline, not by an attempt count alone
  • Ask of your own design: what does a request consult while it is in flight, and could each of those things be gone for an hour?

Sources

  • Istio documentation: Architecture Fetched in this run. Used for: an Istio service mesh is logically split into a data plane and a control plane; the data plane is a set of Envoy proxies deployed as sidecars that mediate all network communication between microservices; Istiod converts high level routing rules into Envoy-specific configuration and propagates it to the sidecars at runtime.
  • Amazon Builders' Library: Static stability using Availability Zones Fetched in this run (the page may redirect). Used for: the Amazon EC2 service consists of a control plane and a data plane; the data plane is designed to be statically stable in the face of control plane availability events; the machine has local access to the information it needs to route packets; during a control plane impairment, traffic it had been able to send and receive before the event continues to work.
  • Amazon Builders' Library: Timeouts, retries, and backoff with jitter Fetched in this run (the page redirects). Used for: in a five-deep stack of service calls with three retries at each layer, if each layer retries independently the load on the database will increase 243x; retrying at a single point in the stack is the best practice; limiting retries locally with a token bucket.
  • Site Reliability Engineering (Google), chapter 22: Addressing Cascading Failures Fetched in this run. Used for: consider a server-wide retry budget, for example only allow 60 retries per minute in a process, and if the budget is exceeded don't retry, just fail the request; the strategy can contain the retry effect and be the difference between dropped queries and a global cascading failure; randomly distributed retries to avoid retry ripples.
  • Envoy documentation: Circuit breaking Fetched in this run. Used for: in general retry budgets are recommended; static circuit breaking should aggressively circuit break retries so that retries for sporadic failures are allowed but the overall retry volume cannot explode and cause a large scale cascading failure.

Narration voice: Kokoro-82M v1.0, voice af_heart (kokoro-onnx 0.6.1). Apache-2.0 (Kokoro-82M weights).

Keep going

More walkthroughs

All walkthroughs →