Architecture One-Pager

Edge Cache and CDN Platform · Solution Architecture v1.0 · Edge Architecture · 2026-09 · 29 views · open source, fully on-premises

A cache hit is a claim that a stored response is the correct answer for this request. The cache key is the whole content of that claim. This architecture is built around getting the key right, and around everything that becomes cheap once it is.

Delivery platforms are sold on latency, but what they are worth is the origin traffic they prevent. Both come from the same mechanism: a request answered from a nearby cache instead of a distant origin. The failure that matters most does not show up on a latency graph. It is a hit on the wrong key: one user's signed-in page served to thousands of strangers, from cache, at line rate, with every metric green. The estate here is 18 points of presence on four continents, 40 properties belonging to business units of one organisation, 250,000 requests per second at peak and 900 million cacheable objects, all on open-source software and hardware the organisation operates itself. The job is not only to serve bytes quickly. It is to serve only correct bytes, make the origin see a small and predictable fraction of the traffic, make a content change visible everywhere within seconds with proof, and keep working when anything outside the PoP is down.

Each PoP announces four anycast prefix classes through BIRD. Katran spreads connections across cache nodes using consistent hashing that understands QUIC. On each node, Envoy terminates TLS 1.3, HTTP/2 and HTTP/3, verifies signed URLs and enforces per-property ceilings, then passes the request over loopback to Apache Traffic Server. Traffic Server builds the key with keyspec, a small Rust library that follows the property's declared rules and nothing else. It checks the tag table for invalidations, then serves from memory or NVMe. A miss goes to the node that owns the key in the PoP's ring, then to one of four shield pools placed next to the origins, where concurrent fetches collapse into one and per-origin ceilings are enforced. Property configuration lives in Git, is validated with CUE, invariant checks and a two-user identity probe, compiled into signed bundles and rolled out one ring at a time by a Temporal workflow that halts itself on a regression. Purges are records in a NATS JetStream log mirrored into every PoP. A purge is applied by timestamp comparison, so it costs the same for one object or two million, and each PoP's acknowledgement is recorded. Access records go through Vector and Kafka into ClickHouse, where hit ratio, offload and cost per terabyte are computed per property and per path pattern. None of the control or telemetry systems is on the request path.

What it is, and what it is not

• A cache whose key is declared per path and checked before release — not a cache that trusts whatever Vary header the origin happens to send.
• A platform that measures itself by origin bytes avoided — not a platform that measures itself by bytes served or requests answered.
• Eighteen PoPs that hold nothing authoritative — not eighteen sites with data to back up, replicate or recover.
• A purge that reports which PoPs applied it — not a purge API that returns 200 and hopes.
• PoPs that keep serving for weeks with the control plane gone — not an edge that needs its control plane to answer a request.
• Edge caching, shielding, invalidation and steering — not WAF, DDoS scrubbing, video packaging or edge compute, which are separate products or later phases.

The decisions that are the architecture

1. The key is declared, not inferred (ADR-01) — Every path pattern declares exactly what enters its key. Nothing is cacheable until declared, and an origin Vary outside the declared key stops the object being stored and alerts the owner.
2. One key implementation (ADR-02) — The same Rust library computes the key in the cache plugin, the validator and the explain API. The tool an owner uses to check a key cannot disagree with production.
3. Prove identity before release (ADR-04) — Validation fetches declared test URLs as two users. A path whose response changes with the user cannot become cacheable under a key that leaves the user out.
4. Invalidate by time, not by enumeration (ADR-13) — A purge is one row: tag, time, mode. A lookup compares the object's store time with it. Two million objects cost the same as one, and serving never waits for a purge.
5. A purge has an outcome (ADR-14) — Invalidations are an ordered, replicated log mirrored into every PoP. Each PoP records how far it has applied, and a returning PoP catches up before it announces.
6. One fetch per object per shield (ADR-10) — Misses hash to a shield node next to the origin, and requests collapse at the node, the PoP and the shield. Origin load follows how often content changes, not how many users arrive.
7. The PoP depends on nothing beyond itself (ADR-21) — PoPs apply only signed bundles, keep the last five on disk and serve on the last good one indefinitely. Control plane, logs and analytics may all be down while delivery continues.
8. Keys that expire before they matter (ADR-24) — Each PoP holds its own 14-day certificates, sealed to its TPM. A stolen server is revoked alone, and its keys are useless within days whatever happens to revocation.

Why this holds up over time

Delivery platforms usually age badly in one of four ways. The cache engine is replaced and every property's behaviour quietly changes with it. The organisation decides to buy the edge instead of running it, and the configuration and purge semantics turn out to be tied to one product. The object count grows past what the purge or index design assumed. Or a vendor changes its licence and a core component has to be replaced under pressure. The design defends against each of these on purpose.

• The central rule comes from HTTP, not from a product. A declared key, not-cacheable by default, and refusing private responses are expressed in HTTP caching semantics (RFC 9111), Cache-Status (RFC 9211) and surrogate keys. They were right for proxy caches twenty years ago and will be right for whatever replaces Traffic Server. The property owner's declaration is the lasting asset, not the engine that reads it.
• Configuration is compiled, so the engine can change underneath it. Owners write engine-neutral YAML. A compiler turns it into Envoy and Traffic Server configuration today. Moving to another engine, or to a commercial CDN's configuration API, means writing a new compiler back end and replaying the ring rollout, not rewriting 40 properties by hand.
• The key is pinned by tests, not by memory. Because one library computes the key everywhere, its behaviour is fixed by a corpus of real requests and expected keys. A new engine or a new team inherits the exact key semantics by linking the library or passing the corpus.
• Nothing at the edge needs migrating. Cached content is derived and disposable, so hardware generations, storage technologies and even the cache engine can be changed one PoP at a time with a cold start as the only cost. There is no data to move and no recovery objective to protect during a migration.
• Costs that do not grow with object count. Invalidation by timestamp comparison, per-node LMDB tables bounded by the TTL ceiling and hashed ownership inside a PoP mean that going from 900 million to 2 billion objects lowers the hit ratio at the margin but does not change a single design element.
• Standard protocols at every seam. BGP, HTTP/3 and QUIC, ACME, SPIFFE, OIDC, the S3 API, the Kafka protocol and Prometheus exposition sit between the components. Any single product can be replaced without its neighbours noticing, which is what keeps a ten-year platform from becoming a ten-year dependency.
• Governance risk is named, with an exit for each. Components are chosen for open governance where it exists: Apache, CNCF, Linux Foundation. OpenBao was chosen over Vault after Vault's licence change showed what a single-vendor project can do. Where one company leads, such as Katran, NATS, Temporal, ClickHouse and Vector, the record names the fallback: IPVS with Maglev, Kafka, plain workers with PostgreSQL, Parquet on Ceph, Fluent Bit.
• The platform carries its own evidence. Per-node capacity, hit ratio per path, cost per terabyte, cost per miss and the cost of n−1 headroom are published every month. Whoever re-decides these choices in five years starts from measurements, not from this document.

Non-functional targets

Targets from the requirement, and the mechanism that meets each one. Every latency, throughput and cost figure is replaced by a measured value in the proof phase and republished monthly.

Scope

In scope

• Anycast addressing, L4 balancing and steering for 18 PoPs on four continents
• TLS, HTTP/2 and HTTP/3 termination with per-property certificates and per-PoP keys
• Declared cache keys, validation, the explain tool and the test path
• Edge caching in memory and on NVMe, partial objects and range requests
• Four shield pools with collapsing, per-origin ceilings and origin mTLS
• Purge by URL, surrogate key and property, with per-PoP acknowledgement and reconciliation
• Versioned configuration, ring rollout, automatic halt and revert
• Access records, hit ratio and offload per path pattern, external probes, cost per terabyte

Explicitly out of scope

• Origin application design, beyond the conditional-request and mTLS contract they are asked to meet
• WAF, bot management and DDoS scrubbing as products; the edge enforces its own ceilings only
• Edge compute and personalisation, which are Phase 3 and must extend the key contract when they come
• Video packaging, transcoding and DRM
• Peering and transit commercials, beyond consuming their measured cost

The eight-week proof

The proof does not try out every feature. It tests the properties everything else depends on: a key that cannot leak, a purge whose outcome is known, a PoP that keeps serving when everything else is gone, and a monthly traffic volume that justifies running the edge at all.

1. Two real properties onboarded with declared keys, the explain API and the identity probe, reviewed by a security engineer trying to make one leak
2. Two PoPs (FRA and IAD) with Katran, Envoy and Traffic Server, one shield pool, measured per-node Gbps ceiling at TLS
3. Tag table plugin with JetStream mirror, measured purge acknowledgement across the Atlantic
4. Signed bundles with a ring controller that halts on a staged hit-ratio regression
5. Traffic mirrored from the current delivery path for four weeks to measure real monthly volume and peak-to-average ratio

• Switch the control plane off for seven days; the PoPs keep serving, certificates stay valid, purges queue and then drain
• Withdraw one PoP for nine hours, issue 1,000 purges, bring it back and prove it serves nothing stale
• Push a rule that fragments the cache; the canary halts and reverts without a person deciding
• Send Vary: Cookie from an origin whose rule excludes cookies; the object is refused and the owner alerted

Open risks, carried rather than hidden

The reasoning behind every component and technology choice is in the Architecture Decision Record: 29 records across 8 areas, each with the alternatives that lost and what the choice costs.