Architecture Views

Who decides what makes two requests the same request?

Context

A key of scheme, host and path gives the best hit ratio and is wrong for anything that varies. A key built from the origin's Vary header is correct only if forty origin teams always send the right headers, and it hands cache fragmentation, and therefore cost, to whoever writes those headers. The catastrophic failure of a CDN is not a miss or an outage but a hit on the wrong key, which produces no error anywhere.

Decision

Each path pattern declares its key as an explicit list: scheme, host, path, a named allow-list of query parameters, named request headers, an optional device or geography class, and an identity field that is either none or the header carrying identity. Anything not listed never enters the key. A path with no declaration is not cacheable. The platform applies its own maximum TTL per property and a 30-day ceiling overall, whatever the origin sends.

How it works on-premise

Owners write the declaration as YAML in the property's directory in Forgejo. The CUE schema requires the identity field on every cacheable rule, with no default. The compiler turns rules into a compact table that the keyspec plugin in Traffic Server loads from the signed bundle, indexed by the rule ID that Envoy stamps on the request after matching the property.

Options weighed

• ChosenDeclared per-path key, not cacheable by default: Explicit, reviewable and testable before release. The owner has to understand their content, which is the point.
• RejectedMinimal key plus origin Vary: Correct only while every origin sends perfect headers. A single missing Vary: Cookie becomes a disclosure.
• RejectedCache everything, bypass on cookies or Authorization: The common default. It leaks as soon as identity travels in a header or parameter nobody thought of.
• Right elsewhereOrigin-controlled keys through a surrogate header: Reasonable for a single team that owns both edge and origin. Here forty teams would each hold a security boundary.

Consequences

What it buys

• A cache hit means what the owner declared it means
• Keys can be reviewed, diffed and tested before traffic sees them
• Fragmentation has a named owner and a visible cost

What it costs

• Lower hit ratio than a permissive platform, visible in cost per TB
• Onboarding requires every owner to do the work before getting any benefit
• Owners need an explain tool and good defaults to succeed

Choose differently when

A single-tenant platform where the same team owns every origin and the edge, and has a tested Vary discipline, can safely let the origin drive the key and take the higher hit ratio.

Why it holds up over time

What makes two requests equivalent is a property of the content, not of the cache engine, so the declaration survives any change of engine, hardware or provider. If the organisation later buys its delivery, the declarations become the specification the vendor is configured against.

How does an owner check a key and trust that production computes the same one?

Context

A key is computed in at least three places: the cache on every request, the validator before release, and a tool owners use to understand their configuration. If each has its own implementation, they drift. Percent-encoding, parameter ordering and header folding are exactly the details where two implementations disagree, and the disagreement is invisible until a user reports someone else's content.

Decision

Key computation, normalisation and the refusal rules live in one library, keyspec, with a C interface. The Traffic Server plugin, the validator and the explain API all link the same build. Its behaviour is fixed by a corpus of real requests and expected keys that must pass before any bundle is built.

How it works on-premise

keyspec is written in Rust with no allocation on the request path and exposes a small C ABI that the Traffic Server plugin calls. The explain API runs as a service in the core and on an internal-only listener at each PoP, so an owner can ask a PoP running a specific bundle version what key it would compute. Hashes use BLAKE3 truncated to 128 bits.

Options weighed

• ChosenOne shared library in Rust with a C ABI: Memory-safe code on the hottest and most security-critical path, callable from C and C++ without a runtime.
• RejectedNative Traffic Server cachekey plugin plus a separate validator: Good plugin, but the validator and explain tool would reimplement its rules and eventually disagree.
• RejectedLua or Wasm key logic: Flexible, but slower per request and makes the key easy to change in ways the validator does not see.
• DeferredKey computed in Envoy before the cache: Possible if the cache engine changes; the library would move with it unchanged.

Consequences

What it buys

• The explain tool is correct by construction
• A regression in normalisation fails CI, not production
• Engine changes keep identical key semantics

What it costs

• A Rust toolchain and FFI boundary to maintain
• Plugin upgrades must be tested against each Traffic Server major version
• Owners must accept a truncated hash in logs rather than the key itself

Choose differently when

If the platform uses only a managed CDN's key configuration, there is no plugin to share and the corpus becomes a conformance test run against the vendor's staging endpoint instead.

Why it holds up over time

The library and its corpus hold the platform's most important behaviour in a form that outlives the engine around it. Rust with a C ABI can be linked from any likely successor to Traffic Server.

What happens when the origin's response contradicts the owner's declaration?

Context

Declarations are made in advance; responses arrive at run time. An origin deploy can start setting a session cookie on a path declared cacheable, or send Vary: Cookie on a path whose key excludes cookies. Honouring the Vary silently fragments the cache; ignoring it can leak. Both hide the contradiction from the owner.

Decision

A response with Cache-Control private or no-store, a session Set-Cookie, or credentials is never stored, regardless of the rule. A Vary naming a header outside the declared key stops the object being stored and raises an owner alert with the path pattern. Accept-Encoding is the one normalised exception, stored as up to three declared variants. Every refusal is counted per path pattern.

How it works on-premise

The keyspec plugin checks responses at the Traffic Server read-response hook and marks refusals as uncacheable. Refusal counters go to Prometheus, and the access record carries a refusal reason. Alertmanager routes Vary-mismatch alerts to the owning team using the property's owner metadata.

Options weighed

• ChosenRefuse and alert: The contradiction is surfaced to the person who can fix it, and nothing unsafe is stored meanwhile.
• RejectedHonour Vary silently: Correct but quietly destroys hit ratio, and the owner never learns why.
• RejectedIgnore Vary, trust the declaration: Keeps the hit ratio and risks a disclosure the moment the origin is right and the declaration is wrong.
• RejectedAuto-extend the key with the Vary header: Lets origin headers change the key, which ADR-01 exists to prevent.

Consequences

What it buys

• Origin drift cannot turn into a leak
• Refusals are measurable per path
• Owners learn about header bugs from an alert, not an incident

What it costs

• A bad origin deploy drops the hit ratio on that path until fixed
• Alert noise if an owner ignores refusals
• Some legitimate Vary use must be declared explicitly

Choose differently when

For a public, anonymous-only property with no cookies at all, an organisation might accept honouring Vary to save owners effort. It remains unsafe the day anyone adds authentication.

Why it holds up over time

HTTP will keep letting origins say one thing and configuration say another. Refusing on contradiction is a rule about trust, not about a product, and it applies equally to any future engine or vendor.

How does the platform reject a change that would make a user-specific path cacheable without the user in the key?

Context

Schema validation catches malformed configuration but not a well-formed wrong one. The requirement asks for a validator that rejects any change making an identity-varying path cacheable under a key that omits identity. The platform cannot know from configuration alone which paths vary by identity; it has to either trust the owner's declaration or check the origin.

Decision

Three layers run on every merge request. CUE checks types, required fields and defaults. Invariant checks enforce cross-rule rules, for example that a rule marked identity-varying names its identity header, and that no cacheable rule overlaps a pattern another rule marks private. A differential probe fetches each rule's declared test URLs from the origin as two test users and fails the change if the bodies differ while the key omits identity.

How it works on-premise

Forgejo Actions runners in the core run CUE and the invariant checks, written in Go against the compiled rule table. The probe runs from the core through the origin's mTLS path using two test identities per property held in OpenBao, and ignores declared volatile fields such as timestamps and CSRF tokens. CUE also requires at least three test URLs per cacheable rule.

Options weighed

• ChosenSchema, invariants and differential probe: Checks the declaration and the origin's actual behaviour, before any user is exposed.
• RejectedSchema validation only: Catches typos, not a correct-looking key on a user-specific path.
• RejectedManual security review of every change: Does not scale to forty teams and weekly changes, and reviewers miss what a test would catch.
• DeferredShadow traffic comparison at the canary: Stronger, but mirrors real user traffic including personal data; considered for Phase 2 with privacy review.

Consequences

What it buys

• The most dangerous class of change is caught before release
• Owners get a failing check that names the rule
• Test identities make leaks reproducible

What it costs

• Owners must maintain test URLs and identities
• Probe coverage is only as good as the declared URLs
• Volatile fields need declaring or the probe is noisy

Choose differently when

If properties are entirely anonymous, such as software downloads or public media, the probe adds little and schema plus invariants are enough.

Why it holds up over time

Pre-release checks tied to observable origin behaviour do not depend on the engine, and the probe is the same whether the edge is self-operated or bought. As origins change over years, the probe keeps testing what they do now, not what they did at onboarding.

Which open-source cache engine can meet the key, storage and shielding requirements on hardware we run?

Context

The requirement names Traffic Server or Varnish. The engine must keep a cache across restarts so a node rejoins warm, stream in-flight fetches to waiters, serve ranges from partially cached large objects, route misses by consistent hash to parents, and support a plugin that can override lookup results. It will hold hundreds of terabytes per PoP on NVMe.

Decision

Apache Traffic Server 10 on every edge and shield node, with a raw-device NVMe cache that persists across restarts, a RAM cache for the hot set, read-while-writer collapsing, the slice and cache_range_requests plugins for large objects, next-hop strategies for peers and shields, stale_response for stale-while-revalidate and stale-if-error, and three platform plugins: keyspec, tagguard and cache-status.

How it works on-premise

Traffic Server runs as a systemd service from the signed node image. Cache spans are the raw NVMe devices opened through LUKS2 mappings. Configuration files (remap, strategies, volumes, plugin tables) come only from the active bundle directory, and changes are applied with a configuration reload, not a restart.

Options weighed

• ChosenApache Traffic Server: Persistent disk cache, parent selection and a lookup-level plugin API; long history as a CDN engine at large operators; Apache governance.
• RejectedVarnish Cache (open source): Excellent collapsing and VCL, but the open-source storage does not persist across restarts and it needs a separate TLS terminator; persistence is in the commercial edition.
• RejectedNGINX proxy_cache: Simple and familiar, but file-per-object storage struggles at hundreds of millions of objects and advanced collapsing and purge features are commercial.
• RejectedBuild a cache on a key-value store: Years of work to reach what Traffic Server already does.

Consequences

What it buys

• Node restarts and upgrades rejoin warm
• Shield routing and intra-PoP ownership use built-in strategies
• One engine for edge and shield roles

What it costs

• Smaller community and fewer engineers who know it than NGINX or Varnish
• Plugin code in C++ against a changing API
• Configuration is spread across several files, which the compiler has to own

Choose differently when

A platform mostly caching APIs with small objects and frequent logic changes, and where warm restarts matter less, may prefer Varnish for VCL's expressiveness.

Why it holds up over time

Traffic Server is chosen for capabilities, not its configuration language, and the compiler (ADR-21) and keyspec (ADR-02) keep the rest of the platform independent of it. If it stops being the right engine, a replacement is a new compiler back end and a plugin port, rolled out one ring at a time.

Where do TLS termination, token checks and per-property limits run?

Context

The edge must terminate HTTP/1.1, HTTP/2 and HTTP/3 with TLS 1.2 and 1.3 and per-property certificates, verify signed URLs before lookup, enforce per-property rate, connection and bandwidth ceilings, and drain gracefully. Private keys sit in facilities the organisation does not control, and the process that parses untrusted origin responses is the one most likely to have a memory-safety bug.

Decision

Envoy runs on every cache node in front of Traffic Server, connected over loopback. It holds TLS keys and does termination, HTTP/3, SNI-based certificate selection, signed URL verification, token stripping, local rate limits and drain. Traffic Server holds no private keys.

How it works on-premise

Envoy loads listeners, routes and limits from files in the active bundle and receives certificates through SDS from the node agent (ADR-24). Signed URLs follow the CDNI URI Signing shape (RFC 9246) and are verified with the jwt_authn filter against keys in the bundle. Limits use the local rate-limit filter, sized per node from the property ceiling divided by node count. QUIC connection IDs carry a server ID so Katran keeps QUIC flows on the same node.

Options weighed

• ChosenEnvoy in front of Traffic Server: Mature HTTP/3, hot certificate updates, draining and local limits, and key material isolated from the cache process.
• RejectedTraffic Server terminating TLS itself: One process fewer, but keys would live in the process that parses origin responses, and its HTTP/3 support is less mature.
• Right elsewhereNGINX or HAProxy as terminator: Both are sound; Envoy is chosen for SDS certificate delivery and QUIC-LB connection IDs.
• RejectedGlobal rate-limit service: Accurate across nodes, but puts a network dependency on the request path.

Consequences

What it buys

• A cache bug cannot read TLS keys
• Certificates rotate without restarts
• Drain behaviour is precise and testable

What it costs

• Two proxies per node to operate and upgrade
• A loopback hop and some CPU on every request
• Local limits are approximate across nodes

Choose differently when

On small edge nodes where CPU is scarce and physical security is good, letting the cache terminate TLS saves a hop and a process.

Why it holds up over time

Splitting the key-holding terminator from the content-handling cache is a security boundary that holds whatever the products are. Either side can be replaced independently because the seam between them is plain HTTP over loopback.

Does every node in a PoP hold everything, or does the PoP shard its cache?

Context

Replicating the working set on every node wastes storage in proportion to node count and makes node loss trivial. Consistent hashing across nodes multiplies effective PoP cache size but adds a hop and turns node changes into redistribution. The working set is around 150 TB of distinct bytes, larger than one node's NVMe, and collapsing only works when requests for one key reach one process (view 15).

Decision

Each node serves its own RAM and NVMe cache. On a local miss, the node asks the key's owner on the PoP's consistent-hash ring before going to the shield. The owner fetches from the shield and both nodes store the object. Popular objects therefore end up on many nodes, and long-tail objects on roughly one.

How it works on-premise

Traffic Server next-hop strategies in peering-ring mode define the PoP's edge nodes as a ring with virtual nodes. The ring is generated from NetBox inventory into each PoP's bundle. Katran health checks remove a failed node from connection balancing, and the ring skips it to the next owner.

Options weighed

• ChosenLocal hot set plus owner ring for the tail: Near N times the effective cache for the long tail, one owner fetch per key per PoP, and only 1/N of the tail lost with a node.
• RejectedFull replication on every node: Simple, but the long tail would be fetched from the shield once per node and PoP capacity would equal one node's disk.
• RejectedPure sharding with no local hot set: Popular objects would always take an extra hop and concentrate load on their owners.
• RejectedA separate storage tier inside the PoP: More hardware and a new dependency inside the site.

Consequences

What it buys

• Effective PoP cache size close to the sum of its nodes
• Node loss costs 1/N of the tail, never a PoP flush
• Collapsing works across the PoP, not only per node

What it costs

• An extra intra-PoP hop on tail misses, under 1 ms
• Adding or removing a node moves about 1/N of ownership
• Intra-PoP fabric must carry owner traffic

Choose differently when

When a working set fits comfortably on one node, or nodes are replaced very often, full replication is simpler and the extra shield fetches are cheap.

Why it holds up over time

The principle, which is to converge each key on one owner so collapsing works while keeping hot objects local, holds however large disks become. When disks outgrow the working set the ring still helps collapsing and costs almost nothing.

How does one property's traffic avoid evicting another property's cache?

Context

Forty properties share every PoP. A large property publishing a video catalogue can fill NVMe and evict a smaller property's long tail, dropping its hit ratio without any change of its own. The requirement asks for a per-property footprint floor and ceiling at each PoP and for per-property accounting.

Decision

Each node has a RAM cache for the hot set and NVMe for the long tail. NVMe is divided into Traffic Server cache volumes: large properties get a dedicated volume sized between their declared floor and ceiling, and small properties share a pooled volume with a combined floor. Eviction happens inside a volume, so one property cannot evict another's objects. Per-property request, connection and bandwidth ceilings are enforced by Envoy.

How it works on-premise

Volume layout per PoP is compiled from property declarations and PoP disk inventory into Traffic Server's volume and hosting configuration. Changing a volume layout reinitialises that volume, so it is rolled out one node at a time, like a node restart. Per-volume fill and eviction rates are exported to Prometheus.

Options weighed

• ChosenDedicated volumes for large properties, pooled volume for the rest: Hard isolation where it matters, without fragmenting disk into forty small partitions.
• RejectedOne shared cache with global LRU: Best total hit ratio, no isolation; one release can wreck a neighbour.
• RejectedOne volume per property: Strong isolation but wastes space in small volumes and hits volume count limits.
• DeferredSoft quotas by eviction weighting: Would give finer control but needs engine changes.

Consequences

What it buys

• A property's hit ratio depends on its own traffic
• Storage cost per property is directly measurable
• Floors protect small, important properties

What it costs

• Total hit ratio slightly below a shared cache
• Resizing a volume costs that slice of cache on each node
• Owners must estimate a footprint, revised quarterly from measurements

Choose differently when

A single-tenant platform, or tenants with no internal chargeback, gets more from one shared cache.

Why it holds up over time

Isolating tenants by storage allocation works with any storage medium and any engine that can partition its cache. The declarations are in engine-neutral configuration, so the policy survives an engine change even if the mechanism differs.

When the origin fails or the PoP is overloaded, what does a user get?

Context

Serving stale content during an origin outage keeps a news site up and would show a wrong balance on a banking page. That is a product decision, not a platform default. The platform also needs a declared order for shedding load, so overload degrades service predictably rather than randomly.

Decision

Each path declares a freshness class. Strict revalidates before serving once stale and returns the declared error page if the origin cannot answer. Standard serves stale while revalidating for a declared window, 60 seconds by default. Resilient also serves stale on origin error, for up to 24 hours. The platform caps all windows whatever the origin sends. Under overload the edge sheds bypass and uncacheable traffic first, then low-priority properties, then revalidation, and cache hits last. Every stale response is labelled in Cache-Status.

How it works on-premise

The compiler turns classes into Traffic Server stale_response plugin settings and revalidation behaviour per rule. Envoy's overload manager and route priorities implement shedding by rule class and property priority. Stale-served ratio per path pattern is computed in ClickHouse and alerted in Prometheus.

Options weighed

• ChosenThree declared classes with platform caps: Owners choose in advance; the platform keeps the choice bounded and measurable.
• RejectedFollow origin Cache-Control extensions only: Puts outage behaviour in origin headers that nobody reviews.
• RejectedAlways serve stale on error: Right for media, wrong for prices, balances and security-sensitive pages.
• RejectedPer-request stale decisions in code: Flexible and impossible to reason about during an incident.

Consequences

What it buys

• Incident behaviour is known before the incident
• Stale content is visible and countable
• Overload hits the cheapest traffic first

What it costs

• Owners must classify every path
• Strict paths show errors during origin outages, by design
• Shedding low-priority properties needs agreed priorities

Choose differently when

A platform serving one kind of content, such as only static software packages, can use one class everywhere.

Why it holds up over time

Stale-while-revalidate and stale-if-error are standard HTTP extensions, and the classes are expressed as policy, not engine settings. Whether wrong or missing is worse is a business question that stays with the owner however the technology changes.

What shape should the shield tier be?

Context

One global shield per origin maximises offload and concentrates failure. A shield per region shortens miss paths and multiplies origin fetches by the number of regions. A hashed mid-tier across tier-1 PoPs couples origin protection to PoP membership changes. Origins are in two data centres, one in the eastern US and one in Frankfurt, and origin egress is the largest avoidable cost.

Decision

Four shield pools run on dedicated nodes in four tier-1 sites: IAD and DFW for origins in the eastern data centre, FRA and LHR for origins in Frankfurt. Each origin set declares a primary and a secondary shield on the same continent. Every miss is hashed to one node within the primary shield pool, so an object converges on one node and one fetch. Asia and South America edges use the same shields and accept higher miss latency.

How it works on-premise

Edge strategies use a consistent-hash parent group whose members are the primary pool's nodes, with the secondary pool as the failover group. Shield nodes run the same image as edge nodes with shield role configuration and 16 NVMe drives each. A regional shield pool at SIN is defined in configuration but not enabled.

Options weighed

• ChosenOrigin-adjacent shields in continental pairs: Highest offload for this origin placement, short failover distance, no dependence on edge PoP membership.
• RejectedOne global shield per origin: Similar offload, but one site's failure sends full miss volume to a secondary or the origin.
• DeferredRegional shields on every continent: Better miss latency for Asia and South America, and one origin fetch per region. Enabled if measured miss share there justifies it.
• RejectedConsistent-hash mid-tier across all tier-1 PoPs: No dedicated nodes, but a PoP withdrawal reshuffles origin load.

Consequences

What it buys

• Origin sees fetches from a small, known set of addresses
• One fetch per object per shield per TTL
• Shield tier scales with miss volume, independently of edges

What it costs

• Asia and South America shield hits exceed the 60 ms budget
• Sixteen dedicated nodes carry no user traffic directly
• Shield failover changes miss latency

Choose differently when

If origins become multi-region, or Asian traffic grows so that its miss share drives p95, move to regional shields and accept one origin fetch per region.

Why it holds up over time

Shield placement follows origin placement. When origins move, shields are reassigned by editing origin sets, not by redesigning the tier. The convergence rule, one key per shield node, holds at any size.

How is an origin protected from the edge, including when the shield itself fails?

Context

An origin's protection must not depend on the origin behaving well. A flash crowd, a cold PoP refilling or an expired popular object can all send more requests than an origin can take. When a shield fails, falling back to direct edge-to-origin fetches at full miss volume causes the origin outage the shield exists to prevent.

Decision

Each origin set declares a concurrency and request-rate ceiling, a slow threshold and a down threshold. Shields queue up to the ceiling and then serve stale or shed according to the freshness class. Slow origins are handled by waiting within bounds; down origins trigger ordered endpoint failover. If both shields are unavailable, edges may fetch directly from origins only under a per-origin rate cap of 5% of normal miss volume.

How it works on-premise

Traffic Server per-origin connection limits with queueing enforce concurrency at the shield. Health is judged on error rate and connect failures for down, and on time to first byte against the declared threshold for slow. Endpoint order and thresholds come from the origin set in the bundle. The direct-path cap is an Envoy rate limit on a separate origin cluster that is active only while shields are marked down.

Options weighed

• ChosenCeilings at the shield, slow and down treated differently, capped direct path: The origin is protected by the platform, and failures degrade without cascading.
• RejectedRely on origin autoscaling and rate limits: Moves protection to forty teams and fails the first time one is under-provisioned.
• RejectedFree direct fallback when shields fail: An unshielded fallback at full miss volume is an origin outage with extra steps.
• RejectedNo direct path at all: Safer for origins, but a double shield failure would stop all misses including for strict paths.

Consequences

What it buys

• Origin load is bounded and predictable
• Slow origins do not trigger unnecessary failovers
• Shield failure degrades gracefully

What it costs

• Owners must declare realistic ceilings
• Queued requests add latency at peaks
• Capped direct path means some misses fail during double shield failure

Choose differently when

Origins that are themselves large object stores with effectively unlimited capacity may need only generous ceilings, but should still have them.

Why it holds up over time

Protecting a dependency from its callers with ceilings and bounded fallback is standard reliability practice and does not depend on the engine. Thresholds are data in origin sets that owners revise as origins change.

How can an origin refuse traffic that did not come through the platform?

Context

An origin reachable directly lets attackers skip edge limits, token checks and caching. IP allow-lists break when shields are added and do not prove which service is calling. The requirement asks for mTLS or signed requests from shield to origin.

Decision

Shields call origins over mTLS with a short-lived client certificate that carries the shield's SPIFFE identity. Origins accept only that identity for platform traffic. The edge-to-origin direct path uses a separate, distinguishable identity, so an origin can see and alert on its use.

How it works on-premise

SPIRE issues X.509 SVIDs to shield nodes after TPM attestation, rotated hourly. The SPIRE trust bundle is published to origin teams, who configure their ingress (for example Envoy or NGINX) to require it. OpenBao's PKI serves as the upstream CA for SPIRE.

Options weighed

• ChosenmTLS with SPIFFE identities: Proves which platform component is calling, rotates automatically, standard for origin ingress to verify.
• RejectedIP allow-lists: Brittle, and proves only an address.
• RejectedShared secret header: Easy to leak and hard to rotate across forty origins.
• DeferredHTTP message signatures (RFC 9421): Useful for origins behind TLS-terminating load balancers that cannot see client certificates.

Consequences

What it buys

• Origins can close their public exposure
• Direct-path use is visible to the origin
• No long-lived secrets to rotate

What it costs

• Origin teams must configure client certificate validation
• SPIRE becomes a dependency of shield-to-origin fetches, cached for an hour
• Debugging from outside the platform needs a test identity

Choose differently when

Origins that are public object stores serving public content may skip mTLS and use signed URLs instead.

Why it holds up over time

SPIFFE is an open standard with several implementations, and mTLS is supported by every origin stack. Changing identity providers does not change the origin contract.

How can a purge of two million objects reach 18 PoPs in seconds without affecting serving?

Context

The requirement asks for purge by URL, surrogate key and property, 99% of PoPs acknowledging within 5 seconds p95 for a two-million-object surrogate key purge, and purge storms that never block serving. Deleting matching objects means finding them: an index of tags on every node, and millions of delete operations competing with serving I/O.

Decision

An invalidation is a record of selector, time and mode. Selectors are a key hash, a surrogate key or a property generation. Each node keeps a table from selector to latest invalidation time. On lookup, an object whose store time is earlier than any of its selectors' invalidation times is a miss (purge) or stale (expire). Entries are dropped after the TTL ceiling, which bounds the table. Nothing is deleted; invalidated objects are overwritten on refetch or evicted normally.

How it works on-premise

The table is an LMDB file on each node's NVMe, written only by the PoP agent and memory-mapped read-only by the tagguard plugin in Traffic Server. Surrogate keys come from an origin header or configuration and are stored with the object. tagguard runs after cache lookup and changes the lookup result. A property-wide purge advances the property generation, a selector every stored object carries, so the same comparison invalidates everything the property has stored without touching the key.

Options weighed

• ChosenTimestamp comparison at lookup: Cost independent of object count, no I/O storm, serving only reads a memory-mapped table.
• RejectedTag index with active deletion: Accurate and familiar, but two million deletes on each of 90 nodes compete with serving.
• RejectedRegex invalidation: Evaluated per lookup against a growing list, and a single broad regex can invalidate far more than intended.
• RejectedOnly versioned URLs, no purge: Ideal where possible (ADR-16), impossible for URLs the publisher does not control.

Consequences

What it buys

• Purge cost does not depend on how many objects match
• Purge storms never touch serving I/O
• Returning PoPs replay a small log instead of scanning caches

What it costs

• Invalidated objects occupy disk until evicted or refetched
• Lookup reads a table on every request, microseconds
• Depends on a plugin changing lookup status, to be confirmed

Choose differently when

A platform with few objects per tag and rare purges can use active deletion, which frees disk immediately.

Why it holds up over time

Timestamp comparison is independent of object count, storage medium and engine, so it keeps working at 2 billion objects. It relies only on objects recording when they were stored, which every cache does.

What carries invalidations so that none acknowledged is lost and every PoP's outcome is known?

Context

Invalidation state is small, ordered, must not be lost (RPO 0) and must reach 18 PoPs across continents in seconds. PoPs are sometimes unreachable for hours. A purge whose outcome is unknown has not happened, so each PoP's position must be known, and a returning PoP must be able to replay what it missed.

Decision

Purges are published to a NATS JetStream stream replicated three times across the two core data centres and a witness site. The purge API returns only after quorum. Each PoP runs a leaf node with a local mirror of the stream, so delivery survives WAN interruptions and replay is local. The PoP agent applies records in order and publishes an acknowledgement with the highest sequence applied. The purge API records acknowledgements per PoP and reports which PoPs are pending. The stream keeps 30 days.

How it works on-premise

Three JetStream servers form the core cluster (DC East, DC EU, DFW compute). Each PoP's leaf node connects to the nearest core server with credentials that allow subscribing to the invalidation stream and publishing only on that PoP's acknowledgement subject. Per-property publish budgets are enforced in the purge API; JetStream flow control provides back-pressure during purge storms.

Options weighed

• ChosenNATS JetStream with leaf-node mirrors: Designed for edge topologies: a small footprint per PoP, local durable mirrors, quorum replication in the core.
• RejectedApache Kafka with MirrorMaker to each PoP: Proven, but running Kafka in 18 PoPs is heavy, and it is already used for telemetry where different trade-offs apply.
• RejectedHTTP fan-out from the purge API to each PoP: Simple until a PoP is down: then the API has to own retries, ordering and replay.
• Rejectedetcd or PostgreSQL as the log: Strong consistency, but not built for fan-out streaming to remote sites.

Consequences

What it buys

• Acknowledged purges survive any single site loss
• PoPs catch up locally after an outage
• Per-PoP outcome is a recorded fact

What it costs

• A second messaging system alongside Kafka
• Operators must learn JetStream operations
• NATS is led by one company, with Kafka as the named fallback

Choose differently when

An organisation already running Kafka at the edge with strong operational skills might use it for both logs and invalidations and accept the heavier footprint.

Why it holds up over time

The design depends on an ordered, replicated, replayable log with per-consumer positions, which several products provide. The records are small and product-neutral, so moving to Kafka or another log is a transport change, not a redesign.

What stops a PoP that missed purges while away from serving stale content when it returns?

Context

The requirement puts reconciliation in Phase 2 but also says no acknowledged invalidation may be lost to an absent PoP. Without reconciliation, a PoP back from a nine-hour transit outage would serve old prices immediately. With a replayable log already in place (ADR-14), reconciliation is small.

Decision

A PoP that has been withdrawn stays withdrawn until the agent confirms that its tag tables have applied every invalidation up to the current stream head and that its configuration bundle matches the current ring version. If the gap is older than the log's retention, or a tag table is missing, the agent bumps every property generation, which turns every stored object into a miss. Only then does pop-guard announce, starting in cold mode.

How it works on-premise

The agent reads applied_through_seq from LMDB, resumes the leaf mirror from the next sequence and applies records in order. pop-guard queries the agent's readiness endpoint before instructing BIRD to announce. The 30-day stream retention exceeds the declared 14-day maximum absence.

Options weighed

• ChosenReconcile before announcing, in MVP: Makes purge correct from day one at little extra cost.
• RejectedDefer to Phase 2 as the requirement suggests: Ships a purge that can silently not happen for any PoP that was down.
• RejectedFlush every returning PoP: Correct but turns every short outage into a cold start and a shield refill.
• RejectedReconcile while serving: Faster return, but serves stale content during the replay.

Consequences

What it buys

• No stale serving after an outage
• Short outages return warm
• Long outages are handled safely

What it costs

• Return takes seconds longer
• Missing tag tables force a full logical flush
• pop-guard depends on the agent being healthy to announce

Choose differently when

A platform that uses only versioned URLs and never purges has nothing to reconcile.

Why it holds up over time

Catching up from a log before serving is how replicas, databases and caches have safely rejoined for decades. It remains correct at any purge volume or PoP count.

Should content change be handled by purge, by versioned URLs or by short TTLs with revalidation?

Context

Purge gives immediate control but needs reliable global fan-out. Versioned URLs make invalidation unnecessary and long TTLs safe, but only work where the publisher controls URL generation. Short TTLs need no invalidation machinery and put permanent revalidation load on origins. Mixing strategies silently is how a stale asset survives a release, because nobody knows which path relies on which.

Decision

Each path declares an invalidation strategy: versioned, purge or expire. Versioned paths get the property's maximum TTL and are rejected by the purge API, so a mistaken purge is visible. Purge and expire paths must carry surrogate keys. Short TTLs with conditional revalidation are allowed for content that genuinely changes every few seconds, such as live scores, and are never a substitute for a missing purge.

How it works on-premise

The strategy is a field on path_rule, compiled into the bundle and into the purge API's rule index. Asset pipelines hash file contents into file names and set long Cache-Control; the edge still applies its 30-day ceiling. Revalidation uses ETag and If-None-Match end to end from edge through shield to origin, so an unchanged large object costs a 304.

Options weighed

• ChosenDeclared per path, versioned preferred: Uses the cheapest safe mechanism where possible and makes the choice explicit everywhere.
• RejectedPurge everything on release: Works, costs a refill after every release, and hides which paths needed it.
• RejectedShort TTLs everywhere: No invalidation machinery, but permanent origin load and a hit ratio ceiling set by TTL.
• RejectedVersioned URLs only: Impossible for HTML, APIs and user-generated content whose URLs are fixed.

Consequences

What it buys

• Most bytes, the assets, never need invalidation
• Purge volume falls to the paths that really need it
• A release's invalidation plan can be read from configuration

What it costs

• Owners must change asset pipelines to hash names
• The purge API must know rule strategies
• Declaring strategy is one more field to get right

Choose differently when

A property whose content is entirely third-party URLs has no versioning option and must rely on purge and TTLs.

Why it holds up over time

Content-addressed URLs are the most durable invalidation strategy there is: they need no infrastructure and work identically on any CDN. Declaring strategy per path keeps the platform honest as properties change over the years.

How do users reach a PoP, and how are connections spread across its nodes?

Context

Anycast addresses reach the topologically nearest healthy PoP and converge quickly on withdrawal. Inside a PoP, connections must be spread across nodes so that a node change does not break existing flows. HTTP/3 over QUIC complicates this because a client's address can change during a connection, so plain five-tuple hashing drops migrated flows.

Decision

Every PoP announces the service prefixes through BGP. Two dedicated L4 hosts per PoP run Katran, an XDP load balancer using Maglev consistent hashing, forwarding to cache nodes over IP-in-IP. QUIC connection IDs carry a server identifier, so Katran routes QUIC packets by connection ID rather than address. BIRD on the L4 hosts announces prefixes to the border switches only while pop-guard reports the PoP healthy.

How it works on-premise

Border switches are whitebox hardware running SONiC with FRRouting, each PoP with two transit providers and a local exchange port. Katran hosts have 2 × 100 GbE and forward to nodes that decapsulate IP-in-IP. Envoy's QUIC-LB connection ID generator encodes each node's server ID. Every announced prefix has an RPKI ROA.

Options weighed

• ChosenAnycast plus Katran on dedicated hosts: Consistent hashing that survives node changes, QUIC-aware routing, and packet processing in the kernel's fast path.
• RejectedECMP from switches straight to nodes: No extra hosts, but a node change rehashes flows across the PoP and QUIC migration breaks.
• Right elsewhereIPVS with Maglev scheduling: Mature and in the mainline kernel; the named fallback if Katran's single-company governance becomes a problem, at the cost of QUIC-ID routing.
• RejectedDNS-based PoP selection only: Slow, per-resolver rather than per-user, and unbounded drain tails.

Consequences

What it buys

• Node failure affects only its own connections
• QUIC flows survive client address changes
• PoP withdrawal converges in about 30 seconds

What it costs

• Two extra hosts at every PoP
• XDP and eBPF skills needed on the network team
• IP-in-IP adds encapsulation overhead and MTU planning

Choose differently when

Small PoPs with few nodes and little QUIC traffic can run ECMP directly and save the L4 hosts.

Why it holds up over time

Anycast with consistent-hash L4 balancing has been the standard CDN front door for over a decade and does not depend on the products chosen. Katran can be swapped for IPVS or another Maglev implementation without changing anything above it.

Anycast alone, DNS steering, or both?

Context

Anycast cannot say 'move 30% of this property away from Frankfurt'. DNS steering can, but resolvers ignore TTLs, so a drain has a long, unbounded tail. Running both means two systems that can disagree about where traffic belongs, and an on-call engineer has to reason about both at 3 a.m.

Decision

Properties are distributed across four anycast prefix classes. Withdrawing one class at one PoP moves roughly a quarter of that PoP's traffic within BGP convergence time, which covers most finer-than-PoP steering. DNS is used only as a manual, per-property override that points a hostname at a different class. BGP and DNS are both generated from one steering intent object per property, so they cannot be edited separately.

How it works on-premise

Each class is a pair of IPv4 /24 and IPv6 /48 prefixes. PowerDNS Authoritative, fronted by dnsdist and itself anycast from the tier-1 PoPs, serves property hostnames with 60-second TTLs. Steering intent lives in Git beside property configuration and goes through the same validation and ring rollout.

Options weighed

• ChosenPrefix classes plus DNS override from one intent: Fast, BGP-speed steering for most cases, precise per-property moves when needed, one source of truth.
• RejectedAnycast only: Simple, but no way to move one heavy property without moving everything.
• RejectedDNS steering as the primary mechanism: Precise on paper, slow and leaky in practice.
• RejectedAnycast and DNS managed separately: Two systems that will eventually disagree during an incident.

Consequences

What it buys

• Most steering happens at BGP speed
• Per-property moves exist when needed
• No split-brain between steering systems

What it costs

• Four times the address space per PoP
• Class assignment must be balanced as properties grow
• DNS moves still have a slow tail

Choose differently when

Platforms with long-lived sessions and few tenants may prefer DNS for everything and accept slow drains.

Why it holds up over time

Prefix classes are plain BGP and work with any router and any transit provider. Deriving every steering mechanism from one declared intent stays useful whatever mechanisms are added later, including a second CDN.

When should a PoP remove itself from the internet, and what stops a health check from taking everything down?

Context

Withdrawal is the main way to take a PoP out of service. A PoP that stays announced while broken black-holes its users; a PoP that withdraws on one failed probe flaps. Worse, a shared cause such as a bad health rule or a problem the probes themselves have can make every PoP withdraw in turn, which is a global outage caused by the safety mechanism.

Decision

pop-guard withdraws only on sustained local conditions: fewer than half the cache nodes healthy, a cache tier failing its checks, or transit loss confirmed by external probes for three minutes. Planned withdrawals drain first, with Envoy sending GOAWAY and closing QUIC connections and in-flight requests given 120 seconds. pop-guard never withdraws if its own external probes cannot reach the anycast prefix through at least two other PoPs. Re-announcement requires 15 clean minutes and a reconciled PoP (ADR-15).

How it works on-premise

pop-guard is a small Go service on both L4 hosts that controls BIRD's announcements through its control socket. Evidence comes from local node health, BGP session state on the SONiC switches and external probe results fetched over the out-of-band network. Every withdrawal and suppressed withdrawal is logged and paged.

Options weighed

• ChosenSustained evidence, drain, never the last PoP: Removes broken PoPs, avoids flapping, and cannot cascade into a global withdrawal.
• RejectedWithdraw on any failed health check: Fast, and the known route to a self-inflicted global outage.
• RejectedManual withdrawal only: Safe from cascades, slow for real failures at night.
• RejectedCentral controller decides withdrawals: Puts the control plane on the availability path.

Consequences

What it buys

• Broken PoPs stop taking traffic in minutes
• Planned work breaks few sessions
• A shared fault cannot withdraw every PoP

What it costs

• Three minutes of degraded service before automatic withdrawal
• External probes become part of the decision
• The never-last rule can keep a bad PoP announced during a wide incident, which pages instead

Choose differently when

Platforms with many more PoPs per region and very short sessions can tolerate faster, more aggressive withdrawal.

Why it holds up over time

Guarding automation against acting on shared bad evidence is a lesson from well-known internet-scale outages and applies to any future steering mechanism. The rules are expressed as policy, independent of BIRD or Katran.

How much capacity does the edge carry, and how is a cold PoP kept from hurting the shield?

Context

Losing the largest PoP in a region must not push its neighbours over their ceilings, and the edge must reach twice the assumed peak by adding nodes. Anycast does not split load evenly: the busiest PoP often carries two to three times the average share. A PoP that returns empty pulls hard from the shield until it warms.

Decision

Each region is sized so that its largest PoP can be withdrawn at regional peak without any neighbour exceeding 80% of its measured ceiling. Per-node ceilings are measured on each hardware generation under TLS load and are the only figures capacity plans may use. A PoP starting cold caps its shield fetch concurrency at 25% of steady state until its byte hit ratio passes 80%. The cost of n−1 headroom is reported separately every month.

How it works on-premise

Planning uses 20 Gbps per edge node until the proof phase measures the real figure. Tier-1 PoPs have 6 edge nodes and edge-only PoPs have 3, which gives node-level n−1 inside every PoP as well. Cold mode is a flag in the PoP's local state that sets Traffic Server parent connection limits.

Options weighed

• ChosenRegional n−1 from measured ceilings, capped refill: Survives the loss that is certain to happen, and the refill that follows it.
• RejectedGlobal n−1 only: Cheaper, but anycast shifts load regionally, so a regional loss can still overload neighbours.
• Rejectedn−2 per region: Safer and roughly doubles headroom cost for an event the organisation has not required.
• DeferredOverflow to a commercial CDN instead of headroom: Possibly cheaper at true peaks, but requires purge parity across two platforms (ADR-29).

Consequences

What it buys

• A PoP loss is a latency event, not a capacity event
• Capacity plans rest on measured numbers
• Refill never becomes a shield incident

What it costs

• Significant idle capacity, reported as insurance
• Measurements needed per hardware generation
• Cold PoPs take hours to reach full efficiency

Choose differently when

With a contracted commercial overflow path that shares purge semantics, headroom can be lowered to what the overflow does not cover.

Why it holds up over time

Capacity based on measured ceilings and explicit failure assumptions stays correct as hardware changes. The planning rule does not change when nodes get faster; only the measured numbers do.

Where is configuration authoritative, and how does a PoP keep serving without the control plane?

Context

Configuration must be versioned, reviewed, auditable and never edited on a PoP. PoPs must serve on last-known-good configuration indefinitely while the control plane is unreachable, and accept configuration from no other source. A PoP in a third-party facility must not trust anything it downloads just because it came from the expected address.

Decision

Property and steering configuration lives in Git. On merge, a compiler produces an immutable bundle per PoP containing Envoy, Traffic Server and keyspec configuration, signs it and stores it by content hash. PoPs pull bundles when notified, verify the signature against a public key built into the node image, and switch atomically. Each PoP keeps its last five bundles on disk. With no control plane, a PoP keeps serving its current bundle and accepts nothing new.

How it works on-premise

Forgejo hosts the repositories, with signed commits and protected branches. Bundles are signed with cosign using a key held in OpenBao Transit and stored in a Ceph RGW bucket. The PoP agent is notified over NATS, downloads over HTTPS, verifies, writes to a new directory, switches a symlink and triggers an Envoy file-based configuration reload and a Traffic Server reload.

Options weighed

• ChosenGit plus signed, content-addressed bundles pulled by PoPs: Reviewable changes, cryptographic proof of origin at the PoP, and instant revert from local disk.
• RejectedConfiguration API writing to a database, pushed live over xDS: Convenient, but live-streamed configuration disappears when the stream breaks, and reviews happen in a web form.
• RejectedConfiguration management tools editing files on nodes: Drift, partial application and no single artefact to revert to.
• RejectedUnsigned bundles over mTLS: Trusts the transport rather than the content; a compromised bucket could push configuration.

Consequences

What it buys

• Every change is a reviewed commit
• A PoP cannot be pushed configuration by anyone except the signer
• Revert does not need the network

What it costs

• Owners use Git, not a web console
• The signing key is a crown jewel
• Compiler bugs affect every PoP at once, which is why rings exist

Choose differently when

A platform with one operator team and a few properties can reasonably drive xDS directly from a small control plane and skip bundles.

Why it holds up over time

Declarative configuration in version control, compiled to signed artefacts, is how critical infrastructure has come to be managed and does not depend on Envoy or Traffic Server. The same pipeline can target another engine or a vendor's API.

How does a validated change reach 18 PoPs without a bad one reaching all of them?

Context

Validation catches known mistake classes; production catches the rest. The requirement asks for canary, region and global rings with automatic halt within 30 seconds of a regression and revert within 60 seconds. A rollout lasts tens of minutes and must survive a control-plane restart partway through.

Decision

Every bundle rolls out through three rings: one canary PoP for 10 minutes, one continent for 20 minutes, then global. At each stage the ring controller compares 5xx rate, hit ratio per path pattern and stale-served ratio against the same PoPs a week earlier. A regression beyond thresholds halts the rollout and reverts affected PoPs to the previous bundle. Emergency reverts skip bakes. Software upgrades follow the same rings.

How it works on-premise

The ring controller is a Temporal workflow with durable timers and activities that notify PoPs and query Thanos and ClickHouse. Temporal runs on the core Kubernetes cluster with PostgreSQL persistence. Ring membership comes from NetBox; the canary is WAW.

Options weighed

• ChosenTemporal workflow with metric-gated rings: Durable state across restarts, explicit timers, readable workflow code.
• RejectedCI pipeline stages with sleeps: Common, but loses state on runner failure and handles halts badly.
• RejectedArgo Rollouts or Flagger: Built for Kubernetes workloads, and PoPs are not Kubernetes.
• RejectedGlobal push with fast revert: Simpler, and exposes every PoP to every bad change.

Consequences

What it buys

• A bad change reaches one PoP for minutes, not the world
• Halts need no human
• Rollouts survive control-plane restarts

What it costs

• Standard changes take about 30 minutes end to end
• Thresholds need tuning to avoid false halts
• Temporal is another system to run, with plain workers on PostgreSQL as the fallback

Choose differently when

Emergency security changes, such as blocking a path under active abuse, use the revert-speed path and skip bakes by design.

Why it holds up over time

Progressive delivery with automatic halt applies to any configuration or software on any fleet. The ring definitions and thresholds are data, and the workflow can move to another durable engine without changing them.

Where does each part of the platform run, and what does it depend on?

Context

The requirement puts the control plane, logs and origins in a public cloud; this design is fully on-premises. PoP nodes are in third-party facilities with remote hands only. Control plane targets are 99.9% for configuration and 99.95% for purge, with configuration RPO 5 minutes, RTO 30 minutes and invalidation RPO 0.

Decision

PoP nodes boot a signed, read-only A/B image with systemd services and no scheduler, provisioned remotely and attested through their TPM. The control plane runs on Kubernetes: active in Core DC East, warm in Core DC EU. JetStream spans both data centres and a witness in DFW. PostgreSQL uses a synchronous pair in DC East and a streaming standby in DC EU.

How it works on-premise

Node images are built with mkosi, signed, and provisioned by Tinkerbell over the out-of-band network, using NetBox as the inventory. Core clusters run RKE2 with Argo CD. Patroni manages PostgreSQL. Ceph RGW in each data centre stores bundles and cold logs.

Options weighed

• ChosenBare-metal image at PoPs, Kubernetes in the core: Fewest moving parts where nobody is on site, and standard orchestration where operators are.
• RejectedKubernetes at every PoP: Uniform tooling, but adds a scheduler and control plane that can decide not to run the cache.
• Right elsewhereControl plane in a public cloud, as the requirement states: Sound for an organisation with cloud operations; this organisation asked for fully on-premises.
• RejectedActive-active control plane across the Atlantic: Cross-ocean consensus latency on every write for a plane that is allowed 99.9%.

Consequences

What it buys

• PoP nodes have nothing to drift and little to fail
• Control plane recovers within targets without cross-ocean synchronous writes
• Remote provisioning without site visits

What it costs

• Two operating models: image-based PoPs and Kubernetes core
• Warm standby needs regular failover drills
• Image builds become part of every software change

Choose differently when

An organisation with mature cloud operations and no on-premises requirement should put the control plane in a cloud region pair and keep PoPs as they are.

Why it holds up over time

Immutable images and a small core cluster are replaceable separately: a new OS, a new orchestrator or a move of the core to a cloud does not change the PoPs. Because the serving path does not depend on the core, the core can be rebuilt without an edge outage.

Where does a TLS private key live in a facility the organisation does not own?

Context

Keys on PoP disks are simple and fast, and a stolen server is a stolen key. HSMs raise the bar and complicate provisioning at 18 sites. Keyless termination keeps keys in a trusted site at the cost of a round trip in every full handshake and a new dependency. Delegated credentials are not supported by enough browsers. The PoP must also survive seven days without the control plane.

Decision

Each PoP gets its own key pairs and certificates for the properties it serves. Certificates last 14 days and are renewed at day 5, so every PoP always holds at least 9 days of validity. Keys are sealed to the node's TPM on disk and decrypted only into memory. A compromised PoP has its node identities revoked and its own certificates revoked; no other PoP is affected. Keyless termination is Phase 3, for properties whose threat model justifies the latency.

How it works on-premise

A certificate issuer in the core creates per-PoP keys in OpenBao, obtains certificates from the public ACME CA with DNS-01 challenges against PowerDNS, and stores key and certificate for delivery. Node agents authenticate with SPIRE identities after TPM attestation, fetch response-wrapped keys, seal them to the TPM and hand them to Envoy through SDS. Session ticket keys are rotated hourly per PoP.

Options weighed

• ChosenPer-PoP 14-day certificates with TPM-sealed keys: Loss is bounded in time and to one PoP, with no handshake latency cost and seven-day autonomy.
• RejectedShared long-lived keys on PoP disks: One stolen server means reissuing every PoP's certificates.
• RejectedHSMs at every PoP: Strong, but expensive and slow to provision at 18 sites, and handshake throughput needs sizing.
• DeferredKeyless termination for all properties: Keeps keys off PoPs, but adds a round trip to every full handshake and a dependency on the key servers.
• RejectedDelegated credentials (RFC 9345): Ideal in principle, not supported by enough browsers.

Consequences

What it buys

• A stolen PoP is revoked alone
• Stolen keys expire within days
• No latency cost on handshakes

What it costs

• Certificate issuance volume is 18 times higher and must be contracted with the CA
• Renewal every 5 days makes issuance a production dependency
• A running server stolen whole is still useful for up to 14 days

Choose differently when

For a property whose compromise would be catastrophic, such as a login domain, keyless termination from the core is worth the latency. For PoPs in facilities with strong physical controls, longer certificates are acceptable.

Why it holds up over time

Certificate lifetimes across the web are getting shorter, so this design moves in the same direction as the ecosystem. Per-site keys, short lifetimes and hardware sealing do not depend on a particular CA or TLS library.

Who can change what, how is a PoP's cache destroyed, and how is every change recorded beyond tampering?

Context

Changing configuration, purging and reading delivery logs are different powers with different abuse potential. Purge is the easiest to abuse, because a wildcard purge is a self-inflicted origin load event. Cached content on PoP drives in third-party facilities must be protected at rest and destroyable on decommission. Every purge, configuration change and administrative access must be logged somewhere the writer cannot delete from.

Decision

Keycloak groups map to three roles: config-change, purge and log-read, held by different groups. Wildcard purge needs purge-admin in addition, and is limited to one per property per hour. NVMe caches are LUKS2 volumes with keys sealed to the TPM; decommissioning destroys the sealed keys. Audit events are hash-chained and written to a Ceph RGW bucket with Object Lock in compliance mode for three years, by a role that can put but never delete.

How it works on-premise

Operators sign in to Keycloak with WebAuthn. The purge API and Forgejo check role claims. The audit writer is a small service with a Ceph STS role allowing PutObject only; the daily chain head is also written to the Forgejo audit repository, so tampering in either place is detectable from the other.

Options weighed

• ChosenSeparate roles, sealed disk encryption, Object Lock audit: Limits each role's damage, makes decommission fast and certain, and keeps an audit record that survives a compromised writer.
• RejectedOne admin role for the platform team: Simple, and gives every engineer the power to cause a global origin incident.
• RejectedNo encryption at rest for cached public content: Caches hold authenticated responses keyed by identity, which are not public.
• RejectedAudit in a normal database table: Anyone with database admin rights can edit history.

Consequences

What it buys

• No single role can both change and hide
• A decommissioned PoP's cache is unreadable in seconds
• Audit evidence survives insider tampering

What it costs

• More roles to administer
• LUKS2 costs a little CPU on NVMe I/O
• Object Lock storage is kept for three years whatever it costs

Choose differently when

A small platform run by one trusted team may combine config and purge roles, but should still keep audit write-once.

Why it holds up over time

Least privilege, disk encryption with hardware-bound keys and write-once audit are controls that auditors will keep asking for, and each uses a standard mechanism with several implementations.

How are one structured record per request delivered centrally without ever slowing delivery?

Context

At steady state the edge produces about 7.8 billion access records a day. They must arrive at least once within a declared delay when the pipeline is healthy, but delivery must never slow when it is not, and the gap must be counted. Hit ratio, offload and cost per path pattern are computed from these records.

Decision

Traffic Server and Envoy write structured logs locally. Vector on each node ships them with a 20 GB disk buffer and drops, counting, when the buffer is full. Kafka in each core data centre buffers three days. ClickHouse keeps 30 days hot and moves older partitions to Parquet on Ceph for 400 days. Records carry a key hash rather than the key, and client addresses are reduced to country and ASN.

How it works on-premise

Vector's disk buffer is set to drop newest when full, with a counter exported to Prometheus. Kafka runs three brokers per core data centre. ClickHouse runs six nodes, three per data centre, with ReplicatedMergeTree and ClickHouse Keeper, reading from Kafka through the Kafka table engine. Parquet export uses ClickHouse's S3 table functions against Ceph RGW.

Options weighed

• ChosenVector, Kafka, ClickHouse with drop-and-count: Bounded local impact, replayable central buffer, and fast analytical queries at this volume.
• RejectedShip directly from nodes to ClickHouse: Fewer parts, but an analytics outage means lost data or node buffers filling up.
• RejectedElasticsearch or OpenSearch for logs: Good for search, several times more expensive to store for aggregate analytics at this volume.
• Right elsewhereFluent Bit instead of Vector: CNCF-governed and lighter; the named fallback if Vector's single-company stewardship becomes a problem.

Consequences

What it buys

• Delivery never waits on logging
• Gaps are measured, not guessed
• Per-path analytics at billions of rows a day

What it costs

• Three systems in the pipeline
• About 28 TB hot and 190 TB cold to keep
• Records dropped in long outages are gone

Choose differently when

For billing that must be exact, bill from per-node counters, which are never dropped, and use logs for analysis only. This design already bills bytes from counters.

Why it holds up over time

Separating shipping, buffering and analysis lets each piece change without the others, and the record schema is independent of all three. Parquet on object storage remains readable by almost any analytics engine.

How does the platform know it is failing users it cannot see?

Context

A PoP that is withdrawn, black-holed by a transit provider or unreachable from some networks looks perfectly healthy from inside: no errors, no latency, just fewer requests. The requirement sets 99.99% delivery availability measured from outside, and asks for alerts on absence of traffic as well as on errors.

Decision

Probes in 12 external networks request a test object from every property through every PoP's unicast test address and through the anycast address. Their success rate is the availability SLI. Each PoP's request rate is compared with the same hour a week earlier, and a rate below 20% of it pages. Metrics are collected by Prometheus in every PoP, with Thanos for global queries and 13-month retention.

How it works on-premise

Probe hosts are small virtual machines in colocation facilities and eyeball networks that the platform does not use for transit, running blackbox_exporter and reporting to a Prometheus instance outside the PoPs. Alertmanager routes pages. Grafana shows SLO burn rates per property and PoP.

Options weighed

• ChosenExternal probes plus absence alerts: Sees the failures that produce no errors, and measures what users experience.
• RejectedInternal success rate as the SLI: Cannot see requests that never arrive.
• Right elsewhereCommercial synthetic monitoring service: Wider coverage without running probe hosts; acceptable as a supplement if the fully on-premises constraint allows an outside service.
• DeferredReal-user monitoring beacons only: Valuable for experience data, but depends on each property's front end.

Consequences

What it buys

• The SLI matches what users see
• Silent failures page
• Probe results also feed pop-guard's evidence

What it costs

• Probe hosts to run in other networks
• Twelve vantage points miss some networks
• Absence thresholds need adjusting around holidays and events

Choose differently when

A platform serving only internal users on one network can measure internally and skip external probes.

Why it holds up over time

Outside-in measurement and alerting on silence apply to any service at any scale, and do not depend on the monitoring products chosen.

How is the platform's value measured, and is the stated cost target achievable?

Context

The requirement targets $4 per delivered terabyte at 6 PB a month and origin bytes at or below 1.5%. The stated figures conflict: 6 PB a month averages about 18.5 Gbps, which puts the 400 Gbps peak at about 22 times the average. At 6 PB a month the fixed cost of 18 self-operated PoPs is around $30 per TB. The target is realistic only near 45 PB a month with half the bytes on settlement-free peering.

Decision

Cost per delivered terabyte is published monthly per property and PoP, decomposed into transit and peering, hardware amortisation, facility, origin egress and control plane. The cost of a miss (origin egress plus shield transit plus added latency) and origin offload per path pattern are first-class reports, and properties above 1.5% origin bytes are named with the patterns responsible. Hardware beyond the six tier-1 PoPs is ordered only after the proof phase has measured real monthly volume and peak ratio.

How it works on-premise

ClickHouse joins delivered and origin bytes with cost inputs: transit invoices per PoP, hardware amortisation from NetBox asset records over five years, colocation contracts and a control-plane allocation. Monthly results export to the finance ledger for chargeback. Marginal hit ratio per added terabyte is estimated from eviction age data per volume.

Options weighed

• ChosenCost and offload as published metrics, with a volume gate: Makes hit-ratio regressions visible as money and avoids buying 18 PoPs for traffic that may not exist.
• RejectedBuild all 18 PoPs on the stated assumptions: Commits several million in hardware to figures that contradict each other.
• RejectedReport bytes served and hit ratio only: Performance numbers that do not show what a regression costs.
• RejectedChargeback on requests only: Ignores bytes and misses, which drive cost.

Consequences

What it buys

• A key regression shows up as money
• Owners can see what their misses cost
• Investment follows measured demand

What it costs

• Cost models need finance input and upkeep
• Chargeback invites arguments, which is intended
• Scale-out waits on the proof phase

Choose differently when

If measured volume is well above the stated figure, skip the gate and build to the roadmap; if it is near 6 PB, see ADR-29.

Why it holds up over time

Unit economics expressed per terabyte and per miss let the organisation compare this platform with any alternative at any time, including buying delivery. The metrics outlast every technology choice here.

Is this the only delivery path, or one of several?

Context

A single self-operated platform gives full control of keys, purge and cost, and its worst day is the organisation's worst day. A commercial CDN alongside adds peak headroom and reach, but splits invalidation across two systems with different purge semantics. A second path used rarely is not warm enough to be trusted when it is needed.

Decision

The MVP has one delivery path. Everything that would need to move is kept portable: engine-neutral property declarations, surrogate keys, versioned URLs and Cache-Status. A second path is added only when purge can be expressed identically on both and acknowledged from both, and it must carry a constant share of traffic to stay warm. If the proof phase measures volume near 6 PB a month, buying delivery and keeping only the key contract and purge semantics in-house is a valid outcome.

How it works on-premise

The compiler's back-end interface is designed so a vendor configuration back end can be added. The purge API's record format includes a target path field from the start. Steering intent (ADR-18) can express a share of traffic per path, which a second CDN would need.

Options weighed

• ChosenSingle path now, portable semantics, conditional second path: Full control while the platform proves itself, without closing the door on buying or blending.
• RejectedSelf-operated plus commercial CDN from day one: Headroom and reach, but two purge semantics and a second path that is cold most of the time.
• Right elsewhereCommercial CDN only: Likely cheaper at 6 PB a month; the organisation asked for self-operated open source, and the volume check decides.
• RejectedSelf-operated only, forever: Ignores the chance that measured economics favour buying.

Consequences

What it buys

• One set of semantics to get right first
• No lock-in to either model
• The build-or-buy decision is made on data

What it costs

• No external overflow for extreme peaks at launch
• Portability features cost some design effort now
• A future second path still needs integration work

Choose differently when

If one outage of the self-operated platform would cost more than a year of a commercial contract, run a second path from the start and invest in purge parity first.

Why it holds up over time

Keeping the valuable parts portable, meaning the key contract, invalidation semantics and steering intent, lets the organisation change delivery models as prices and scale change without redesigning how properties describe their content.