Architecture Decision Record

Edge Cache and CDN Platform · Solution Architecture v1.0 · Edge Architecture · 2026-09 · 29 views · open source, fully on-premises

The argument these decisions serve is summarised in the Architecture One-Pager.

Twenty-nine decisions make up this architecture. Everything else on the twenty-nine views is convention, and convention needs no defence. Each record starts with the question that forced a decision, says what was chosen and how it is built on hardware the organisation owns, lists the options that lost and why, and states what the choice costs. It then gives the conditions under which a different organisation should choose differently, and explains why the choice should still be right after the products, the hardware and the team have all changed. Read the one-pager first. Read a record when you want to argue with it.

Status of this document. This is a design, not a report on a running system. Latency, throughput, hit-ratio and cost figures are targets and stated assumptions taken from the requirement or from planning prices, and each is replaced by a measured value in the proof phase. Five product behaviours are relied on and must be confirmed before production depends on them: Traffic Server next-hop strategies in peering-ring mode for intra-PoP ownership; a Traffic Server plugin overriding the cache lookup status after the lookup completes, which the tag table needs; Envoy's QUIC-LB connection ID generator working with Katran's QUIC server-ID routing; NATS JetStream mirrors on leaf nodes replaying correctly after a 14-day disconnection; and the public ACME CA's issuance limits accommodating per-PoP certificates renewed every five days.

How to read a record

• Question: The forcing question: why a decision was needed at all.
• Context: The requirement, the scale and the constraint that make it hard.
• Decision: What this architecture does, stated so it can be checked.
• How it works on-premise: The concrete mechanism: which package, configured how, on whose hardware.
• Options weighed: Chosen, rejected, deferred, or right elsewhere, with the reason for each.
• Consequences: What the choice buys and what it costs, both kept visible.
• Choose differently when: The conditions that would flip the decision for your system.
• Why it holds up over time: What keeps the decision right as scale, staff and technology change.
• Lesson: The principle that transfers beyond this platform.

Decision map

The cache key: What a cache hit is allowed to mean, who decides it, and how the platform stops a wrong decision from reaching users.

• ADR-01 · The cache key is declared per path pattern, and nothing is cacheable until declared
• ADR-02 · One Rust library computes the key in the cache, the validator and the explain API
• ADR-03 · Private responses are refused whatever the rule says, and an unexpected Vary is a configuration error
• ADR-04 · Validation proves identity is in the key with invariants and a two-user probe

Serving path: The software on a PoP node, how a PoP behaves as one cache, how tenants share it, and what gets served when things go wrong.

• ADR-05 · Apache Traffic Server is the cache engine
• ADR-06 · Envoy terminates TLS and HTTP/3 in a separate process from the cache
• ADR-07 · Inside a PoP, every node keeps its own hot set and the long tail has one owner on a peering ring
• ADR-08 · Memory and NVMe tiers, with per-property cache volumes that set a floor and a ceiling
• ADR-09 · Freshness is declared per path as strict, standard or resilient, and overload sheds in a fixed order

Shield and origins: Where misses go, how an origin is protected from the edge, and how an origin tells the platform apart from everyone else.

• ADR-10 · Shield pools sit next to the origins, in continental pairs, with each key converging on one shield node
• ADR-11 · Origin protection is enforced at the shield, slow is distinguished from down, and the direct path is capped
• ADR-12 · Origins authenticate the platform with mTLS from workload identities

Invalidation: How content changes everywhere within seconds, how the platform knows it did, and when not to purge at all.

• ADR-13 · Invalidation compares timestamps at lookup instead of deleting objects
• ADR-14 · NATS JetStream is the invalidation log, mirrored into every PoP, with acknowledgement per PoP
• ADR-15 · A returning PoP reconciles before serving, and this ships in the MVP
• ADR-16 · Versioned URLs first, purge second, short TTLs last, with the strategy declared per path

Topology and steering: How users reach a PoP, how traffic is moved, when a PoP removes itself, and how much spare capacity is carried.

• ADR-17 · Anycast from every PoP, with Katran on dedicated hosts for QUIC-aware L4 balancing
• ADR-18 · Four anycast prefix classes for fine steering, and DNS as a manual per-property override from one intent
• ADR-19 · Automatic withdrawal needs sustained evidence, drains first, and never removes the last PoP
• ADR-20 · Capacity is planned to n−1 per region from measured per-node ceilings, and cold PoPs refill under a cap

Configuration and control: Where configuration is authoritative, how it reaches 18 sites safely, and where the control plane runs.

• ADR-21 · Git is the configuration authority, and PoPs apply only signed bundles and keep the last known good
• ADR-22 · Rollout is a Temporal workflow through canary, region and global rings, halting on regressions without a person
• ADR-23 · PoPs run a signed image on bare metal; the control plane runs on Kubernetes in two core data centres and a witness

Security: Private keys in buildings the organisation does not own, separated privileges, and a record nobody can quietly edit.

• ADR-24 · Each PoP has its own short-lived certificates, with keys sealed to its TPM; keyless termination only later and only for a few properties
• ADR-25 · Three separated privileges, encrypted caches that can be destroyed by key, and an audit trail its writer cannot delete

Telemetry and economics: What is measured, from where, what it costs, and whether running the edge is the right call at all.

• ADR-26 · Access records are droppable: Vector at the node, Kafka in the core, ClickHouse for analysis
• ADR-27 · Availability is measured from outside, and silence at a PoP is an alert
• ADR-28 · Offload and the cost of a miss are financial metrics, and the traffic assumptions are checked before scaling out
• ADR-29 · One self-operated delivery path for now, with configuration and purge semantics kept portable

Technology by capability

Every capability on the views, the package that provides it, a credible alternative, and the record that justifies the choice. Everything runs on hardware the organisation owns, with no managed service and no external dependency on the request path.

The decisions, and the alternatives that lost

The cache key

What a cache hit is allowed to mean, who decides it, and how the platform stops a wrong decision from reaching users.

ADR-01 · The cache key is declared per path pattern, and nothing is cacheable until declared

Status: Accepted · Shown on views: 04, 12, 13

Who decides what makes two requests the same request?

Context. A key of scheme, host and path gives the best hit ratio and is wrong for anything that varies. A key built from the origin's Vary header is correct only if forty origin teams always send the right headers, and it hands cache fragmentation, and therefore cost, to whoever writes those headers. The catastrophic failure of a CDN is not a miss or an outage but a hit on the wrong key, which produces no error anywhere.

Decision. Each path pattern declares its key as an explicit list: scheme, host, path, a named allow-list of query parameters, named request headers, an optional device or geography class, and an identity field that is either none or the header carrying identity. Anything not listed never enters the key. A path with no declaration is not cacheable. The platform applies its own maximum TTL per property and a 30-day ceiling overall, whatever the origin sends.

How it works on-premise. Owners write the declaration as YAML in the property's directory in Forgejo. The CUE schema requires the identity field on every cacheable rule, with no default. The compiler turns rules into a compact table that the keyspec plugin in Traffic Server loads from the signed bundle, indexed by the rule ID that Envoy stamps on the request after matching the property.

What it buys

• A cache hit means what the owner declared it means
• Keys can be reviewed, diffed and tested before traffic sees them
• Fragmentation has a named owner and a visible cost

What it costs

• Lower hit ratio than a permissive platform, visible in cost per TB
• Onboarding requires every owner to do the work before getting any benefit
• Owners need an explain tool and good defaults to succeed

Choose differently when. A single-tenant platform where the same team owns every origin and the edge, and has a tested Vary discipline, can safely let the origin drive the key and take the higher hit ratio.

Why it holds up over time. What makes two requests equivalent is a property of the content, not of the cache engine, so the declaration survives any change of engine, hardware or provider. If the organisation later buys its delivery, the declarations become the specification the vendor is configured against.

Lesson. When a mistake is silent and irreversible, make the risky choice explicit and opt-in rather than inferred.

ADR-02 · One Rust library computes the key in the cache, the validator and the explain API

Status: Accepted · Shown on views: 09, 13

How does an owner check a key and trust that production computes the same one?

Context. A key is computed in at least three places: the cache on every request, the validator before release, and a tool owners use to understand their configuration. If each has its own implementation, they drift. Percent-encoding, parameter ordering and header folding are exactly the details where two implementations disagree, and the disagreement is invisible until a user reports someone else's content.

Decision. Key computation, normalisation and the refusal rules live in one library, keyspec, with a C interface. The Traffic Server plugin, the validator and the explain API all link the same build. Its behaviour is fixed by a corpus of real requests and expected keys that must pass before any bundle is built.

How it works on-premise. keyspec is written in Rust with no allocation on the request path and exposes a small C ABI that the Traffic Server plugin calls. The explain API runs as a service in the core and on an internal-only listener at each PoP, so an owner can ask a PoP running a specific bundle version what key it would compute. Hashes use BLAKE3 truncated to 128 bits.

What it buys

• The explain tool is correct by construction
• A regression in normalisation fails CI, not production
• Engine changes keep identical key semantics

What it costs

• A Rust toolchain and FFI boundary to maintain
• Plugin upgrades must be tested against each Traffic Server major version
• Owners must accept a truncated hash in logs rather than the key itself

Choose differently when. If the platform uses only a managed CDN's key configuration, there is no plugin to share and the corpus becomes a conformance test run against the vendor's staging endpoint instead.

Why it holds up over time. The library and its corpus hold the platform's most important behaviour in a form that outlives the engine around it. Rust with a C ABI can be linked from any likely successor to Traffic Server.

Lesson. When a rule has to hold in several places, build it once and link it everywhere; do not rely on keeping several copies in step.

ADR-03 · Private responses are refused whatever the rule says, and an unexpected Vary is a configuration error

Status: Accepted · Shown on views: 13, 29

What happens when the origin's response contradicts the owner's declaration?

Context. Declarations are made in advance; responses arrive at run time. An origin deploy can start setting a session cookie on a path declared cacheable, or send Vary: Cookie on a path whose key excludes cookies. Honouring the Vary silently fragments the cache; ignoring it can leak. Both hide the contradiction from the owner.

Decision. A response with Cache-Control private or no-store, a session Set-Cookie, or credentials is never stored, regardless of the rule. A Vary naming a header outside the declared key stops the object being stored and raises an owner alert with the path pattern. Accept-Encoding is the one normalised exception, stored as up to three declared variants. Every refusal is counted per path pattern.

How it works on-premise. The keyspec plugin checks responses at the Traffic Server read-response hook and marks refusals as uncacheable. Refusal counters go to Prometheus, and the access record carries a refusal reason. Alertmanager routes Vary-mismatch alerts to the owning team using the property's owner metadata.

What it buys

• Origin drift cannot turn into a leak
• Refusals are measurable per path
• Owners learn about header bugs from an alert, not an incident

What it costs

• A bad origin deploy drops the hit ratio on that path until fixed
• Alert noise if an owner ignores refusals
• Some legitimate Vary use must be declared explicitly

Choose differently when. For a public, anonymous-only property with no cookies at all, an organisation might accept honouring Vary to save owners effort. It remains unsafe the day anyone adds authentication.

Why it holds up over time. HTTP will keep letting origins say one thing and configuration say another. Refusing on contradiction is a rule about trust, not about a product, and it applies equally to any future engine or vendor.

Lesson. When two sources of truth disagree on a safety question, fail closed and tell the owner; do not quietly pick one.

ADR-04 · Validation proves identity is in the key with invariants and a two-user probe

Status: Accepted · Shown on views: 04, 19, 23

How does the platform reject a change that would make a user-specific path cacheable without the user in the key?

Context. Schema validation catches malformed configuration but not a well-formed wrong one. The requirement asks for a validator that rejects any change making an identity-varying path cacheable under a key that omits identity. The platform cannot know from configuration alone which paths vary by identity; it has to either trust the owner's declaration or check the origin.

Decision. Three layers run on every merge request. CUE checks types, required fields and defaults. Invariant checks enforce cross-rule rules, for example that a rule marked identity-varying names its identity header, and that no cacheable rule overlaps a pattern another rule marks private. A differential probe fetches each rule's declared test URLs from the origin as two test users and fails the change if the bodies differ while the key omits identity.

How it works on-premise. Forgejo Actions runners in the core run CUE and the invariant checks, written in Go against the compiled rule table. The probe runs from the core through the origin's mTLS path using two test identities per property held in OpenBao, and ignores declared volatile fields such as timestamps and CSRF tokens. CUE also requires at least three test URLs per cacheable rule.

What it buys

• The most dangerous class of change is caught before release
• Owners get a failing check that names the rule
• Test identities make leaks reproducible

What it costs

• Owners must maintain test URLs and identities
• Probe coverage is only as good as the declared URLs
• Volatile fields need declaring or the probe is noisy

Choose differently when. If properties are entirely anonymous, such as software downloads or public media, the probe adds little and schema plus invariants are enough.

Why it holds up over time. Pre-release checks tied to observable origin behaviour do not depend on the engine, and the probe is the same whether the edge is self-operated or bought. As origins change over years, the probe keeps testing what they do now, not what they did at onboarding.

Lesson. Validate what the system will actually do, not only what the configuration says it will do.

Serving path

The software on a PoP node, how a PoP behaves as one cache, how tenants share it, and what gets served when things go wrong.

ADR-05 · Apache Traffic Server is the cache engine

Status: Accepted · Shown on views: 07, 08, 09

Which open-source cache engine can meet the key, storage and shielding requirements on hardware we run?

Context. The requirement names Traffic Server or Varnish. The engine must keep a cache across restarts so a node rejoins warm, stream in-flight fetches to waiters, serve ranges from partially cached large objects, route misses by consistent hash to parents, and support a plugin that can override lookup results. It will hold hundreds of terabytes per PoP on NVMe.

Decision. Apache Traffic Server 10 on every edge and shield node, with a raw-device NVMe cache that persists across restarts, a RAM cache for the hot set, read-while-writer collapsing, the slice and cache_range_requests plugins for large objects, next-hop strategies for peers and shields, stale_response for stale-while-revalidate and stale-if-error, and three platform plugins: keyspec, tagguard and cache-status.

How it works on-premise. Traffic Server runs as a systemd service from the signed node image. Cache spans are the raw NVMe devices opened through LUKS2 mappings. Configuration files (remap, strategies, volumes, plugin tables) come only from the active bundle directory, and changes are applied with a configuration reload, not a restart.

What it buys

• Node restarts and upgrades rejoin warm
• Shield routing and intra-PoP ownership use built-in strategies
• One engine for edge and shield roles

What it costs

• Smaller community and fewer engineers who know it than NGINX or Varnish
• Plugin code in C++ against a changing API
• Configuration is spread across several files, which the compiler has to own

Choose differently when. A platform mostly caching APIs with small objects and frequent logic changes, and where warm restarts matter less, may prefer Varnish for VCL's expressiveness.

Why it holds up over time. Traffic Server is chosen for capabilities, not its configuration language, and the compiler (ADR-21) and keyspec (ADR-02) keep the rest of the platform independent of it. If it stops being the right engine, a replacement is a new compiler back end and a plugin port, rolled out one ring at a time.

Lesson. Choose the engine for the properties that are hardest to add later, such as persistence and parent routing, not for the ones that are easy to configure.

ADR-06 · Envoy terminates TLS and HTTP/3 in a separate process from the cache

Status: Accepted · Shown on views: 08, 09, 27

Where do TLS termination, token checks and per-property limits run?

Context. The edge must terminate HTTP/1.1, HTTP/2 and HTTP/3 with TLS 1.2 and 1.3 and per-property certificates, verify signed URLs before lookup, enforce per-property rate, connection and bandwidth ceilings, and drain gracefully. Private keys sit in facilities the organisation does not control, and the process that parses untrusted origin responses is the one most likely to have a memory-safety bug.

Decision. Envoy runs on every cache node in front of Traffic Server, connected over loopback. It holds TLS keys and does termination, HTTP/3, SNI-based certificate selection, signed URL verification, token stripping, local rate limits and drain. Traffic Server holds no private keys.

How it works on-premise. Envoy loads listeners, routes and limits from files in the active bundle and receives certificates through SDS from the node agent (ADR-24). Signed URLs follow the CDNI URI Signing shape (RFC 9246) and are verified with the jwt_authn filter against keys in the bundle. Limits use the local rate-limit filter, sized per node from the property ceiling divided by node count. QUIC connection IDs carry a server ID so Katran keeps QUIC flows on the same node.

What it buys

• A cache bug cannot read TLS keys
• Certificates rotate without restarts
• Drain behaviour is precise and testable

What it costs

• Two proxies per node to operate and upgrade
• A loopback hop and some CPU on every request
• Local limits are approximate across nodes

Choose differently when. On small edge nodes where CPU is scarce and physical security is good, letting the cache terminate TLS saves a hop and a process.

Why it holds up over time. Splitting the key-holding terminator from the content-handling cache is a security boundary that holds whatever the products are. Either side can be replaced independently because the seam between them is plain HTTP over loopback.

Lesson. Keep secrets in the smallest process that needs them, and away from the code that parses untrusted input.

ADR-07 · Inside a PoP, every node keeps its own hot set and the long tail has one owner on a peering ring

Status: Accepted · Shown on views: 09, 15, 29

Does every node in a PoP hold everything, or does the PoP shard its cache?

Context. Replicating the working set on every node wastes storage in proportion to node count and makes node loss trivial. Consistent hashing across nodes multiplies effective PoP cache size but adds a hop and turns node changes into redistribution. The working set is around 150 TB of distinct bytes, larger than one node's NVMe, and collapsing only works when requests for one key reach one process (view 15).

Decision. Each node serves its own RAM and NVMe cache. On a local miss, the node asks the key's owner on the PoP's consistent-hash ring before going to the shield. The owner fetches from the shield and both nodes store the object. Popular objects therefore end up on many nodes, and long-tail objects on roughly one.

How it works on-premise. Traffic Server next-hop strategies in peering-ring mode define the PoP's edge nodes as a ring with virtual nodes. The ring is generated from NetBox inventory into each PoP's bundle. Katran health checks remove a failed node from connection balancing, and the ring skips it to the next owner.

What it buys

• Effective PoP cache size close to the sum of its nodes
• Node loss costs 1/N of the tail, never a PoP flush
• Collapsing works across the PoP, not only per node

What it costs

• An extra intra-PoP hop on tail misses, under 1 ms
• Adding or removing a node moves about 1/N of ownership
• Intra-PoP fabric must carry owner traffic

Choose differently when. When a working set fits comfortably on one node, or nodes are replaced very often, full replication is simpler and the extra shield fetches are cheap.

Why it holds up over time. The principle, which is to converge each key on one owner so collapsing works while keeping hot objects local, holds however large disks become. When disks outgrow the working set the ring still helps collapsing and costs almost nothing.

Lesson. Request collapsing only works if the requests for a key arrive at the same process, so plan where they converge.

ADR-08 · Memory and NVMe tiers, with per-property cache volumes that set a floor and a ceiling

Status: Accepted · Shown on views: 09, 11

How does one property's traffic avoid evicting another property's cache?

Context. Forty properties share every PoP. A large property publishing a video catalogue can fill NVMe and evict a smaller property's long tail, dropping its hit ratio without any change of its own. The requirement asks for a per-property footprint floor and ceiling at each PoP and for per-property accounting.

Decision. Each node has a RAM cache for the hot set and NVMe for the long tail. NVMe is divided into Traffic Server cache volumes: large properties get a dedicated volume sized between their declared floor and ceiling, and small properties share a pooled volume with a combined floor. Eviction happens inside a volume, so one property cannot evict another's objects. Per-property request, connection and bandwidth ceilings are enforced by Envoy.

How it works on-premise. Volume layout per PoP is compiled from property declarations and PoP disk inventory into Traffic Server's volume and hosting configuration. Changing a volume layout reinitialises that volume, so it is rolled out one node at a time, like a node restart. Per-volume fill and eviction rates are exported to Prometheus.

What it buys

• A property's hit ratio depends on its own traffic
• Storage cost per property is directly measurable
• Floors protect small, important properties

What it costs

• Total hit ratio slightly below a shared cache
• Resizing a volume costs that slice of cache on each node
• Owners must estimate a footprint, revised quarterly from measurements

Choose differently when. A single-tenant platform, or tenants with no internal chargeback, gets more from one shared cache.

Why it holds up over time. Isolating tenants by storage allocation works with any storage medium and any engine that can partition its cache. The declarations are in engine-neutral configuration, so the policy survives an engine change even if the mechanism differs.

Lesson. In shared caches, isolation is a capacity decision: give each tenant space its neighbours cannot take.

ADR-09 · Freshness is declared per path as strict, standard or resilient, and overload sheds in a fixed order

Status: Accepted · Shown on views: 18, 29

When the origin fails or the PoP is overloaded, what does a user get?

Context. Serving stale content during an origin outage keeps a news site up and would show a wrong balance on a banking page. That is a product decision, not a platform default. The platform also needs a declared order for shedding load, so overload degrades service predictably rather than randomly.

Decision. Each path declares a freshness class. Strict revalidates before serving once stale and returns the declared error page if the origin cannot answer. Standard serves stale while revalidating for a declared window, 60 seconds by default. Resilient also serves stale on origin error, for up to 24 hours. The platform caps all windows whatever the origin sends. Under overload the edge sheds bypass and uncacheable traffic first, then low-priority properties, then revalidation, and cache hits last. Every stale response is labelled in Cache-Status.

How it works on-premise. The compiler turns classes into Traffic Server stale_response plugin settings and revalidation behaviour per rule. Envoy's overload manager and route priorities implement shedding by rule class and property priority. Stale-served ratio per path pattern is computed in ClickHouse and alerted in Prometheus.

What it buys

• Incident behaviour is known before the incident
• Stale content is visible and countable
• Overload hits the cheapest traffic first

What it costs

• Owners must classify every path
• Strict paths show errors during origin outages, by design
• Shedding low-priority properties needs agreed priorities

Choose differently when. A platform serving one kind of content, such as only static software packages, can use one class everywhere.

Why it holds up over time. Stale-while-revalidate and stale-if-error are standard HTTP extensions, and the classes are expressed as policy, not engine settings. Whether wrong or missing is worse is a business question that stays with the owner however the technology changes.

Lesson. Decide in advance whether a wrong answer or no answer is worse, and write it down per path.

Shield and origins

Where misses go, how an origin is protected from the edge, and how an origin tells the platform apart from everyone else.

ADR-10 · Shield pools sit next to the origins, in continental pairs, with each key converging on one shield node

Status: Accepted · Shown on views: 02, 15, 20

What shape should the shield tier be?

Context. One global shield per origin maximises offload and concentrates failure. A shield per region shortens miss paths and multiplies origin fetches by the number of regions. A hashed mid-tier across tier-1 PoPs couples origin protection to PoP membership changes. Origins are in two data centres, one in the eastern US and one in Frankfurt, and origin egress is the largest avoidable cost.

Decision. Four shield pools run on dedicated nodes in four tier-1 sites: IAD and DFW for origins in the eastern data centre, FRA and LHR for origins in Frankfurt. Each origin set declares a primary and a secondary shield on the same continent. Every miss is hashed to one node within the primary shield pool, so an object converges on one node and one fetch. Asia and South America edges use the same shields and accept higher miss latency.

How it works on-premise. Edge strategies use a consistent-hash parent group whose members are the primary pool's nodes, with the secondary pool as the failover group. Shield nodes run the same image as edge nodes with shield role configuration and 16 NVMe drives each. A regional shield pool at SIN is defined in configuration but not enabled.

What it buys

• Origin sees fetches from a small, known set of addresses
• One fetch per object per shield per TTL
• Shield tier scales with miss volume, independently of edges

What it costs

• Asia and South America shield hits exceed the 60 ms budget
• Sixteen dedicated nodes carry no user traffic directly
• Shield failover changes miss latency

Choose differently when. If origins become multi-region, or Asian traffic grows so that its miss share drives p95, move to regional shields and accept one origin fetch per region.

Why it holds up over time. Shield placement follows origin placement. When origins move, shields are reassigned by editing origin sets, not by redesigning the tier. The convergence rule, one key per shield node, holds at any size.

Lesson. Put the shield where the expensive bytes are, which is next to the origin, not next to the users.

ADR-11 · Origin protection is enforced at the shield, slow is distinguished from down, and the direct path is capped

Status: Accepted · Shown on views: 15, 18, 29

How is an origin protected from the edge, including when the shield itself fails?

Context. An origin's protection must not depend on the origin behaving well. A flash crowd, a cold PoP refilling or an expired popular object can all send more requests than an origin can take. When a shield fails, falling back to direct edge-to-origin fetches at full miss volume causes the origin outage the shield exists to prevent.

Decision. Each origin set declares a concurrency and request-rate ceiling, a slow threshold and a down threshold. Shields queue up to the ceiling and then serve stale or shed according to the freshness class. Slow origins are handled by waiting within bounds; down origins trigger ordered endpoint failover. If both shields are unavailable, edges may fetch directly from origins only under a per-origin rate cap of 5% of normal miss volume.

How it works on-premise. Traffic Server per-origin connection limits with queueing enforce concurrency at the shield. Health is judged on error rate and connect failures for down, and on time to first byte against the declared threshold for slow. Endpoint order and thresholds come from the origin set in the bundle. The direct-path cap is an Envoy rate limit on a separate origin cluster that is active only while shields are marked down.

What it buys

• Origin load is bounded and predictable
• Slow origins do not trigger unnecessary failovers
• Shield failure degrades gracefully

What it costs

• Owners must declare realistic ceilings
• Queued requests add latency at peaks
• Capped direct path means some misses fail during double shield failure

Choose differently when. Origins that are themselves large object stores with effectively unlimited capacity may need only generous ceilings, but should still have them.

Why it holds up over time. Protecting a dependency from its callers with ceilings and bounded fallback is standard reliability practice and does not depend on the engine. Thresholds are data in origin sets that owners revise as origins change.

Lesson. Protect a dependency at the point you control, and make every fallback smaller than the failure it replaces.

ADR-12 · Origins authenticate the platform with mTLS from workload identities

Status: Accepted · Shown on views: 10, 27

How can an origin refuse traffic that did not come through the platform?

Context. An origin reachable directly lets attackers skip edge limits, token checks and caching. IP allow-lists break when shields are added and do not prove which service is calling. The requirement asks for mTLS or signed requests from shield to origin.

Decision. Shields call origins over mTLS with a short-lived client certificate that carries the shield's SPIFFE identity. Origins accept only that identity for platform traffic. The edge-to-origin direct path uses a separate, distinguishable identity, so an origin can see and alert on its use.

How it works on-premise. SPIRE issues X.509 SVIDs to shield nodes after TPM attestation, rotated hourly. The SPIRE trust bundle is published to origin teams, who configure their ingress (for example Envoy or NGINX) to require it. OpenBao's PKI serves as the upstream CA for SPIRE.

What it buys

• Origins can close their public exposure
• Direct-path use is visible to the origin
• No long-lived secrets to rotate

What it costs

• Origin teams must configure client certificate validation
• SPIRE becomes a dependency of shield-to-origin fetches, cached for an hour
• Debugging from outside the platform needs a test identity

Choose differently when. Origins that are public object stores serving public content may skip mTLS and use signed URLs instead.

Why it holds up over time. SPIFFE is an open standard with several implementations, and mTLS is supported by every origin stack. Changing identity providers does not change the origin contract.

Lesson. Authenticate the caller with an identity it proves at connection time, not with an address or a shared secret.

Invalidation

How content changes everywhere within seconds, how the platform knows it did, and when not to purge at all.

ADR-13 · Invalidation compares timestamps at lookup instead of deleting objects

Status: Accepted · Shown on views: 05, 09, 16

How can a purge of two million objects reach 18 PoPs in seconds without affecting serving?

Context. The requirement asks for purge by URL, surrogate key and property, 99% of PoPs acknowledging within 5 seconds p95 for a two-million-object surrogate key purge, and purge storms that never block serving. Deleting matching objects means finding them: an index of tags on every node, and millions of delete operations competing with serving I/O.

Decision. An invalidation is a record of selector, time and mode. Selectors are a key hash, a surrogate key or a property generation. Each node keeps a table from selector to latest invalidation time. On lookup, an object whose store time is earlier than any of its selectors' invalidation times is a miss (purge) or stale (expire). Entries are dropped after the TTL ceiling, which bounds the table. Nothing is deleted; invalidated objects are overwritten on refetch or evicted normally.

How it works on-premise. The table is an LMDB file on each node's NVMe, written only by the PoP agent and memory-mapped read-only by the tagguard plugin in Traffic Server. Surrogate keys come from an origin header or configuration and are stored with the object. tagguard runs after cache lookup and changes the lookup result. A property-wide purge advances the property generation, a selector every stored object carries, so the same comparison invalidates everything the property has stored without touching the key.

What it buys

• Purge cost does not depend on how many objects match
• Purge storms never touch serving I/O
• Returning PoPs replay a small log instead of scanning caches

What it costs

• Invalidated objects occupy disk until evicted or refetched
• Lookup reads a table on every request, microseconds
• Depends on a plugin changing lookup status, to be confirmed

Choose differently when. A platform with few objects per tag and rare purges can use active deletion, which frees disk immediately.

Why it holds up over time. Timestamp comparison is independent of object count, storage medium and engine, so it keeps working at 2 billion objects. It relies only on objects recording when they were stored, which every cache does.

Lesson. When an operation's cost grows with the data, see whether a comparison at read time can replace the write.

ADR-14 · NATS JetStream is the invalidation log, mirrored into every PoP, with acknowledgement per PoP

Status: Accepted · Shown on views: 11, 16, 17

What carries invalidations so that none acknowledged is lost and every PoP's outcome is known?

Context. Invalidation state is small, ordered, must not be lost (RPO 0) and must reach 18 PoPs across continents in seconds. PoPs are sometimes unreachable for hours. A purge whose outcome is unknown has not happened, so each PoP's position must be known, and a returning PoP must be able to replay what it missed.

Decision. Purges are published to a NATS JetStream stream replicated three times across the two core data centres and a witness site. The purge API returns only after quorum. Each PoP runs a leaf node with a local mirror of the stream, so delivery survives WAN interruptions and replay is local. The PoP agent applies records in order and publishes an acknowledgement with the highest sequence applied. The purge API records acknowledgements per PoP and reports which PoPs are pending. The stream keeps 30 days.

How it works on-premise. Three JetStream servers form the core cluster (DC East, DC EU, DFW compute). Each PoP's leaf node connects to the nearest core server with credentials that allow subscribing to the invalidation stream and publishing only on that PoP's acknowledgement subject. Per-property publish budgets are enforced in the purge API; JetStream flow control provides back-pressure during purge storms.

What it buys

• Acknowledged purges survive any single site loss
• PoPs catch up locally after an outage
• Per-PoP outcome is a recorded fact

What it costs

• A second messaging system alongside Kafka
• Operators must learn JetStream operations
• NATS is led by one company, with Kafka as the named fallback

Choose differently when. An organisation already running Kafka at the edge with strong operational skills might use it for both logs and invalidations and accept the heavier footprint.

Why it holds up over time. The design depends on an ordered, replicated, replayable log with per-consumer positions, which several products provide. The records are small and product-neutral, so moving to Kafka or another log is a transport change, not a redesign.

Lesson. If the outcome of an action matters, record it per recipient; do not assume delivery from a successful call.

ADR-15 · A returning PoP reconciles before serving, and this ships in the MVP

Status: Accepted · Shown on views: 06, 17, 22

What stops a PoP that missed purges while away from serving stale content when it returns?

Context. The requirement puts reconciliation in Phase 2 but also says no acknowledged invalidation may be lost to an absent PoP. Without reconciliation, a PoP back from a nine-hour transit outage would serve old prices immediately. With a replayable log already in place (ADR-14), reconciliation is small.

Decision. A PoP that has been withdrawn stays withdrawn until the agent confirms that its tag tables have applied every invalidation up to the current stream head and that its configuration bundle matches the current ring version. If the gap is older than the log's retention, or a tag table is missing, the agent bumps every property generation, which turns every stored object into a miss. Only then does pop-guard announce, starting in cold mode.

How it works on-premise. The agent reads applied_through_seq from LMDB, resumes the leaf mirror from the next sequence and applies records in order. pop-guard queries the agent's readiness endpoint before instructing BIRD to announce. The 30-day stream retention exceeds the declared 14-day maximum absence.

What it buys

• No stale serving after an outage
• Short outages return warm
• Long outages are handled safely

What it costs

• Return takes seconds longer
• Missing tag tables force a full logical flush
• pop-guard depends on the agent being healthy to announce

Choose differently when. A platform that uses only versioned URLs and never purges has nothing to reconcile.

Why it holds up over time. Catching up from a log before serving is how replicas, databases and caches have safely rejoined for decades. It remains correct at any purge volume or PoP count.

Lesson. A replica that was away must catch up before it answers; being reachable is not the same as being current.

ADR-16 · Versioned URLs first, purge second, short TTLs last, with the strategy declared per path

Status: Accepted · Shown on views: 05, 12

Should content change be handled by purge, by versioned URLs or by short TTLs with revalidation?

Context. Purge gives immediate control but needs reliable global fan-out. Versioned URLs make invalidation unnecessary and long TTLs safe, but only work where the publisher controls URL generation. Short TTLs need no invalidation machinery and put permanent revalidation load on origins. Mixing strategies silently is how a stale asset survives a release, because nobody knows which path relies on which.

Decision. Each path declares an invalidation strategy: versioned, purge or expire. Versioned paths get the property's maximum TTL and are rejected by the purge API, so a mistaken purge is visible. Purge and expire paths must carry surrogate keys. Short TTLs with conditional revalidation are allowed for content that genuinely changes every few seconds, such as live scores, and are never a substitute for a missing purge.

How it works on-premise. The strategy is a field on path_rule, compiled into the bundle and into the purge API's rule index. Asset pipelines hash file contents into file names and set long Cache-Control; the edge still applies its 30-day ceiling. Revalidation uses ETag and If-None-Match end to end from edge through shield to origin, so an unchanged large object costs a 304.

What it buys

• Most bytes, the assets, never need invalidation
• Purge volume falls to the paths that really need it
• A release's invalidation plan can be read from configuration

What it costs

• Owners must change asset pipelines to hash names
• The purge API must know rule strategies
• Declaring strategy is one more field to get right

Choose differently when. A property whose content is entirely third-party URLs has no versioning option and must rely on purge and TTLs.

Why it holds up over time. Content-addressed URLs are the most durable invalidation strategy there is: they need no infrastructure and work identically on any CDN. Declaring strategy per path keeps the platform honest as properties change over the years.

Lesson. The cheapest invalidation is a new name for new content; use purge only where names cannot change.

Topology and steering

How users reach a PoP, how traffic is moved, when a PoP removes itself, and how much spare capacity is carried.

ADR-17 · Anycast from every PoP, with Katran on dedicated hosts for QUIC-aware L4 balancing

Status: Accepted · Shown on views: 07, 09, 20, 21

How do users reach a PoP, and how are connections spread across its nodes?

Context. Anycast addresses reach the topologically nearest healthy PoP and converge quickly on withdrawal. Inside a PoP, connections must be spread across nodes so that a node change does not break existing flows. HTTP/3 over QUIC complicates this because a client's address can change during a connection, so plain five-tuple hashing drops migrated flows.

Decision. Every PoP announces the service prefixes through BGP. Two dedicated L4 hosts per PoP run Katran, an XDP load balancer using Maglev consistent hashing, forwarding to cache nodes over IP-in-IP. QUIC connection IDs carry a server identifier, so Katran routes QUIC packets by connection ID rather than address. BIRD on the L4 hosts announces prefixes to the border switches only while pop-guard reports the PoP healthy.

How it works on-premise. Border switches are whitebox hardware running SONiC with FRRouting, each PoP with two transit providers and a local exchange port. Katran hosts have 2 × 100 GbE and forward to nodes that decapsulate IP-in-IP. Envoy's QUIC-LB connection ID generator encodes each node's server ID. Every announced prefix has an RPKI ROA.

What it buys

• Node failure affects only its own connections
• QUIC flows survive client address changes
• PoP withdrawal converges in about 30 seconds

What it costs

• Two extra hosts at every PoP
• XDP and eBPF skills needed on the network team
• IP-in-IP adds encapsulation overhead and MTU planning

Choose differently when. Small PoPs with few nodes and little QUIC traffic can run ECMP directly and save the L4 hosts.

Why it holds up over time. Anycast with consistent-hash L4 balancing has been the standard CDN front door for over a decade and does not depend on the products chosen. Katran can be swapped for IPVS or another Maglev implementation without changing anything above it.

Lesson. Balance by consistent hashing so that adding or losing a backend moves only the traffic that has to move.

ADR-18 · Four anycast prefix classes for fine steering, and DNS as a manual per-property override from one intent

Status: Accepted · Shown on views: 21

Anycast alone, DNS steering, or both?

Context. Anycast cannot say 'move 30% of this property away from Frankfurt'. DNS steering can, but resolvers ignore TTLs, so a drain has a long, unbounded tail. Running both means two systems that can disagree about where traffic belongs, and an on-call engineer has to reason about both at 3 a.m.

Decision. Properties are distributed across four anycast prefix classes. Withdrawing one class at one PoP moves roughly a quarter of that PoP's traffic within BGP convergence time, which covers most finer-than-PoP steering. DNS is used only as a manual, per-property override that points a hostname at a different class. BGP and DNS are both generated from one steering intent object per property, so they cannot be edited separately.

How it works on-premise. Each class is a pair of IPv4 /24 and IPv6 /48 prefixes. PowerDNS Authoritative, fronted by dnsdist and itself anycast from the tier-1 PoPs, serves property hostnames with 60-second TTLs. Steering intent lives in Git beside property configuration and goes through the same validation and ring rollout.

What it buys

• Most steering happens at BGP speed
• Per-property moves exist when needed
• No split-brain between steering systems

What it costs

• Four times the address space per PoP
• Class assignment must be balanced as properties grow
• DNS moves still have a slow tail

Choose differently when. Platforms with long-lived sessions and few tenants may prefer DNS for everything and accept slow drains.

Why it holds up over time. Prefix classes are plain BGP and work with any router and any transit provider. Deriving every steering mechanism from one declared intent stays useful whatever mechanisms are added later, including a second CDN.

Lesson. If two control mechanisms steer the same traffic, generate both from one source rather than keeping them in sync by hand.

ADR-19 · Automatic withdrawal needs sustained evidence, drains first, and never removes the last PoP

Status: Accepted · Shown on views: 06, 22, 29

When should a PoP remove itself from the internet, and what stops a health check from taking everything down?

Context. Withdrawal is the main way to take a PoP out of service. A PoP that stays announced while broken black-holes its users; a PoP that withdraws on one failed probe flaps. Worse, a shared cause such as a bad health rule or a problem the probes themselves have can make every PoP withdraw in turn, which is a global outage caused by the safety mechanism.

Decision. pop-guard withdraws only on sustained local conditions: fewer than half the cache nodes healthy, a cache tier failing its checks, or transit loss confirmed by external probes for three minutes. Planned withdrawals drain first, with Envoy sending GOAWAY and closing QUIC connections and in-flight requests given 120 seconds. pop-guard never withdraws if its own external probes cannot reach the anycast prefix through at least two other PoPs. Re-announcement requires 15 clean minutes and a reconciled PoP (ADR-15).

How it works on-premise. pop-guard is a small Go service on both L4 hosts that controls BIRD's announcements through its control socket. Evidence comes from local node health, BGP session state on the SONiC switches and external probe results fetched over the out-of-band network. Every withdrawal and suppressed withdrawal is logged and paged.

What it buys

• Broken PoPs stop taking traffic in minutes
• Planned work breaks few sessions
• A shared fault cannot withdraw every PoP

What it costs

• Three minutes of degraded service before automatic withdrawal
• External probes become part of the decision
• The never-last rule can keep a bad PoP announced during a wide incident, which pages instead

Choose differently when. Platforms with many more PoPs per region and very short sessions can tolerate faster, more aggressive withdrawal.

Why it holds up over time. Guarding automation against acting on shared bad evidence is a lesson from well-known internet-scale outages and applies to any future steering mechanism. The rules are expressed as policy, independent of BIRD or Katran.

Lesson. Any automation that removes capacity needs a limit that stops it removing all of it.

ADR-20 · Capacity is planned to n−1 per region from measured per-node ceilings, and cold PoPs refill under a cap

Status: Accepted · Shown on views: 06, 20, 22, 25

How much capacity does the edge carry, and how is a cold PoP kept from hurting the shield?

Context. Losing the largest PoP in a region must not push its neighbours over their ceilings, and the edge must reach twice the assumed peak by adding nodes. Anycast does not split load evenly: the busiest PoP often carries two to three times the average share. A PoP that returns empty pulls hard from the shield until it warms.

Decision. Each region is sized so that its largest PoP can be withdrawn at regional peak without any neighbour exceeding 80% of its measured ceiling. Per-node ceilings are measured on each hardware generation under TLS load and are the only figures capacity plans may use. A PoP starting cold caps its shield fetch concurrency at 25% of steady state until its byte hit ratio passes 80%. The cost of n−1 headroom is reported separately every month.

How it works on-premise. Planning uses 20 Gbps per edge node until the proof phase measures the real figure. Tier-1 PoPs have 6 edge nodes and edge-only PoPs have 3, which gives node-level n−1 inside every PoP as well. Cold mode is a flag in the PoP's local state that sets Traffic Server parent connection limits.

What it buys

• A PoP loss is a latency event, not a capacity event
• Capacity plans rest on measured numbers
• Refill never becomes a shield incident

What it costs

• Significant idle capacity, reported as insurance
• Measurements needed per hardware generation
• Cold PoPs take hours to reach full efficiency

Choose differently when. With a contracted commercial overflow path that shares purge semantics, headroom can be lowered to what the overflow does not cover.

Why it holds up over time. Capacity based on measured ceilings and explicit failure assumptions stays correct as hardware changes. The planning rule does not change when nodes get faster; only the measured numbers do.

Lesson. Plan capacity for the failure you know will happen, and use only numbers you have measured.

Configuration and control

Where configuration is authoritative, how it reaches 18 sites safely, and where the control plane runs.

ADR-21 · Git is the configuration authority, and PoPs apply only signed bundles and keep the last known good

Status: Accepted · Shown on views: 07, 11, 23, 29

Where is configuration authoritative, and how does a PoP keep serving without the control plane?

Context. Configuration must be versioned, reviewed, auditable and never edited on a PoP. PoPs must serve on last-known-good configuration indefinitely while the control plane is unreachable, and accept configuration from no other source. A PoP in a third-party facility must not trust anything it downloads just because it came from the expected address.

Decision. Property and steering configuration lives in Git. On merge, a compiler produces an immutable bundle per PoP containing Envoy, Traffic Server and keyspec configuration, signs it and stores it by content hash. PoPs pull bundles when notified, verify the signature against a public key built into the node image, and switch atomically. Each PoP keeps its last five bundles on disk. With no control plane, a PoP keeps serving its current bundle and accepts nothing new.

How it works on-premise. Forgejo hosts the repositories, with signed commits and protected branches. Bundles are signed with cosign using a key held in OpenBao Transit and stored in a Ceph RGW bucket. The PoP agent is notified over NATS, downloads over HTTPS, verifies, writes to a new directory, switches a symlink and triggers an Envoy file-based configuration reload and a Traffic Server reload.

What it buys

• Every change is a reviewed commit
• A PoP cannot be pushed configuration by anyone except the signer
• Revert does not need the network

What it costs

• Owners use Git, not a web console
• The signing key is a crown jewel
• Compiler bugs affect every PoP at once, which is why rings exist

Choose differently when. A platform with one operator team and a few properties can reasonably drive xDS directly from a small control plane and skip bundles.

Why it holds up over time. Declarative configuration in version control, compiled to signed artefacts, is how critical infrastructure has come to be managed and does not depend on Envoy or Traffic Server. The same pipeline can target another engine or a vendor's API.

Lesson. Make the unit of change an immutable, signed artefact, and let each site keep the last one that worked.

ADR-22 · Rollout is a Temporal workflow through canary, region and global rings, halting on regressions without a person

Status: Accepted · Shown on views: 19, 23

How does a validated change reach 18 PoPs without a bad one reaching all of them?

Context. Validation catches known mistake classes; production catches the rest. The requirement asks for canary, region and global rings with automatic halt within 30 seconds of a regression and revert within 60 seconds. A rollout lasts tens of minutes and must survive a control-plane restart partway through.

Decision. Every bundle rolls out through three rings: one canary PoP for 10 minutes, one continent for 20 minutes, then global. At each stage the ring controller compares 5xx rate, hit ratio per path pattern and stale-served ratio against the same PoPs a week earlier. A regression beyond thresholds halts the rollout and reverts affected PoPs to the previous bundle. Emergency reverts skip bakes. Software upgrades follow the same rings.

How it works on-premise. The ring controller is a Temporal workflow with durable timers and activities that notify PoPs and query Thanos and ClickHouse. Temporal runs on the core Kubernetes cluster with PostgreSQL persistence. Ring membership comes from NetBox; the canary is WAW.

What it buys

• A bad change reaches one PoP for minutes, not the world
• Halts need no human
• Rollouts survive control-plane restarts

What it costs

• Standard changes take about 30 minutes end to end
• Thresholds need tuning to avoid false halts
• Temporal is another system to run, with plain workers on PostgreSQL as the fallback

Choose differently when. Emergency security changes, such as blocking a path under active abuse, use the revert-speed path and skip bakes by design.

Why it holds up over time. Progressive delivery with automatic halt applies to any configuration or software on any fleet. The ring definitions and thresholds are data, and the workflow can move to another durable engine without changing them.

Lesson. Let production judge a change on the metric the change could hurt, and let the rollout stop itself.

ADR-23 · PoPs run a signed image on bare metal; the control plane runs on Kubernetes in two core data centres and a witness

Status: Accepted · Shown on views: 08, 20, 22

Where does each part of the platform run, and what does it depend on?

Context. The requirement puts the control plane, logs and origins in a public cloud; this design is fully on-premises. PoP nodes are in third-party facilities with remote hands only. Control plane targets are 99.9% for configuration and 99.95% for purge, with configuration RPO 5 minutes, RTO 30 minutes and invalidation RPO 0.

Decision. PoP nodes boot a signed, read-only A/B image with systemd services and no scheduler, provisioned remotely and attested through their TPM. The control plane runs on Kubernetes: active in Core DC East, warm in Core DC EU. JetStream spans both data centres and a witness in DFW. PostgreSQL uses a synchronous pair in DC East and a streaming standby in DC EU.

How it works on-premise. Node images are built with mkosi, signed, and provisioned by Tinkerbell over the out-of-band network, using NetBox as the inventory. Core clusters run RKE2 with Argo CD. Patroni manages PostgreSQL. Ceph RGW in each data centre stores bundles and cold logs.

What it buys

• PoP nodes have nothing to drift and little to fail
• Control plane recovers within targets without cross-ocean synchronous writes
• Remote provisioning without site visits

What it costs

• Two operating models: image-based PoPs and Kubernetes core
• Warm standby needs regular failover drills
• Image builds become part of every software change

Choose differently when. An organisation with mature cloud operations and no on-premises requirement should put the control plane in a cloud region pair and keep PoPs as they are.

Why it holds up over time. Immutable images and a small core cluster are replaceable separately: a new OS, a new orchestrator or a move of the core to a cloud does not change the PoPs. Because the serving path does not depend on the core, the core can be rebuilt without an edge outage.

Lesson. Put the least machinery where there are the fewest people.

Security

Private keys in buildings the organisation does not own, separated privileges, and a record nobody can quietly edit.

ADR-24 · Each PoP has its own short-lived certificates, with keys sealed to its TPM; keyless termination only later and only for a few properties

Status: Accepted · Shown on views: 27, 28

Where does a TLS private key live in a facility the organisation does not own?

Context. Keys on PoP disks are simple and fast, and a stolen server is a stolen key. HSMs raise the bar and complicate provisioning at 18 sites. Keyless termination keeps keys in a trusted site at the cost of a round trip in every full handshake and a new dependency. Delegated credentials are not supported by enough browsers. The PoP must also survive seven days without the control plane.

Decision. Each PoP gets its own key pairs and certificates for the properties it serves. Certificates last 14 days and are renewed at day 5, so every PoP always holds at least 9 days of validity. Keys are sealed to the node's TPM on disk and decrypted only into memory. A compromised PoP has its node identities revoked and its own certificates revoked; no other PoP is affected. Keyless termination is Phase 3, for properties whose threat model justifies the latency.

How it works on-premise. A certificate issuer in the core creates per-PoP keys in OpenBao, obtains certificates from the public ACME CA with DNS-01 challenges against PowerDNS, and stores key and certificate for delivery. Node agents authenticate with SPIRE identities after TPM attestation, fetch response-wrapped keys, seal them to the TPM and hand them to Envoy through SDS. Session ticket keys are rotated hourly per PoP.

What it buys

• A stolen PoP is revoked alone
• Stolen keys expire within days
• No latency cost on handshakes

What it costs

• Certificate issuance volume is 18 times higher and must be contracted with the CA
• Renewal every 5 days makes issuance a production dependency
• A running server stolen whole is still useful for up to 14 days

Choose differently when. For a property whose compromise would be catastrophic, such as a login domain, keyless termination from the core is worth the latency. For PoPs in facilities with strong physical controls, longer certificates are acceptable.

Why it holds up over time. Certificate lifetimes across the web are getting shorter, so this design moves in the same direction as the ecosystem. Per-site keys, short lifetimes and hardware sealing do not depend on a particular CA or TLS library.

Lesson. When you cannot guard a key, make it expire quickly and make sure it unlocks as little as possible.

ADR-25 · Three separated privileges, encrypted caches that can be destroyed by key, and an audit trail its writer cannot delete

Status: Accepted · Shown on views: 11, 27

Who can change what, how is a PoP's cache destroyed, and how is every change recorded beyond tampering?

Context. Changing configuration, purging and reading delivery logs are different powers with different abuse potential. Purge is the easiest to abuse, because a wildcard purge is a self-inflicted origin load event. Cached content on PoP drives in third-party facilities must be protected at rest and destroyable on decommission. Every purge, configuration change and administrative access must be logged somewhere the writer cannot delete from.

Decision. Keycloak groups map to three roles: config-change, purge and log-read, held by different groups. Wildcard purge needs purge-admin in addition, and is limited to one per property per hour. NVMe caches are LUKS2 volumes with keys sealed to the TPM; decommissioning destroys the sealed keys. Audit events are hash-chained and written to a Ceph RGW bucket with Object Lock in compliance mode for three years, by a role that can put but never delete.

How it works on-premise. Operators sign in to Keycloak with WebAuthn. The purge API and Forgejo check role claims. The audit writer is a small service with a Ceph STS role allowing PutObject only; the daily chain head is also written to the Forgejo audit repository, so tampering in either place is detectable from the other.

What it buys

• No single role can both change and hide
• A decommissioned PoP's cache is unreadable in seconds
• Audit evidence survives insider tampering

What it costs

• More roles to administer
• LUKS2 costs a little CPU on NVMe I/O
• Object Lock storage is kept for three years whatever it costs

Choose differently when. A small platform run by one trusted team may combine config and purge roles, but should still keep audit write-once.

Why it holds up over time. Least privilege, disk encryption with hardware-bound keys and write-once audit are controls that auditors will keep asking for, and each uses a standard mechanism with several implementations.

Lesson. Separate the privilege that can do the most damage, and keep the record of it somewhere its holder cannot reach.

Telemetry and economics

What is measured, from where, what it costs, and whether running the edge is the right call at all.

ADR-26 · Access records are droppable: Vector at the node, Kafka in the core, ClickHouse for analysis

Status: Accepted · Shown on views: 11, 14, 24

How are one structured record per request delivered centrally without ever slowing delivery?

Context. At steady state the edge produces about 7.8 billion access records a day. They must arrive at least once within a declared delay when the pipeline is healthy, but delivery must never slow when it is not, and the gap must be counted. Hit ratio, offload and cost per path pattern are computed from these records.

Decision. Traffic Server and Envoy write structured logs locally. Vector on each node ships them with a 20 GB disk buffer and drops, counting, when the buffer is full. Kafka in each core data centre buffers three days. ClickHouse keeps 30 days hot and moves older partitions to Parquet on Ceph for 400 days. Records carry a key hash rather than the key, and client addresses are reduced to country and ASN.

How it works on-premise. Vector's disk buffer is set to drop newest when full, with a counter exported to Prometheus. Kafka runs three brokers per core data centre. ClickHouse runs six nodes, three per data centre, with ReplicatedMergeTree and ClickHouse Keeper, reading from Kafka through the Kafka table engine. Parquet export uses ClickHouse's S3 table functions against Ceph RGW.

What it buys

• Delivery never waits on logging
• Gaps are measured, not guessed
• Per-path analytics at billions of rows a day

What it costs

• Three systems in the pipeline
• About 28 TB hot and 190 TB cold to keep
• Records dropped in long outages are gone

Choose differently when. For billing that must be exact, bill from per-node counters, which are never dropped, and use logs for analysis only. This design already bills bytes from counters.

Why it holds up over time. Separating shipping, buffering and analysis lets each piece change without the others, and the record schema is independent of all three. Parquet on object storage remains readable by almost any analytics engine.

Lesson. Telemetry must never become a dependency of the thing it observes; drop it deliberately and count what you dropped.

ADR-27 · Availability is measured from outside, and silence at a PoP is an alert

Status: Accepted · Shown on views: 24, 29

How does the platform know it is failing users it cannot see?

Context. A PoP that is withdrawn, black-holed by a transit provider or unreachable from some networks looks perfectly healthy from inside: no errors, no latency, just fewer requests. The requirement sets 99.99% delivery availability measured from outside, and asks for alerts on absence of traffic as well as on errors.

Decision. Probes in 12 external networks request a test object from every property through every PoP's unicast test address and through the anycast address. Their success rate is the availability SLI. Each PoP's request rate is compared with the same hour a week earlier, and a rate below 20% of it pages. Metrics are collected by Prometheus in every PoP, with Thanos for global queries and 13-month retention.

How it works on-premise. Probe hosts are small virtual machines in colocation facilities and eyeball networks that the platform does not use for transit, running blackbox_exporter and reporting to a Prometheus instance outside the PoPs. Alertmanager routes pages. Grafana shows SLO burn rates per property and PoP.

What it buys

• The SLI matches what users see
• Silent failures page
• Probe results also feed pop-guard's evidence

What it costs

• Probe hosts to run in other networks
• Twelve vantage points miss some networks
• Absence thresholds need adjusting around holidays and events

Choose differently when. A platform serving only internal users on one network can measure internally and skip external probes.

Why it holds up over time. Outside-in measurement and alerting on silence apply to any service at any scale, and do not depend on the monitoring products chosen.

Lesson. Measure availability where users are, and treat unexpected silence as a failure.

ADR-28 · Offload and the cost of a miss are financial metrics, and the traffic assumptions are checked before scaling out

Status: Accepted · Shown on views: 14, 25, 26

How is the platform's value measured, and is the stated cost target achievable?

Context. The requirement targets \(4 per delivered terabyte at 6 PB a month and origin bytes at or below 1.5%. The stated figures conflict: 6 PB a month averages about 18.5 Gbps, which puts the 400 Gbps peak at about 22 times the average. At 6 PB a month the fixed cost of 18 self-operated PoPs is around \)30 per TB. The target is realistic only near 45 PB a month with half the bytes on settlement-free peering.

Decision. Cost per delivered terabyte is published monthly per property and PoP, decomposed into transit and peering, hardware amortisation, facility, origin egress and control plane. The cost of a miss (origin egress plus shield transit plus added latency) and origin offload per path pattern are first-class reports, and properties above 1.5% origin bytes are named with the patterns responsible. Hardware beyond the six tier-1 PoPs is ordered only after the proof phase has measured real monthly volume and peak ratio.

How it works on-premise. ClickHouse joins delivered and origin bytes with cost inputs: transit invoices per PoP, hardware amortisation from NetBox asset records over five years, colocation contracts and a control-plane allocation. Monthly results export to the finance ledger for chargeback. Marginal hit ratio per added terabyte is estimated from eviction age data per volume.

What it buys

• A key regression shows up as money
• Owners can see what their misses cost
• Investment follows measured demand

What it costs

• Cost models need finance input and upkeep
• Chargeback invites arguments, which is intended
• Scale-out waits on the proof phase

Choose differently when. If measured volume is well above the stated figure, skip the gate and build to the roadmap; if it is near 6 PB, see ADR-29.

Why it holds up over time. Unit economics expressed per terabyte and per miss let the organisation compare this platform with any alternative at any time, including buying delivery. The metrics outlast every technology choice here.

Lesson. Put a price on the metric engineers tune, and check the demand figures before buying capacity for them.

ADR-29 · One self-operated delivery path for now, with configuration and purge semantics kept portable

Status: Accepted · Shown on views: 25, 26

Is this the only delivery path, or one of several?

Context. A single self-operated platform gives full control of keys, purge and cost, and its worst day is the organisation's worst day. A commercial CDN alongside adds peak headroom and reach, but splits invalidation across two systems with different purge semantics. A second path used rarely is not warm enough to be trusted when it is needed.

Decision. The MVP has one delivery path. Everything that would need to move is kept portable: engine-neutral property declarations, surrogate keys, versioned URLs and Cache-Status. A second path is added only when purge can be expressed identically on both and acknowledged from both, and it must carry a constant share of traffic to stay warm. If the proof phase measures volume near 6 PB a month, buying delivery and keeping only the key contract and purge semantics in-house is a valid outcome.

How it works on-premise. The compiler's back-end interface is designed so a vendor configuration back end can be added. The purge API's record format includes a target path field from the start. Steering intent (ADR-18) can express a share of traffic per path, which a second CDN would need.

What it buys

• One set of semantics to get right first
• No lock-in to either model
• The build-or-buy decision is made on data

What it costs

• No external overflow for extreme peaks at launch
• Portability features cost some design effort now
• A future second path still needs integration work

Choose differently when. If one outage of the self-operated platform would cost more than a year of a commercial contract, run a second path from the start and invest in purge parity first.

Why it holds up over time. Keeping the valuable parts portable, meaning the key contract, invalidation semantics and steering intent, lets the organisation change delivery models as prices and scale change without redesigning how properties describe their content.

Lesson. Own the semantics, and treat the machinery that implements them as something you might one day buy.

Every package used, in one table

Every open-source package named on the views, what it is, the job it does here, and what was considered instead.