Between 2016 and 2026 Cloudflare rebuilt almost every component of its data plane, and made its configuration distribution faster at every step. Reconstructed from eight of its own incident reports, its engineering accounts and the issue threads in its open-source proxy, this guide traces which of those rebuilds paid, and why the outages kept arriving through the one path that never got release engineering: the push that carries data rather than code.
A company runs one fleet of identical machines in hundreds of cities. It must be able to change what those machines do, globally, within seconds. Ten years of evidence on what that capability costs.
State the problem without the company's name in it and it becomes general. You operate a large number of identical machines. The machines run code, which you release carefully, and they read configuration, which decides what the code does to each customer's traffic. Code moves slowly because releasing it is expensive. Configuration moves fast because the whole point of it is to move fast: a new attack signature is worthless if it takes a day to reach the edge. So you build a distribution system that puts a new value on every machine on earth in under a second, and you have thereby built the most powerful single-action blast radius in your architecture, and it is not covered by any of your release engineering.
Cloudflare is the best-documented instance of this in public: it publishes long incident reports with timelines and named mechanisms, and its architecture is unusually uniform, because every server in every city can serve any service on any of its addresses, so there are no natural partitions to limit a bad value's reach. The ten-year arc is easy to summarise and hard to accept. The data plane was rebuilt component by component into something markedly faster and safer. The configuration plane was rebuilt too, and each rebuild made it faster: in 2015 the team replaced the config store because what "worked for the first 25 cities began to show its age as the network passed 100", and in 2026 the successor to that replacement reaches over 300 locations with a p99 of 256 ms. Nothing in the public record describes a corresponding increase in the number of gates a configuration change has to pass.
On 18 November 2025 the same oversized data file hit two generations of proxy, and the
new one failed worse. Cloudflare's report says customers on the Rust proxy, FL2, "saw 5xx
errors for most bot-protected traffic, while FL customers saw bot scores default to zero
instead". The old unstructured Lua served wrong answers; the new memory-safe Rust called
unwrap() on a failing result and panicked. Memory safety is not failure
safety, and a rewrite that tightens correctness without moving validation to the
ingestion boundary converts data errors into availability errors.
Who else has this problem: anyone running a feature-flag service, a WAF ruleset, a threat-intelligence feed, a routing table, a pricing table, a model artefact, or any machine-generated file that a fleet reads at request time. The specifics below are Cloudflare's. The shape is not.
This guide covers the request path, the compute platform and the configuration distribution path, 2016 to 2026. It does not cover Cloudflare's security products as products, its commercial performance, the Workers developer experience, or any comparison with other content delivery networks. Claims about internal systems stop where the public record does: Oxy's internals, Quicksilver v2's storage engine and the current FL1-to-FL2 traffic split are all unpublished, and are marked as open below.
On method: this session's network policy permitted direct fetches only from
GitHub-family hosts. Every GitHub row in the ledger was read in full. Cloudflare's
incident reports and engineering posts were retrieved through in-session search, which
returns extracted text and the canonical URL rather than the whole page, and each
load-bearing number from them is cross-checked here against an independent host. The
ledger in sources.md marks every row as direct or
search so a reader can discount accordingly. Treat the GitHub evidence as
first-hand and the rest as faithfully quoted but once removed.
One edge machine, reconstructed from the engineering accounts, and the three separate ways something new arrives on it.
Reconstructing the request path from the public accounts gives a stack that is unusually shallow for its scale. A packet arrives at the nearest city by anycast. Inside the data centre it meets Unimog, a layer 4 load balancer running as an eBPF program on XDP, which picks a server from a forwarding table its own control plane generates; Cloudflare's eBPF account lists "Load balancer, running on XDP" as one layer of a stack of BPF programs on every edge server. Unimog exists because of a property worth dwelling on: "within a single data center, any of the servers can handle a connection for any of our services on any of our anycast IP addresses". There is no service-to-machine assignment to protect you.
On the chosen server, TLS terminates and the request enters the component Cloudflare calls the core proxy, FL. This is where each customer's configuration is applied: firewall rules, bot management, caching behaviour, routing to Workers or to R2. FL was built on NGINX, OpenResty and LuaJIT. Its replacement, FL2, is Rust on Oxy, Cloudflare's internal proxy framework, and InfoQ's report of the migration describes the sequencing choice precisely: rather than maintain two copies of the product logic, the team "added a layer inside the old NGINX/OpenResty FL that lets the new FL2 modules run". The stated motivation was not raw speed. Engineers were spending growing time working around LuaJIT bugs, and the unstructured Lua made it unsafe to add product logic.
Upstream of the proxy sits Pingora, which Cloudflare describes as the proxy that connects it to the Internet, and which is the one component of this stack you can read. Its README claims "more than 40 million Internet requests per second for more than a few years". Compute is Workers: a V8 isolate per tenant inside a shared process, not a container, because the 2018 account states the requirement as running untrusted code "securely, with low overhead" and notes that one process can hold "hundreds or thousands of Isolates". HTTP/3 at the edge comes from quiche, a separate Rust library whose README says it "powers Cloudflare edge network's HTTP/3 support".
Three pieces of direct evidence say the open-source artefacts are not the production
ones. The workerd README states it "does not contain suitable defense-in-depth
against the possibility of implementation bugs" and that the hosting service "uses many
additional layers of defense-in-depth". Pingora's HTTP/3 support has been an open issue
since March 2024 while the edge has served HTTP/3 via quiche for years. And when a
contributor sent a large TLS refactor as pull request #336, the maintainer called it "a
giant pr", asked for it to be marked do-not-merge, said he was recreating the changes
internally, and closed it two months later with "This has been merged!". The public
repository is downstream of an internal one. Architects citing Pingora in a design review
are citing a library Cloudflare publishes, not the proxy that serves their traffic.
Now the part that matters, and that no single source draws. Three different kinds of change reach that machine, through three different pipelines, with three very different amounts of safety machinery around them.
The code path has a protocol, and you can read it. Pingora's own documentation describes a
graceful upgrade in which the new instance starts with --upgrade, takes over
the listening socket from the old one on SIGQUIT, and guarantees that "no
request will see connection refused when trying to connect to the server endpoints". That
is a designed hand-over, built because releasing code is understood to be dangerous.
The configuration path has Quicksilver, a replicated key-value store written to replace Kyoto Tycoon from 2015 onward, which fans customer configuration out to every edge machine. Its published performance characteristic is worth noting for a different reason than speed: read latency degrades sharply under concurrent writes, from a p99 of about 9 ms with no writers to about 154 ms with one writer pushing 40 kB values and about 701 ms with two. Configuration distribution is not a background concern; it competes with request serving for the same disks. The 2026 successor reports a write replication p99 of 256 ms to over 300 locations, against 4.38 s for the older mode.
The third path is the one with no name in the sources, so I will name it: the data deploy. A query runs against an analytics cluster, its output becomes a file, the file is regenerated every few minutes and shipped to every proxy, and the proxy parses it at load and uses it per request. Cloudflare's bot-management feature file is exactly this, regenerated every five minutes. Nobody calls it a deploy. It has no version the operator names, no staged rollout, no canary, and in November 2025 no schema check. It is, in every operational sense, a global code change: it alters what the proxy does to every request on earth, and it does so faster than any code release could.
Any server serves any service on any address. This is what makes the architecture cheap to operate and what removes every natural bulkhead. A bad global value has no partition to stop it.
Stated at: Cloudflare, Unimog, 2020
FL applies every customer's security and routing config in one process. In November 2025 bypassing it was the mitigation for Workers KV and Access, which tells you how much sits behind it.
Stated at: Cloudflare, 18 November 2025
Quicksilver is in the request path's dependency set, not beside it. Its write load shows up as read latency on the machines serving traffic, which is why batching and compression were part of its original design.
Stated at: Cloudflare, Quicksilver, 2020
Eight forks taken over ten years, each with the stated reason and the condition that would flip it for you.
| Decision | Chosen | Rejected | Because | Flips when | Evidence |
|---|---|---|---|---|---|
| Multi-tenant compute unit | V8 isolate per tenant in a shared process | Container or VM per tenant | Start-up cost and per-tenant memory; one process holds hundreds to thousands of isolates | The workload needs OS APIs, native code or long-lived processes. Cloudflare's own follow-up concedes the isolate tooling ecosystem "is still young" | Cloudflare, 2018 |
| Core proxy language | Rust on Oxy (FL2) | Keep NGINX, OpenResty and LuaJIT (FL1) | Time lost to LuaJIT bugs; unstructured Lua made product changes unsafe | Never purely on language grounds: on 18 Nov 2025 the Rust path 5xx'd where Lua degraded. Rewrite only with input validation moved to ingestion | InfoQ, 2025 |
| Migration shape for the proxy rewrite | Run new Rust modules inside the old proxy | Two parallel proxies with duplicated product logic | Avoid maintaining two copies of every feature | Your old runtime cannot host the new one. The cost of this choice is a long period where both failure modes are live, which both 2025 incidents demonstrate | InfoQ, 2025 |
| Config replication topology | Full copy on every server (Quicksilver v1) | Partitioned or on-demand fetch | Edge read latency; a cache miss on config is a request-path stall | The dataset outgrows the cheapest disk. Quicksilver v2 exists because "storing a full copy on every server was becoming expensive" | Cloudflare, 2026 |
| Config correctness method | Declarative policies in a verified DSL, for DNS only | Imperative generators that build query-to-IP maps | Imperative assignment hides behaviour "particularly when objectives conflict"; policies let conflicts be caught before deployment | Nothing in the record flips it. The open question is why the method stayed inside DNS and never reached the rule and feature files that caused the outages | Topaz, SIGCOMM 2024 |
| Control-plane placement | Primary facility plus disaster recovery (pre-2023) | Multi-region active-active | The control plane was not on the data plane's critical path, so its availability target was lower | When customers cannot change configuration or read logs during an incident. Flipped by November 2023, after which failover was the explicit fix | Cloudflare, 2023 |
| Storage backend for Workers KV | Single backend, partly third-party | Redundant backends across providers | Shipping speed; the dependency was not treated as tier zero | When a downstream product makes KV a hard dependency. Flipped after 12 June 2025; Cloudflare then re-architected for redundancy | Squiz incident record, 2025 |
| Object storage egress pricing | Zero egress (R2) | Per-GB egress like the incumbents | Egress is the line item that locks customers in; removing it is a structural, not tactical, difference | Storage-heavy, read-light workloads: a third-party model found S3 73% cheaper in one of two scenarios, because R2's per-operation charges are higher | Vantage, 2026 |
Eight published incidents, grouped into four classes. Six of the eight began with a change to configuration or machine-generated data.
Grouping these by mechanism rather than by date produces four classes, and the classes are more useful than the stories. Class A is a global configuration push with no stage gate. Class B is a latent configuration error activated by a later, unrelated change. Class C is a network configuration change whose effect depended on statement order or topology. Class D is a dependency the data plane was assumed not to have. Note what is absent: no incident in this corpus was caused by a code release that passed review and then failed. The release engineering around code works.
unwrap() on a failing result and panicked.unwrap() in a long-running proxy is a code-style question rather than an availability one.peer.rs and health_check.rs and proposing Clippy's unwrap_used lint. It was closed with no visible closing comment. In June 2026 issue #921 reopened the same ground for HttpPeer and HttpHealthCheck, where an unreachable DNS server panics a worker. In September 2026 issue #1023 found one in the logging path, on non-UTF-8 bytes.Result, propagate with ?, and lint against new unwrap() calls. PR #920 is open.No public account describes a Cloudflare configuration rollout that a stage gate caught. The record contains only the pushes that failed. So nobody outside the company can say whether the controls promised in November and December 2025 work, and the same is almost certainly true of your own flag pipeline: you have evidence of the incidents it caused and none of the incidents it prevented. The only way to produce that evidence is to inject a bad value deliberately and watch the gate hold.
Scale, propagation latency, incident duration and the one cost figure that bears on architecture. Every row carries its date.
| Metric | Value | Kind | Context | As of | Source |
|---|---|---|---|---|---|
| Interconnection footprint | 500+ Tbps, 330+ cities | Reported | Sum of transit, peering, exchange and interconnect ports | 2026-04 | Cloudflare |
| Proxy throughput | 40M+ req/s | Reported | Pingora, "for more than a few years" | 2026-10 | pingora README |
| Proxy rewrite efficiency | ~70% less CPU, ~67% less memory | Claimed | Pingora against the previous service at equal load; Cloudflare's own measurement | 2022-09 | Phoronix, reporting Cloudflare |
| Config write replication, current | 107ms p50, 181ms p95, 256ms p99 | Reported | Quicksilver v2 "Instant" mode to all edge locations, over 300 at publication | 2026-10 | Cloudflare |
| Config write replication, legacy mode | 4.38s p99 | Reported | Same table, "Classic" mode | 2026-10 | Cloudflare |
| Config read latency under write load | 9ms → 154ms → 701ms p99 | Measured | Quicksilver v1 with zero, one and two concurrent writers at 40 kB values | 2020 | Cloudflare |
| Feature file size at failure | ~60 → 200+ features, limit 200 | Reported | Bot management file, duplicated rows from a two-schema query | 2025-11-18 | Cloudflare |
| Data-deploy regeneration interval | 5 min | Reported | Why availability oscillated: staggered refresh across the fleet | 2025-11-18 | Cloudflare |
| Change to first customer error | ~23 min | Derived | 11:05 UTC change, errors from about 11:28 UTC; arithmetic on the published timeline | 2025-11-18 | Cloudflare |
| Time to full restoration | ~6h 1min | Derived | 11:05 to 17:06 UTC; most services recovered at 14:30 | 2025-11-18 | Cloudflare |
| Blast radius of one flag change | 28% of HTTP traffic, 25 min | Reported | FL1 customers on the managed ruleset; 08:47 to 09:12 UTC | 2025-12-05 | Cloudflare |
| Worst traffic loss in the corpus | up to 82% down, ~27 min | Reported | The 2019 WAF regex incident | 2019-07-02 | Cloudflare |
| Partial recovery ceiling | ~77% on route restore | Reported | 1.1.1.1; the other 23% of edge servers needed IP bindings restored | 2025-07-14 | Cloudflare |
| Verified-config query rate | ~1M DNS queries/s | Measured | Topaz executing declarative policies live at a global CDN | 2024-08 | SIGCOMM paper |
| Object storage egress | $0/GB vs $0.09/GB | Claimed | R2 against S3 internet egress; R2's per-operation charges are higher and one of two modelled scenarios favours S3 by 73% | 2026 | Vantage |
Three cautions. The Pingora efficiency figures are Cloudflare's own measurement reported second-hand, with no independent benchmark in this corpus, so treat them as a claim. The 2026 replication latencies come from a product announcement describing the Workers KV path, which shares Quicksilver but is not identical to internal configuration distribution; the inference that internal config propagates on the same order is mine, and it is consistent with the December 2025 report of a change propagating "within seconds". The cost row is a third-party model, not either vendor's price list.
Three quantities nobody has published, which is where your risk sits if you are planning against this architecture: the share of traffic now on FL2 rather than FL1, the number of distinct machine-generated files the core proxy reads at request time, and any measurement of how long a configuration rollback takes end to end. The last one is the number that would tell you whether the November 2025 remediation is real.
Every source behind this page, graded. GitHub rows were fetched in full in this session; the rest were retrieved through in-session search, as section 1 explains. Filter by kind.
A data deploy as an outage mechanism, end to end: a permissions change upstream, duplicate rows, a file past a preallocated limit, a panic. The only document here that compares two proxy generations failing on the same input.
The sequel, seventeen days later, and the most useful document here: Cloudflare names which safeguards were not yet in place, namely versioning and rollback, break-glass, and fail-open handling of configuration errors.
The origin of the pattern: a managed WAF rule skipped progressive deployment by design and a backtracking regex saturated CPU worldwide. The 2019 remediation list and the 2025 one are the same list.
The best published example of a latent configuration error: planted 6 June with no effect, activated five weeks later by a test location added to an unrelated non-production service. Route restoration alone did not recover service.
A utility event at one third-party facility removed the control plane and analytics for two days while the data plane kept serving. The honest part is the admission that failover did not work as expected.
A BGP policy change with misordered statements deleted required prefixes in 19 of the busiest locations, which were exactly the sites already migrated to the newer resilient architecture.
A congestion fix in Atlanta routed all backbone traffic to Atlanta. Useful as the clearest statement that configuration blast radius in a network is set by routing, not by the operator's intent.
The most instructive non-Cloudflare document here: this customer's Worker cached region mappings in Workers KV and threw errors rather than falling back when KV went away.
Independent record of the Workers KV incident: Cloudflare's statement that part of the storage was backed by a third party, and a customer impact window running hours past Cloudflare's resolution time.
Argued nine months before the November 2025 outage demonstrated the class: unwrap in a long-running proxy leads to unrecoverable panics. Proposes Result returns and Clippy's unwrap_used lint. Closed with no visible reason.
The same class, still open, with a concrete trigger: an unreachable DNS server panics a
worker through unwrapped to_socket_addrs(). PR #920 open, so the argument is
unresolved rather than settled.
Third instance of the class in ten months, and the most pointed: the panic is in the logging path, so observability code can kill the request it is describing.
A large external TLS refactor, marked do-not-merge at the maintainer's request, reimplemented internally, then closed as merged. Stated reasons: review size and limiting "how fast the library changes for production users".
Six years after Cloudflare left Lua for Rust, a contributor proposes hot-loadable WebAssembly filters so operators can ship proxy logic without recompiling. Open, unanswered, and exactly the code-versus-config boundary this guide is about.
The scale claim (40M+ requests per second) and the code-release protocol: the new instance takes the listening socket on SIGQUIT so no request sees a refused connection. A designed hand-over for code, with no documented counterpart for config content.
States that the published runtime "does not contain suitable defense-in-depth against the possibility of implementation bugs" while the hosting service adds many more layers: a candid statement of the gap between an open artefact and its service.
HTTP/3 at the edge is served by this library, separate from the open-source proxy. Read alongside pingora issue #95, open since March 2024, asking for HTTP/3 support.
The rewrite rationale, and it is not per-request speed: NGINX's per-worker connection pools meant a request could only reuse connections held by its own worker, producing redundant handshakes at their volume.
Why the previous store was replaced from 2015, plus the measurement that matters most here: config read p99 degrades from about 9 ms to about 701 ms with two concurrent writers, so distribution competes with serving.
Full replication to every server became too expensive, and the replacement reports a p99 write replication of 256 ms to over 300 locations against 4.38 s for the older mode. Faster, after three propagation outages.
The uniformity statement that explains the blast radius of everything else: inside a data centre any server can handle any service on any anycast address. An eBPF and XDP layer 4 balancer whose control plane generates the forwarding tables.
The isolate decision and, in the companion post, its honest cost: the tooling ecosystem for isolates was young. Together, the clearest statement of when the choice flips toward containers.
Names the production framework FL2 is built on, which is how you know Pingora is not the thing applying your WAF rules.
The company does validate configuration formally, with a custom Lisp-like language and a verifier built on Racket and Rosette, for DNS. Nothing in the record extends the method to rule files or feature files.
The migration shape: new Rust modules hosted inside the old NGINX and OpenResty proxy, to avoid two copies of product logic. Motivation given as time lost to LuaJIT bugs and unsafe Lua.
The dissenting read: large organisations will not adopt formal methods wholesale, but should on their most critical paths. Included because it disagrees with the conclusion that better process alone is the answer.
Imperatively generated query-to-IP maps hide nameserver behaviour when objectives conflict; encoding objectives as policies in a formally verified DSL catches conflicts before deployment. Runs at roughly one million queries per second.
The only independent view in this corpus of how a global config failure looks from outside: not regional, and not a clean on-off.
The isolate architecture defended by its author: multi-tenancy without VMs or containers, with claimed cold starts 10 to 100 times faster. No timestamp, because the recording could not be opened in this session.
A session on the configuration distribution system itself, listed because the mechanism is central to this guide's argument. The recording could not be opened in this session, so no claim rests on it.
Six rungs that turn this reading into a tested control in your own config pipeline. The line from toy to real is between rungs three and four.
Three containers reading one JSON file from a shared store on a two-second poll, each serving a request that consults it. Change the file and record when each replica's behaviour changes.
Done when: you can state your own propagation p99 from write to last replica. Teaches: you now own a global-change mechanism with no gate on it, which is the starting position of every incident in section 4.
Publish a file with a duplicated section so it exceeds whatever your consumer preallocated, and a second one with a field of the wrong type. Do not add handling yet. Watch what the replicas do.
Done when: you have reproduced both a crash and a wrong-answer failure from the same channel. Teaches: the same bad value produces different failures in different consumers, which is why the November 2025 report shows FL2 returning 5xx while FL1 returned zeros.
Put a schema, a size ceiling and a row-count sanity bound in the publisher. Refuse to publish a file that fails any of them, and emit a metric when you refuse. Keep the previous file in place.
Done when: the bad files from rung two are rejected before distribution and the fleet keeps serving the last good one. Teaches: the cheapest place to stop a bad value is the one place it exists once.
Give every published file a content hash the operator can name. On parse failure the consumer logs, increments a counter, and keeps the last known good version rather than erroring. Add a one-command rollback to any previous hash, and time it.
Done when: you can state your rollback time from decision to last replica, and a deliberately corrupt file causes zero failed requests. Teaches: the three controls Cloudflare named after December 2025, versioning, rollback and fail-open, are one mechanism and none of them works alone.
Add rings: one replica, then ten percent, then the rest, with an automatic halt if the error rate or a behavioural metric in the current ring diverges from the previous one. Include a documented break-glass path that skips the rings, and log every use of it.
Done when: a bad value injected at ring one never reaches ring two, and the halt is visible in your dashboard. Teaches: why the emergency channel becomes the normal channel unless using it is recorded and reviewed.
Make rung two a game day you run monthly against the real pipeline, injecting an oversized file and a type error. Track time to halt and whether anyone had to intervene.
Done when: two consecutive exercises halt automatically with no human action, and you have a trend line. Teaches: the evidence nobody in section 4 has: that your gate holds. An untested gate and no gate have the same expected value.
The transferable conclusion is narrower than "test your configuration". It is this: in a uniform fleet, the fastest path to global behaviour change is the one that carries data, and it is the path least likely to have a release protocol, because nobody involved thinks of it as a release. Cloudflare spent ten years making its data plane better and its configuration plane faster, and the public record shows the second investment repaying in capability and billing in outages. If you cannot slow the propagation, and for good reasons you often cannot, then every remaining control lives at the content boundary: validate where the value is produced, version it so a human can name and revoke one, and make the consumer prefer a stale answer to no answer.
The queries that actually produced the material above. The company-specific ones generalise by swapping the name; the vocabulary ones are the valuable half.
<company> outage <date> postmortem root cause configuration file<company> "incident report" "propagated" OR "rolled out globally" seconds<company> outage "this was our error and not the result of an attack"<company> postmortem "what we are doing" remediation "kill switch"repo:<org>/<proxy> is:issue panic unwrap production stabilityrepo:<org>/<repo> is:pr is:closed is:unmerged sort:comments-descrepo:<org>/<repo> is:issue "[RFC]" OR "[DNM]"<repo> README "does not contain" defense-in-depth production<company> engineering "we replaced" OR "we outgrew" proxy CPU memory<company> "configuration distribution" OR "config propagation" latency p99<company> blog "began to show its age" OR "was becoming expensive"<company> research publications SIGCOMM OR NSDI verifiable declarative configurationstatus.<customer>.com incident "behind Cloudflare" OR "upstream provider"<incident date> outage analysis vantage points measured independently<company> outage customer postmortem "we removed the caching" fallback