Security postmortem for CVE-2019-9900, CVE-2019-9901
An access-control and routing bypass traced to two different views of one header string, and the first real run of a security process borrowed from Kubernetes.
Between 2015 and 2026 Lyft built a service registry, a sidecar proxy, a global rate limiter, a secret store, a credential shim, a stats library, a config plane, an operations console, a workflow engine, a data catalogue and an asset graph, and published all of them. This guide reads the repositories and the package registries to establish which ones are still alive in 2026, what distinguishes them from the ones that are not, and what an architect should therefore decide differently the next time a platform team proposes to build a component.
A platform team builds the pieces its own company needs, and then has to decide, piece by piece, whether to keep them inside, hand them to someone else, or let them stop. Ten years of one company’s artefacts say which of those choices held.
The pattern in those four numbers is the guide. Of the eleven published components examined
here, the six that still ship in October 2026 are all owned by someone other than Lyft: the
proxy and the rate limiter belong to the Envoy project, the validator belongs to Buf, the
workflow engine belongs to the Flyte organisation and its commercial steward, the asset graph
belongs to a CNCF project, and the stats library survives on one or two commits a year. The
five that are dead or dormant all kept a lyft/ import path and a single operator.
That is not a story about generosity. It is a measurement of what keeps infrastructure alive,
and the answer the artefacts give is: other people running it.
The surprise sits one layer down. Donation did not preserve Lyft’s influence over the
things it gave away, and the record is unusually precise about when that influence ended.
Envoy’s governance file still lists eight lyft.com addresses, and every one of them is
under Emeritus maintainers or Friends of Envoy; none is a current maintainer,
extension maintainer or member of the security team. The project’s founder is still a
senior maintainer, now listed with a personal address. The person stayed and the employer
left. Meanwhile the DynamoDB filter Lyft contributed, the one its own 2016 documentation said
it used “for all application communication with DynamoDB”, was moved out of the
core tree into contrib/ in June 2022 by a maintainer at another company.
What this guide covers. Lyft’s published platform components from October 2015 to October 2026, as recorded in git history, repository files, and the package, container and module registries that date their distribution. What it does not cover. Lyft’s internal architecture, its request volumes, its cloud footprint and its costs. This session’s egress policy resolved only GitHub, PyPI, npm, the Go module proxy and index, Maven Central and Docker Hub; the engineering blog, the documentation sites, the conference and paper hosts were all unreachable. So there is no blog, talk, paper or financial evidence here, every claim about Lyft’s internal systems is a reconstruction from artefacts and is marked as one, and the reasons behind each retirement are absent from the record. The earlier company guides in this collection, on Uber and on Monzo, hit the same artefact-only boundary and found the same three exit doors: donation, a dated banner, or silence.
Nobody published a diagram of this. What follows is reconstructed from READMEs, package documentation and the shared project’s own 2016 docs, and the reconstruction is visible: each box names the artefact it comes from.
Start with the registry, because it dates the rest. lyft/discovery is a Flask
application in front of DynamoDB with four routes: register a host, delete a host, list the
hosts of a service, and update a host’s load-balancing weight. Hosts heartbeat, and a
host that has not checked in for HOST_TTL seconds, 600 by default, is dropped.
Reads are served from a 30-second cache. The README states what consumes it: the registered
port is “the port on which the host expects connections, Envoy will connect to this
port”, and a host carries an availability zone, a region, a canary flag and a weight
between 1 and 100. That is a complete picture of a 2016 service mesh control plane: liveness
by heartbeat with a ten-minute horizon, zone awareness, canary marking and weighted routing,
all expressed as rows in a key-value store and polled over HTTP.
The data plane that polled it became Envoy, whose public history begins on 8 August 2016.
Two details in its first documented year matter more than the famous ones. First, Lyft put
application-protocol knowledge inside the proxy: filters that parsed DynamoDB and MongoDB
traffic to produce per-table, per-operation and per-partition statistics, which the 2016 docs
justify as “an invaluable source of data agnostic to the application platform and
specific AWS SDK in use”. That is a deliberate architectural trade, buying uniform
telemetry across a polyglot fleet at the price of protocol-specific code in a shared binary.
Second, overload control was pulled out of the application into a separate decision service.
The 2016 rate-limiting document states the condition precisely: circuit breaking per host is
not enough “when a large number of hosts are forwarding to a small number of hosts and
the average request latency is low”, so Envoy calls a global service, and
“Lyft provides a reference implementation written in Go which uses a Redis
backend”. That reference implementation is the repository now called
envoyproxy/ratelimit.
Around those two pieces sat the parts that never left the company. Configuration was a
filesystem: Envoy’s runtime mechanism, mirrored for Go services by
lyft/goruntime, whose package documentation describes feature flags, kill
switches, regional settings and experiment settings, usually one key-value pair per file,
with a loader, a snapshot and a refresher watching for filesystem events. Metrics went
through lyft/gostats. AWS credentials reached containers through
lyft/metadataproxy, which intercepted the instance metadata service so that
“scoped IAM roles” could be given to individual containers rather than to the
host. Application secrets lived in Confidant, a secret store that, in its own words,
“stores secrets in DynamoDB, encrypted at rest”. By 2020 there was a console in
front of all of it: Clutch, which describes itself as an extensible platform for
infrastructure management whose resolver pattern turns the many names a resource has into one
lookup, and in which “basic auditing, authorization, stats, and logging come for free
with every endpoint”.
The shape is consistent, and worth naming because the sources never name it: every cross-cutting concern was moved out of application code into a process or a file the application did not own. Routing and telemetry into a sidecar, admission into a remote service, configuration into a watched directory, credentials into a local proxy, secrets into a service, operations into a console, inventory into a graph. One team could then change policy fleet-wide without touching an application, and that is the strength. The cost, which the next sections quantify, is that each concern became a separate thing somebody had to keep alive, and the company could keep alive only the ones other people were also running.
Routing, retries, zone awareness, protocol statistics and admission all sit in a process the application does not own, which is why a polyglot fleet could be changed centrally.
Evidence: Envoy 2016 DynamoDB filter document
One value per file, a snapshot in memory, a refresher on filesystem events. Cheap to reason about, and it makes a config change an infrastructure operation rather than a deploy.
Evidence: goruntime package documentation
Cartography arrived in 2019, after the mesh and the console, and answers questions about identity and exposure across providers. It is the only component of the set whose activity is still rising.
Evidence: Cartography README, 2026
Four forks, each with the artefact that records it and the condition that would reverse it. The reasons are rarely written down, so where a reason is inferred the wording says so.
contrib/ in June 2022, in the
same window Lyft’s commit share fell to 3 percent; the MongoDB filter stayed in
core and others still harden it, with a BSON depth limit added in July 2026That last point is the part teams underestimate, and the record quantifies it twice. The validator Lyft started in 2017 is now Buf’s, and Buf has put it in maintenance mode and recommends that “new and existing projects transition to using protovalidate”, its own replacement, with a deprecation flag being compiled into generated code as of May 2026. The workflow engine Lyft started in 2019 is now on its second generation under a commercial steward: the README announces that Flyte 2 is generally available and tells anyone on the generation Lyft built to “go to the master branch, where Flyte 1 is now maintained”. In both cases the original author is now a consumer on someone else’s migration schedule, which is the same position as any other adopter, except that the adopter chose it knowingly.
| Decision | Chosen | Rejected | Because, as recorded | Evidence |
|---|---|---|---|---|
| Overload control for high fan-in paths | Central gRPC rate-limit service with a Redis backend | Per-host circuit breaking alone | Hard to tune when many hosts call few hosts at low latency | Envoy document, 2016 |
| Credentials for containers | Local proxy in front of instance metadata | Host-wide IAM role shared by every container | Scoped roles per container rather than per host | metadataproxy README |
| Secret storage | In-house service over DynamoDB with encryption at rest | Not recorded | Not recorded; retired on 14 February 2025 with a dated notice and no stated reason | Archiving announcement |
| Mobile networking | Run the same data plane in the app, then fold it into the proxy’s own repository | A separate mobile library with its own release train | Stated only as a move notice, on the repository’s last commit, 2 December 2022 | Move notice |
| Protocol versions for the rate limiter | Delete the legacy protocols on a schedule | Carrying them indefinitely | Not stated; the effect is two forced migrations for every operator | Removal commit, 2021 |
| Governance for the surviving asset graph | Move to CNCF, publish governance guidelines | Remaining a company project | Not stated; activity rose from 96 commits in 2024 to 921 in 2026 | CNCF documentation commit |
Lyft has published no incident reviews that this corpus can reach. What it can reach are the Envoy project’s two security postmortems, which are about the data plane Lyft built, and two failure records written into commits and registries. Three classes cover all four.
c_str() and string_view, allowing the possibility of inconsistent views between checks and resulting action”. Access control and route intent could be bypassed.std::regex, which is recursive, so “an HTTP request with sufficiently large header values may consume large amounts of stack memory and cause abnormal process termination”, appearing “when matching header values of 16Kb or more”. A second defect validated header size per header added, giving quadratic behaviour.std::regex in user facing paths” and restricts the engine to RE2.All of these are counted from the artefacts in section 06 on 14 October 2026. None of them is a figure about Lyft’s production traffic, because no reachable artefact carries one.
| Component | First commit | Last commit | Commits | Owner in 2026 | State | Evidence |
|---|---|---|---|---|---|---|
| Envoy | 2016-08-08 | 2026-10-14 | 29,150 | Envoy project | Shipping | initial commit |
| Rate limit service | 2017-01-30 | 2026-10-06 | 359 | Envoy project | Shipping | README |
| Cartography | 2019-03-01 | 2026-10-13 | 2,196 | cartography-cncf | Growing | README |
| Flyte | 2019-04-09 | 2026-10-12 | 5,253 | flyteorg | Second generation | Flyte 2 cutover |
| protoc-gen-validate | 2017-10-03 | 2026-05-28 | 591 | Buf | Maintenance mode | README |
| gostats | 2016-12-28 | 2026-10-08 | 112 | Lyft | One or two commits a year | module versions |
| Amundsen | 2019-02-04 | 2026-09-10 | 2,713 | amundsen-io | Drifted, no notice | last release 2024-08-14 |
| Envoy Mobile | 2019-03-04 | 2022-12-02 | 1,794 | Folded into Envoy | Absorbed | move notice |
| Clutch | 2020-07-18 | 2025-04-14 | 2,036 | Lyft | Archived | announcement |
| Confidant | 2015-10-08 | 2025-02-14 | 451 | Lyft | Archived | announcement |
| discovery | 2016-08-14 | 2021-10-14 | 29 | Lyft | Archived | archive notice |
| metadataproxy | 2016-03-02 | 2021-09-16 | 169 | Lyft | Drifted, no notice | README |
| goruntime | 2017-01-26 | 2022-09-08 | 32 | Lyft | Dormant | v0.3.0, 2022-09-08 |
The second table is the one that answers the question this guide opened with. It counts
commits whose author address ends in lyft.com, by year, in the three projects
Lyft seeded and handed over. The method undercounts, because contributors frequently commit
from a personal or a GitHub-provided address: the founder of Envoy authored 331 of his 2016
and 2017 commits from a users.noreply.github.com address and only 95 from
mklein@lyft.com. Treat every figure here as a floor on company participation and
as a reliable indicator of direction rather than level.
| Year | Envoy, lyft.com of total | Flyte, lyft.com of total | Amundsen, lyft.com of total | Source |
|---|---|---|---|---|
| 2016 | 62 / 235 | not started | not started | envoy history |
| 2017 | 172 / 1,271 | not started | not started | envoy history |
| 2019 | 230 / 2,950 | 267 / 367 | 217 / 638 | flyte history |
| 2020 | 226 / 3,433 | 390 / 735 | 89 / 1,149 | amundsen history |
| 2021 | 264 / 3,265 | 11 / 1,077 | 41 / 607 | flyte history |
| 2022 | 95 / 3,745 | 1 / 929 | 6 / 196 | dynamo to contrib |
| 2024 | 1 / 3,359 | 0 / 717 | 3 / 40 | amundsen releases |
| 2026 | 0 / 3,010 | 0 / 540 | 0 / 2 | envoy owners |
Measured: every commit count, date, release date, tag date and version number above,
taken from the repository or the registry named beside it. Derived: the
lyft.com shares, computed by me from author addresses, with the undercount
explained above; and the four years and two months between Confidant’s last image and
its archive notice, which is 21 December 2020 to 14 February 2025. Unknown: how many
services ran the sidecar, how much any of it cost, whether Lyft still operates Envoy, the
rate limiter or Flyte today, and why each retirement happened. The archive notices carry
dates and wording but no reasons, and the only internal references in the whole corpus are
two opaque ticket identifiers.
Twenty artefacts, all retrieved on 14 October 2026 by anonymous clone or by
fetching the registry endpoint shown. The full ledger, with one row per claim and the
supporting quote copied out, is in sources.md beside this page.
An access-control and routing bypass traced to two different views of one header string, and the first real run of a security process borrowed from Kubernetes.
Recursive regular-expression matching crashed the proxy on header values of 16Kb or more, and a per-header size check gave quadratic behaviour. The remedy removed the engine.
Added April 2019, adapted from the Google SRE book. Two postmortems were written against it, both in 2019, and none since.
Requires a sponsoring maintainer and two named reviewers per extension, explicitly so that “incentives are aligned”, with a six-month interval for removing one nobody maintains.
The versioning contract for the configuration API that every mesh operator inherits: independent package versions, no renames, no breaking changes within a major version.
October and November 2024, on the one Lyft-origin project whose activity was rising;
the repository is now cartography-cncf/cartography.
The 2016 docs state that “at Lyft we use this filter for all application
communication with DynamoDB” to get telemetry independent of each service’s SDK.
The filter moved to contrib/ in June 2022.
States the condition that justifies a central limiter over per-host circuit breaking, and notes that “Lyft provides a reference implementation written in Go which uses a Redis backend”.
Lyft’s own protocol filter demoted out of the core tree, authored by a maintainer at another company, in the year Lyft’s commit share fell to 3 percent.
The other filter from Lyft’s 2016 data stack stayed in core, still hardened by people who do not work there.
Eight lyft.com addresses remain, every one under Emeritus maintainers or Friends of Envoy; in 2017 Lyft held the senior maintainer seat and five further entries.
Six Lyft addresses, including the senior maintainer: the cleanest measure of a handover in this corpus is the diff against the current file.
A Flask and DynamoDB registry with heartbeats, a 600-second host TTL, zone tags, canary flags and weights, retired by a one-line banner on its 29th commit.
A credential shim that gave containers scoped IAM roles before the platforms did. No deprecation notice; the commit log simply stops in September 2021.
Dated, explicit, and shipped as release 7.0.0 so that the PyPI summary now starts with “DEPRECATED”. The container images had already stopped in December 2020.
Five years, 2,036 commits and 1,994 npm releases of an operations console, ended two months after the secret store, with a banner and a contact address.
A full engineering account inside a commit message: the unbounded store, the rejected LRU alternative, three concurrency bugs found in review, benchmarks.
Lyft’s reference implementation, now an Envoy-project repository with the module path
github.com/envoyproxy/ratelimit, which records two protocol removals: the
legacy proto in July 2020 and the v2 API in August 2021.
The generation Lyft built is now maintained on a side branch while the new stewards ship a replacement backend and a second-generation IDL, cut over in April 2026.
“Has reached a stable state and is in maintenance mode”, with users directed to
the new owner’s own replacement. The Java artefacts still publish under the
io.envoyproxy group id.
Confidant images stop on 21 December 2020, the archive lands on 14 February 2025, and the client ships again on 18 March 2025. Amundsen’s three packages stop in 2024 with no notice anywhere.
Clutch published 1,994 npm versions and its last one on the day of the archival
announcement. The stats and config libraries still carry github.com/lyft/
module paths, and both have near-flat release histories.
Not one engineering blog post, talk, paper or cost figure appears in this wall, because no host carrying them was reachable. Dates, counts and quoted file contents are therefore solid, and every statement about motive is either quoted from a commit or marked as inference. The artefacts do not carry the reasons, and neither does this page.
The method transfers more cleanly than the conclusions do. Six rungs, from an hour of shell commands to a retirement you can actually execute.
Clone every public repository your company owns with --filter=blob:none, and
tabulate first commit, last commit, commit count and whether an archive banner exists.
Done when: you have one row per repository and can name every one whose last commit is more than a year old. Teaches: that the drifted components, not the archived ones, are the liability.
For each component, fetch the last publication date from every channel people install it through: package registry, container registry, module proxy, internal artefact store.
Done when: you can show, for one component, a gap between the last release and the last commit. Teaches: that repositories and distributions die on different days, and users notice the latter.
For each component, write down how many teams outside the owning team run it in production, and how many organisations outside the company do.
Done when: every component has a number, and the single-operator ones are flagged. Teaches: the one variable that predicted survival across this entire corpus.
For every feature your company maintains inside a shared project, check whether you could staff a sponsor and two reviewers for the next three years. If not, plan for it to be relocated or removed.
Done when: each upstream feature has a named owner or an exit plan. Teaches: that upstreaming converts a maintenance cost into a political one.
For each third-party configuration API or wire protocol you depend on, list the last two breaking removals and how long you had. Turn that into a recurring budget line.
Done when: you can state, in engineer-weeks per year, the cost of staying current on each adopted contract. Teaches: that the cheapest component to build is often the most expensive to keep.
Pick your least-used internal component and retire it properly: a dated notice in the repository, a final release whose metadata says deprecated, a stopped image build, and a named replacement or an explicit none.
Done when: an outsider can tell the component is dead from the registry alone. Teaches: that an honest exit is a deliverable with four artefacts, not an announcement.
These are the commands and endpoints that produced this page, from a session that could reach only code hosts and package registries. They work on any organisation.
git clone --filter=blob:none https://github.com/ORG/REPOgit log --reverse --date=short --format='%ad %h %an %s' | head -5git log --date=format:%Y --format='%ad' | sort | uniq -cgit log --date=short --format='%ad %s' -- README.md | headgit log --date=format:%Y --format='%ad|%ae' | awk -F'|' '{t[$1]++; if ($2 ~ /lyft\.com$/) c[$1]++} END {for (y in t) print y, c[y], t[y]}' | sortgit log --since=2025-01-01 --format='%ae' | sed 's/.*@//' | sort | uniq -c | sort -rngit show HEAD:OWNERS.md | awk '/^#/ {print NR": "$0} /lyft/ {print NR": >> "$0}'git ls-tree -r --name-only HEAD | grep -iE 'postmortem|incident|archive|deprecat'git log -i --grep='archiv' --date=short --format='%ad %h %s'git log --diff-filter=A --date=short --format='%ad %s' -- PATHcurl -s https://pypi.org/pypi/PACKAGE/json | python3 -c "import json,sys; d=json.load(sys.stdin); print(d['info']['summary'])"curl -s https://registry.npmjs.org/PACKAGE | python3 -c "import json,sys; t=json.load(sys.stdin)['time']; print(max(t.values()))"curl -s https://hub.docker.com/v2/repositories/ORG/IMAGE/tags?ordering=last_updatedcurl -s https://proxy.golang.org/MODULE/@v/list