Data loss on Flink job restart: two coordinators coexist
An operator's full timeline of a ten-minute coordinator overlap that silently dropped a batch, with the interleaving of instants reconstructed from marker files. Assigned, not fixed.
A decade of Uber's platform, reconstructed entirely from its own public repositories: what it built, what it gave away, what it archived on a single day in December 2019, and what it left standing with no commits and no warning label. The systems Uber donated are the ones still running in 2026; several of the systems it kept are quietly dead.
Past a certain size, every engineering organisation builds systems it could not buy. The expensive decision is not the build. It is deciding, five years later, which of those systems still earns its maintenance, and how to get out of the ones that do not.
Almost nobody publishes that second decision. Build announcements are a genre; retirement
notices are not. Uber is the rare case where the whole arc is legible, because it published
an unusually large fraction of its platform as open source and then, over ten years, let the
repositories record what happened next. There are 176 repositories in the live
uber organisation and 160 in a second organisation called
uber-archive, whose profile reads, in full, "Uber's open source projects
archive. Support and/or new releases are not available." Both counts were taken on
7 September 2026.
That ratio is the finding this guide is built around, and the sorting of the two piles is
the surprise. The Uber-originated software that is unambiguously alive in 2026 is almost
all software Uber gave to somebody else: Jaeger to the CNCF, Hudi to the Apache Software
Foundation, Horovod and Ludwig to the LF AI & Data Foundation. The software it kept has
fared worse. Cherami, its durable message queue, and Hyperbahn, the routing mesh for its
in-house RPC protocol, were both archived on 18 February 2021. Peloton, its unified
scheduler, has not taken a public commit since 1 October 2019 and still carries no
deprecation notice at all. And Cadence, the workflow engine it neither kept close nor
donated, was forked by its own creators into a venture-backed company and has since moved
out of the uber GitHub organisation entirely.
The corpus is the repository record and only the repository record. This session's network
policy reached github.com and refused every other host tested, including
uber.com. Uber's engineering blog, its conference talks and its papers are
therefore absent, and so is every internal system that was never published: Schemaless,
Docstore, Michelangelo, the domain-oriented service architecture, the move off its own data
centres. None of those appear here, and no claim in this guide should be read as covering
them.
Read positively, the constraint produced a better artefact than a blog survey would have. A company blog post is a sanctioned account written at the moment of maximum optimism about a system. A repository is an unsanctioned one, written continuously, and its most honest fields are the ones nobody edits on purpose: the date of the last commit, the presence or absence of an archive banner, the age of the oldest open pull request. Those fields are what this guide reads. They are also the fields your own architecture review should be reading about every dependency you are about to adopt.
Retired infrastructure leaves by one of three doors, and the sources call them nothing at all because each company invents its own vocabulary. This guide names them donated, deprecated and drifted. Donated means transferred to a foundation with maintainers outside the originating company. Deprecated means archived with a dated, visible notice. Drifted means still presented as a live project, with stars, a README in the present tense and no warning, while the commits stopped years ago. Drift is the one that costs a stranger real money, and it is the most common.
Reconstructing the platform from the repositories that implement it, layer by layer, with the fate of each layer attached. Every box below is a repository that exists and that was read for this guide.
By 2016 Uber had a full-stack, self-built platform, and it had built it in a specific and recognisable order: transport first, then coordination, then the things that ride on top. TChannel was the wire protocol, described in its own repository as "a multiplexing and framing protocol for RPC calls"; Hyperbahn was the router mesh in front of it, offering "service discovery and routing for large-scale microservice operations" plus timeouts, retries, load balancing, rate limiting and circuit breaking. Ringpop sat inside the application processes and provided "scalable, fault-tolerant application-layer sharding for Go applications" through gossip and a consistent hash ring. Cherami carried asynchronous work as "a distributed, scalable, durable, and highly available message queue system we developed at Uber Engineering to transport asynchronous tasks."
Every one of those four is now dead. Hyperbahn and Cherami were archived on the same day, 18 February 2021, which is itself informative: retirement at Uber happens in batches, on a date somebody scheduled, not continuously. Ringpop carries the parenthetical "(This project is no longer under active development.)" in its README but has never been archived, so it still appears in search results as a live library. TChannel's Go implementation has never been archived either, and the organisation listing shows it updated on 1 September 2026, which places a library in an odd state: the protocol client is nominally maintained while the router that gave the protocol its reason to exist has been read-only for five and a half years. Neither repository names a successor. Anyone reading only the repositories would not learn that the industry converged on gRPC.
The layer above is where Uber's building was most aggressive and where the record is most mixed. For scheduling it wrote Peloton, "a unified resource scheduler to co-schedule mixed types of workloads such as batch, stateless and stateful jobs in a single cluster", targeted at "millions of containers and tens of thousands of nodes" and built, decisively, on Mesos. The architecture in its README wires a host manager that "abstracts away Mesos details from other Peloton components" to Zookeeper and Cassandra. That is a 2018 bet against Kubernetes, made by a team with a credible reason to think Kubernetes would not reach their scale, and the public record of it ends on 1 October 2019 with a commit titled "Use mesos plugins to launch tasks in V0 API". No notice, no successor, no archive.
Two neighbouring components in the same layer went the other way, and the difference is instructive. Makisu, the unprivileged image builder "designed for unprivileged containerized environments such as Mesos or Kubernetes", was archived cleanly on 4 May 2021 once the ecosystem produced equivalents. Kraken, the peer-to-peer registry that distributes those images, is still receiving commits in 2026 and is one of the few Uber infrastructure repositories that publishes measured numbers: "in production at Uber since early 2018", "more than 1 million blobs per day", a peak of "20K 100MB-1G blobs in under 30 sec", and a 3 GB image reaching 2,600 concurrent hosts at p50 10 seconds, p99 18 seconds and p99.9 22 seconds. Kraken survived because no industry standard arrived to replace it. Makisu did not, because one did.
Uber's data platform is the clearest demonstration that the survival of a component has almost nothing to do with how good it was. Marmaray, "a generic Hadoop data ingestion and dispersal framework and library", was archived on 5 January 2024. AthenaX, the SQL streaming platform, last moved in June 2020. AresDB, "a GPU-powered real-time analytics storage and query engine" with 3,100 stars, took its last public commit on 23 April 2020 and has no archive banner. uReplicator, the Kafka replicator Uber wrote because MirrorMaker's rebalance behaviour did not hold up, is unarchived and unmarked. Chaperone, the Kafka audit system, stopped in January 2021.
Hudi, the one piece of that stack Uber gave to the Apache Software Foundation, is a different animal in 2026: an active project with a formal RFC process, 111 numbered design proposals, and a tracker in which production data-loss reports arrive weekly. It is demonstrably alive, which is what the donation bought. Section 4 shows what it cost.
A wire protocol is the component with the strongest industry gravity, because its value is entirely in how many other things speak it. Uber's protocol, its router and its application-layer sharding library all stopped inside the same window.
Read at: tchannel-go, hyperbahn, ringpop-go
Peloton, which competed with Kubernetes, is silent. Kraken, which competes with nothing in particular and solves a bandwidth problem the standard registries never addressed, still ships and still publishes benchmarks.
The longest-lived things Uber published are not distributed systems. The Go style guide has 17,700 stars, and NullAway, which promises build-time overhead "usually less than 10%", was still updated on 7 September 2026, nine years in.
Read at: uber-go/guide, NullAway
The last two years of the uber organisation look nothing like the first eight.
The recent repositories are SubmitQueue, "a high-performance speculative merge queue that
keeps your trunk consistently green at scale", updated 4 September 2026; tango,
which computes changed targets for it; hermetic_cc_toolchain for cross-compiling
under Bazel; scip-lsp. Alongside them sits ADR, "an enterprise
security system for AI agents", with an accompanying MLSys 2026 paper and a README saying it
is "deployed in production at Uber". Uber is no longer publishing distributed-systems
primitives. It is publishing monorepo build tooling and agent security. That is what a
platform organisation's output looks like once the primitives underneath it have become
somebody else's problem, which is the state most large engineering organisations are moving
toward and the state this whole guide is really about.
Four forks where the repository record shows both the choice and its consequence, each with the condition that would flip it for a reader facing the same call.
| Decision | Chosen | Outcome by 2026 | Flips when | Evidence |
|---|---|---|---|---|
| RPC transport | TChannel plus Hyperbahn | Router archived 2021-02-18; client drifted | A standard reaches 1.0 before your internal rollout completes | hyperbahn |
| Application sharding | Ringpop gossip and hash ring | "No longer under active development", never archived | Your sharding can move into the data layer or a service mesh | ringpop-go |
| Cluster scheduling | Peloton on Mesos | Last public commit 2019-10-01, no notice | The standard's published ceiling exceeds your projected fleet | commit history |
| Image distribution | Kraken, peer-to-peer | Alive, still publishing benchmarks | Registries solve your tail latency at 15k hosts, which they still do not | kraken |
| Image building | Makisu, unprivileged | Archived 2021-05-04 | An ecosystem builder ships the same unprivileged mode | makisu |
| Protobuf tooling | Prototool | Archived 2022-03-04, points to Buf | A funded competitor appears; hand over rather than compete | prototool |
| Tracing | Jaeger, donated to CNCF | Graduated Oct 2019, 23.2k stars, active | You cannot supply two organisations' worth of committers | graduation proposal |
| Workflow engine | Cadence, company org, no foundation | Forked into Temporal; repository left the uber org | The creators can raise money on it | temporal |
Four operator-filed incident reports and one standing design defect, grouped by failure class. Read the attribution carefully: none of these was written by Uber.
Uber has published no postmortem reachable in this corpus. A company that gave the industry Jaeger, Hudi, Horovod, Cadence and M3 has left the failure record of those designs to be written by other people, in other people's bug trackers, years after the design decision was made. Every incident below was filed by an operator running Uber-originated software, and two of them still have no maintainer response. If you are adopting a donated project, that tracker is your postmortem archive, and it is the thing to read before the README.
close() did not complete for another ten minutes because HDFS was slow. Both wrote to the same table. The new coordinator's write events overwrote buffered events belonging to the old coordinator's instant, and that batch was silently dropped.close() and a scheduling gap, which is why a load test never finds it.
Reconstructed from the timeline in
Apache Hudi issue 19357.ResourceExhausted triggers a long backoff curve whose side effect is desynchronising thousands of shards. A roughly sixty-second database switchover returned Unavailable and deadline errors instead. Only ResourceExhausted takes the long path, so every reader fell back to its fast path and discarded its accumulated backoff. On recovery the whole fleet became eligible at once, producing "an aggregate, fleet-wide synchronized burst whose combined read QPS exceeds the system persistence cap". Acknowledgement advancement competes through the same rate limiter as reads, so the queue could not drain.Hudi's maintainers ran a full triage of their open pull request backlog on 16 July 2026 and published the verdicts: 555 open pull requests, of which 166 were closed as obsolete, 166 were converted to issues because they were still relevant but too stale to rebase, and 223 were kept. The oldest reviewed request dated to 26 February 2021. That is five years of contributor effort, most of it now unusable, in the flagship project of Uber's donated data stack. Donation does not remove the maintenance cost of a system. It moves the cost to a group of volunteers whose funding you no longer control, and the backlog age is the number that tells you how that is going. Check it before you adopt, and check it again annually.
Everything quantitative in this corpus, with its source and its date. Two of these rows are derived arithmetic and are labelled as such; there are no cost figures at all, because the repository record contains none.
| Metric | Value | At | Context | As of | Source |
|---|---|---|---|---|---|
| Microservices instrumented for tracing | 2,700 | Uber | Measured; the only public count of Uber's service estate in this corpus | 2019-09 | CNCF proposal |
| Spans collected per day | ~10 billion | Uber | Measured, across multiple datacentres | 2019-09 | CNCF proposal |
| Engineers on Jaeger at graduation | 7 full time | Uber | Plus "7 committers from 2 organizations" | 2019-09 | CNCF proposal |
| Blobs distributed per day | > 1 million | Uber, busiest Kraken cluster | Measured by the project | 2026-09 | kraken README |
| Peak burst | 20k blobs / 30 s | Uber | Blobs of 100 MB to 1 GB | 2026-09 | kraken README |
| 3 GB image to 2,600 hosts | p50 10 s, p99 18 s, p99.9 22 s | Uber | Project benchmark, at the configured speed limit | 2026-09 | kraken README |
| Hosts supported per registry cluster | ≥ 15,000 | Uber | Claimed capability, not an independent measurement | 2026-09 | kraken README |
| Static-analysis build overhead | < 10% | Uber | "In our measurements"; the project's own figure, unverified elsewhere | 2026-09 | NullAway README |
| Repositories in the archive organisation | 160 | Uber | Counted from the listing header | 2026-09-07 | uber-archive |
| Repositories in the live organisation | 176 | Uber | Counted from the listing header; includes five archived in place | 2026-09-07 | uber |
| Archive to live ratio | 0.91 | Uber | Derived: 160 divided by 176. Excludes repositories deleted outright, so it understates retirement | 2026-09-07 | uber-archive |
| Same-day deprecation relabels | 13 | Uber | On the first page of the archive sorted by last push, all dated 3 December 2019 | 2026-09-07 | uber-archive |
| Hudi design proposals, total | 111 | Apache | 57 completed, 18 under review, 12 ongoing, 9 in progress, 13 abandoned | 2026-09-07 | RFC index |
| Hudi proposal abandonment rate | 12% | Apache | Derived: 13 abandoned of 111. A healthy figure for an open process, and a number most internal design processes cannot produce at all | 2026-09-07 | RFC index |
| Hudi open pull requests at triage | 555 | Apache | 166 closed obsolete, 166 converted, 223 kept; oldest from February 2021 | 2026-07-16 | issue 19306 |
| Peloton, days between last commit and check | ~2,533 | Uber | Derived: 1 October 2019 to 7 September 2026, with no deprecation notice on the repository | 2026-09-07 | commit history |
| Mobile framework scaling claim | hundreds of engineers | Uber | "has proven to scale to hundreds of engineers working on the same codebase"; a claim, not a measurement | 2026-09 | RIBs README |
The Jaeger row is the strongest number here, because it was filed with a foundation as part of a graduation review rather than published as marketing, but it is seven years old and describes a service estate that has certainly changed. The Kraken figures are the project's own benchmarks and no independent reproduction exists. The two derived rows show their arithmetic. There is no cost figure anywhere in this corpus, so any statement about what Uber's platform cost to build or saved by retiring would be invention.
Every source behind this page, graded, with what to carry forward. All of them resolve to one host, which is a real weakness of this corpus and is discussed in the ledger.
An operator's full timeline of a ten-minute coordinator overlap that silently dropped a batch, with the interleaving of instants reconstructed from marker files. Assigned, not fixed.
Failover recovers a checkpoint taken between reserving an instant and describing it, overwriting the live buffer and orphaning files that were already written.
The clearest metastable-failure write-up in this corpus. Long backoff was keyed to one error code, and that backoff was also the only thing desynchronising the fleet.
Replicas provisioned onto reused disks carrying stale index data but no metric data. Reads and writes both reported success. Open with no root cause since 2022.
The single richest document in this corpus and the only place Uber's own scale figures appear: production since 2015, over 2,700 microservices, about 10 billion spans a day, seven full-time engineers, seven committers from two organisations.
A public design process with a visible abandonment rate. The abandoned entries name real forks in the road, including a table management service and a secondary index.
The fork runs a public proposal process, structured by surface, that the parent project never published. Governance visibility is one of the things the fork actually changed.
The rare in-house component that both survived and publishes measured figures: over a million blobs a day, 20k large blobs in 30 seconds, and a 3 GB image reaching 2,600 hosts at p99 18 seconds.
160 repositories under a profile that reads "Uber's open source projects archive. Support and/or new releases are not available." Sorted by last push, thirteen of the first thirty share the date 3 December 2019.
The unified scheduler's last public commit is dated 1 October 2019. The repository has 647 stars, a full architecture section in the present tense, and no notice of any kind.
The routing mesh for Uber's own RPC protocol, offering discovery, timeouts, retries, load balancing, rate limiting and circuit breaking. Archived the same day as Cherami.
"A distributed, scalable, durable, and highly available message queue system we developed at Uber Engineering to transport asynchronous tasks." No successor named.
The only retirement in this corpus that hands users somewhere to go: "We recommend checking out Buf, which is under active development."
"(This project is no longer under active development.)" sits inside the README of a repository that is not archived, has 880 stars, and still appears live in search.
A GPU-powered analytics engine with 3,100 stars whose last commit was "add middleware support (#365)" on 23 April 2020, and which carries no deprecation notice.
"Temporal is a mature technology that originated as a fork of Uber's Cadence. It is developed by Temporal Technologies, a startup by the creators of Cadence." 22,900 stars against Cadence's 9,400.
The repository now serves from cadence-workflow/cadence, and the
organisation profile describes the project as "Originally developed by Uber", with
maintainers, a technical steering committee and a CNCF Slack channel.
Closed unmerged on 19 December 2018. Reviewers objected to exposing raw-history APIs publicly and to history deserialisation crossing a layer boundary.
A standing design defect in the failover path: history is deleted after archival fails, and the target bucket cannot differ per region.
166 closed as obsolete, 166 converted to issues, 223 kept. The oldest reviewed request was opened in February 2021.
Distributed training with 14,700 stars, hosted by the LF AI & Data Foundation since the December 2018 announcement. Alive, and no longer Uber's problem.
"In our measurements, the build-time overhead of running NullAway is usually less than 10%." Last updated 7 September 2026, the day this guide was compiled.
"A high-performance speculative merge queue that keeps your trunk consistently green at scale", with companion repositories for changed-target computation and hermetic cross-compilation. Updated 4 September 2026.
"An enterprise security system for AI agents", covering employee-facing coding agents and customer-facing support agents, "deployed in production at Uber", with a paper accepted to MLSys 2026.
The denominator for the archive ratio, and the place to see that only a handful of retirements were archived in place rather than moved to the graveyard organisation.
"Jaeger is a distributed tracing platform created by Uber Technologies and donated to Cloud Native Computing Foundation." Incubating from September 2017, graduated October 2019, 23,200 stars.
This method transfers. Six rungs, from an afternoon's inventory to a standing control in your architecture review. The line from exercise to real work is at rung four.
Name, owner, last commit date, archived flag, open issue count, oldest open pull request. A listing page and an afternoon is enough for a hundred repositories.
Done when: you have one table and nothing in it is "unknown". Teaches: how much of your surface area you had no owner for.
Everything with no commit in eighteen months and no archive flag or deprecation notice. For Uber that set includes Peloton, AresDB, Ringpop, Zanzibar and Neuropod.
Done when: every entry has a named owner or a scheduled archive date. Teaches: that drift is a default, not a decision.
Do it without consulting the architecture diagram. Then compare. The gaps are the parts of your platform that exist only in someone's head or in a wiki nobody updates.
Done when: you can say which layer each repository serves. Teaches: where documentation and implementation have diverged.
Sort by comment count, not by date. Cadence pull request 1322 took fifteen minutes to read and disclosed a known durability gap in cross-region replication.
Done when: you can name one failure mode each project knows about and has chosen to carry. Teaches: that rejection reasons are the design document nobody writes.
Search the tracker for the phrase "data loss" and group what comes back by mechanism. Hudi's twelve open reports collapse into two or three classes, and both classes have the same shape as the failures in section 4.
Done when: you have a class list and know which classes your usage is exposed to. Teaches: that a tracker is a failure catalogue with no index.
Two paragraphs: the condition under which this system should be retired, and the successor a reader should move to. Put it in the README while the system is healthy, because nobody writes it afterwards. Prototool has one; Peloton does not.
Done when: it is merged. Teaches: that the exit note is the cheapest artefact in this entire guide and the one most consistently missing.
The searches that actually produced this page. Most of them are repository URLs rather than search-engine queries, which is the point: the archaeology is in the listings and the filters, not in the search box.
github.com/<org>-archivegithub.com/orgs/<org>/repositories?type=archived&sort=updatedgithub.com/orgs/<org>/repositories?sort=updated"this project is deprecated and not maintained" <org>github.com/<org>/<repo>/commits/master"This repository was archived by the owner on""no longer under active development" site:github.com"we recommend checking out" deprecated site:github.comgithub.com/<org>/<repo>/pulls?q=is:pr+is:closed+is:unmerged+sort:comments-descgithub.com/<org>/<repo>/issues?q=is:issue+"data loss"github.com/<org>/<repo>/issues?q=is:issue+outage+OR+"root cause"path:rfc README.md status abandonedgithub.com/cncf/toc/tree/main/projectsgraduation proposal adoption "per day" site:github.com"in production at" "since" README site:github.comgithub.com/<org>/<repo>/blob/master/docsEvery artefact fetched for this guide. All were reachable on
7 September 2026. The full ledger, with the quote supporting each claim, is in
sources.md beside this file.