Changing what the services stand on: ten years of ByteDance

The territory

An organisation past a certain size cannot ask its services to change. So when the thing they stand on becomes the bottleneck, it swaps the thing and keeps the shape of the hole. This is the record of a company that did that eighteen times.

Minor versions between upstream Kubernetes and the base of the company's published distribution

kubewharf/enhanced-k8s, checked 2026-10-09

go1.28

Build tag at which the fast JSON path switches itself off and delegates to the standard library

bytedance/sonic compat.go, 2026

3y 4m

From shipping the Thrift JIT serialiser to deleting its code

cloudwego/frugal releases, 2022 to 2025

10,000+

Dependent repositories cited to the Go team when asking for a removed runtime symbol back

golang/go issue 71672, 2025-02

State the problem without naming a technology and it sounds almost administrative. A company has thousands of services written by thousands of engineers. Something underneath all of them, the network layer, the serialiser, the cluster's metadata store, the scheduler, turns out to be the thing capping throughput or cost. Rewriting the services is not available: there are too many, they are owned by too many teams, and the win does not belong to any of those teams. So the platform group does the only thing that scales. It builds a replacement for the layer below that answers to the same interface, and it ships that replacement as a dependency bump.

ByteDance has done this more publicly and more often than anyone. Its two platform organisations on GitHub, CloudWeGo for the runtime and KubeWharf for the cluster, between them publish a replacement for Go's network package, Go's JSON encoder, Go's base64 encoder, Go's HTTP server, Apache Thrift's code generator, protocol buffers' generated code, Unix domain sockets, etcd, the Kubernetes scheduler, the Kubernetes API server's client path, and Kubernetes Federation. Each one presents the interface of the thing it replaces. That is not a coincidence of style, it is the strategy, and it is written into the repository descriptions: base64x is described as a "High performance drop-in replacement of the encoding/base64 library", and KubeBrain as "a component that implements the storage server interface required by the API Server".

The surprise in the record is not that the strategy worked. It is that it worked so well in one half of the stack that the gravity reversed. Go's own runtime source now names these packages in its comments as a compatibility constraint on the allocator, on memmove, on slice growth and on map iteration, under a heading the Go authors call the hall of shame. In February 2025 the Go team restored a runtime symbol it had removed, and backported the restoration to a patch release, because a ByteDance library had depended on it for two years and, in the words of its maintainer on the thread, "sonic is dependent over 1w+ repos". Meanwhile the other half of the same strategy, the Kubernetes half, reaches through a private fork rather than a published seam, and its public distribution is still built on a Kubernetes patch release from September 2022. Same company, same move, opposite outcomes, and section 06 argues the difference is entirely about which kind of seam was used.

Scope, and what this session could not reach

This guide is built from repositories, release metadata, issue and pull-request threads and package registries, because that is all the network policy on this run permitted. Every engineering-blog, conference-video, preprint and digital-library host attempted was refused at the proxy, so there is no blog tier, no talk tier and no paper tier in the evidence wall, and no cost figure anywhere in the guide. ByteDance also publishes no public incident reviews for these systems, so every breakage here is an upstream or downstream build failure rather than an account of the company's own production. What that record can show is the decision and its price. What it cannot show is what any of this felt like to operate inside ByteDance, and the guide does not guess.

Figure 1 · Two stacks, one move

Every box on the right replaces the box on its left while answering to the same interface, which is why the services above never had to change. Compiled from the repository descriptions and READMEs of cloudwego and kubewharf, checked 2026-10-09.

Diagram source

Every substitution has the same five parts

Read eighteen of these next to each other and the common shape is exact. Four of the five parts are what makes the move safe. The fifth is where the whole bill arrives.

Lay the repositories side by side and the anatomy repeats. There is a published interface that someone else is obliged to keep stable, and the replacement answers to it: sonic presents Marshal and Unmarshal, KubeBrain presents the gRPC storage interface the API server already speaks, KubeGateway's control plane is, in its own design document, "equivalent to a complete kube-apiserver" so that "you can use client-go to make configuration changes directly without additional SDK". There is a faster implementation behind it. There is a fallback to the original, which is the part most readers miss and the part that makes the strategy survivable: Kitex, the RPC framework built on netpoll, also ships a package whose doc comment reads "Package gonet contains server and client implementation for go net", and sonic ships a compat.go whose entire body delegates to encoding/json, and its published documentation states the rule plainly: "On non-sonic-supporting environment, the implementation will fall back to encoding/json". Worth noticing in the same reference: full compatibility is an opt-in. ConfigDefault aims "at efficiency and safety", and it is ConfigStd that aims "at being compatible with encoding/json", so a caller who swaps the import and nothing else has not asked for the original's behaviour. There is a version bound, usually encoded in a build tag or a README table. And then there is the fifth part, the reach-through: the places where the published interface was not enough and the replacement went around it into the internals of the thing it was replacing.

The reach-through is where the difference between the two halves of this stack lives. On the Go side it is //go:linkname and unsafe.Pointer. Netpoll's own documentation is explicit about the consequence: it ships separate files behind //+build race and //+build !race "to avoid DATA RACE detection in some code", because "the epoll uses unsafe.Pointer to access the struct pointer, in order to improve performance" and that "is beyond the detection range of the race detector". On the Kubernetes side the reach-through is a patched fork of the platform itself, tracked in a repository, kubewharf/enhanced-k8s, whose stated job is to track "all enhanced patches to the KubeWharf Kubernetes". Katalyst, the resource-utilisation system, states the dependency plainly: "Katalyst runs on a KubeWharf enhanced kubernetes cluster". You cannot run the published artefact on upstream Kubernetes.

Figure 2 · The anatomy of a layer substitution

Four parts make the swap invisible to callers. The fifth, drawn as the hexagon, is the reach-through into internals the interface does not cover; it is the only one that generates a liability, and in this record it generated all of them. Reconstructed across the CloudWeGo and KubeWharf repositories.

Diagram source

The runtime stack, 2021 to 2023

Netpoll went public on 2021-06-23, Kitex on 2021-07-09, sonic v1.0.0 on 2021-12-31, Hertz on 2022-05-31, frugal on 2022-05-16, dynamicgo in May 2023. All of them are substitutions, and all of them reach through. The README rationale is the same every time: the standard library was built for a different shape of workload. Netpoll's names it exactly, that Go's net "is designed for blocking I/O APIs, so that the RPC framework can only follow the One Conn One Goroutine design".

Dates from the Go module proxy: netpoll v0.0.1.

The cluster stack, 2022 to 2024

KubeZoo (multi-tenancy gateway) in December 2022, Katalyst in February 2023, KubeGateway, KubeBrain, Godel and KubeAdmiral across the same window. The forcing function is stated once, in KubeBrain's README: Kubernetes' "official stable operation scale is limited to 5K nodes", which is "still insufficient for applications with millions of machine nodes". Everything else in KubeWharf follows from trying to get past that number.

First releases from the module proxy; KubeBrain README.

The retreats, 2024 to 2026

The frugal JIT was capped at Go 1.23 in August 2024, made optional in November 2024, disabled by default in January 2025 and deleted in September 2025. Its successor, prutal, is "a pure Go alternative to protocol buffers" with an explicit list of what it does not implement. The Rust attempt, Volo on Monoio, restarts the same stack in a language where the company owns the whole thing, and declines to compare itself with the Go one.

frugal releases; volo README.

The decisions that matter

Four forks, each with the stated reason it went that way and the condition that would reverse it. Three of the four conditions have since been met in public.

Decision: replace the language's network layer, or live with one goroutine per connection?

Chosen

Netpoll, a non-blocking event loop, from 2021-06
Two stated reasons: context-switch cost at high connection counts, and net.Conn having "no API to check Alive", which makes a correct RPC connection pool hard to build

Rejected

Go's net, kept only as a secondary transport
Existing event-loop libraries, dismissed in the README as focused on "scenarios like Redis, HAProxy" rather than RPC

Flips when

Connection counts are low. The company's own Rust runtime benchmark reports that "in the case of a single core and very few connections, Monoio's latency will be higher than Tokio", which is the same model losing on the same axis
You need Windows, which netpoll lists under "Unsupported"
You need the race detector to be telling you the truth

Decision: reach through the published API into the runtime's internals, or stay inside it?

Chosen

Reach through. linkname into mallocgc, memmove, growslice, mapassign, morestack_noctxt and more, plus a JIT emitting machine code
Measured return: frugal claims "about 2.5x to 3.7x faster than Apache Thrift (TBinaryProtocol)" on its own benchmark

Rejected

A reflection-only implementation, which is what frugal eventually shipped in v0.2.0 for non-amd64 architectures
Code generation, rejected for a stated workflow reason: "No more meaningless diff of generated code in code review"

Flips when

Upstream starts enforcing the seam. Go did, in 1.23, through issue 67401: "all //go:linkname usage must be in the Handshake form". Within thirteen months frugal had capped the JIT at Go 1.23, added a NoJIT option, disabled it by default and deleted the code

Decision: replace the cluster's consensus store, or add more clusters?

Chosen

Both, on separate axes. KubeBrain replaces etcd behind the API server's storage interface and delegates durability to a distributed key-value store; KubeAdmiral federates clusters
Stated reason: the 5,000-node stable ceiling

Rejected

Staying on etcd and scaling out clusters only, which the README rejects for cost of managing "N clusters"
A store of their own: KubeBrain is "Stateless" and "does not actually store the data"

Flips when

You need the guarantee etcd actually sells. KubeBrain's own README still carries two unchecked TODO items: "Guarantee consistence in critical cases" and "Jepsen Test". Below roughly 5,000 nodes there is no case for taking that trade

Decision: carry a patched fork of the platform, or express the change as an extension?

Chosen

A fork, tracked in kubewharf/enhanced-k8s, with the kubelet and API server patched for colocation and for API-server sharding
Godel is offered as "a potential substitute for the Kubernetes scheduler", with the caveat that "the framework interface deviates slightly"

Rejected

Scheduler plugins and device plugins alone, which cannot see NUMA topology and reclaimed capacity the way Katalyst needs

Flips when

Always, if you are not ByteDance. The published distribution is pinned to Kubernetes v1.24.6, released 2022-09-21, whose line's last patch shipped 2023-08-23, while upstream is at v1.37.1. Godel declares support for 1.21.4 to 1.24.6, KubeAdmiral for 1.16 to 1.24. A fork you cannot rebase is a fork you cannot publish

Figure 3 · Should you replace the layer below?

The only branch that matters is the second one, whether the seam you need is part of a contract upstream is obliged to keep. Derived from the four decisions above and from the outcomes in section 04.

Diagram source

Decision	Chosen	Rejected	Because	Evidence
Network layer for RPC	netpoll event loop	Go `net`, kept as secondary	Goroutine-per-connection cost; no liveness check on `net.Conn`	README, checked 2026-10
JSON codec	sonic, JIT plus SIMD	`encoding/json`, kept as fallback	CPU cost of reflection at fleet scale	compat.go build tag
Thrift serialisation	frugal, no generated codec	Apache Thrift codegen	2.5x to 3.7x on its own benchmark; no codegen diffs in review	README benchmark
HTTP server	hertz, forked from fasthttp	`net/http`	Internal requirements plus fasthttp's allocation model	README
Local IPC	shmipc over shared memory	Unix domain sockets	Copy elimination on large packets	README benchmark
Cluster metadata store	KubeBrain over TiKV or badger	etcd	5,000-node stable ceiling	README
API-server client path	KubeGateway, layer 7	Direct client-to-apiserver connections	TCP connection count per apiserver instance	design.md
Scheduler	Godel, two-tier unit and pod	kube-scheduler	Throughput, and batch semantics for offline work	README
Multi-cluster control	KubeAdmiral, from KubeFed v2	Writing one from scratch; Karmada	Keep the native API; reuse federated types	README
Second-generation stack	Volo on Monoio, in Rust	Continuing to fight the Go runtime	Thread-per-core and io_uring, with no work stealing to design around	README

What broke, and it was never the fast path

Nothing in the public record says a replacement was slower or wrong under load. Every failure is about the seam: upstream closing it, the toolchain losing sight through it, or the fork on the far side of it freezing.

Three classes account for all of it. Class A, upstream closes the seam. The reach-through is only tenable while the project you reached into does not mind. Go decided it did mind, in issue 67401, opened by Russ Cox in May 2024 with the Go 1.23 milestone and the goal that "all //go:linkname usage must be in the Handshake form: both sides must agree to use linkname for a given symbol". Class B, the safety nets go first. A substitution that uses unsafe and emits machine code is invisible to, and sometimes fatal to, the tools built to catch its bugs. Class C, the fork freezes. A patched platform is maintainable in private and unshippable in public, and the record shows exactly where the published artefacts stopped.

Postmortem

Class A: Go 1.23 forbade the reach, and the build stopped linking

AssumptionThat //go:linkname into the standard library's internals was a permanent facility, because nothing had ever taken it away.

What happenedA reporter testing Go 1.23 rc1 hit the linker error "invalid reference to encoding/json.safeSet". Sonic had linknamed a private variable inside the very package it was replacing. The same break arrived through Hertz for a second reporter the next day.

Blast radiusBuild-time, every dependent, on a release candidate. Opened 2024-06-25, labelled a known issue the following day, closed 2024-07-16.

FixRemove the linknames to encoding/json and carry the data locally.

Design ruleA reach-through is a dependency on a decision nobody has made yet. Inventory yours, and put the upstream issue that would end them on a watch list, because the thread that will break you is usually open for a year first.

Sourcebytedance/sonic issue 660, 2024-06

Postmortem

Class A: the symbol came back, and upstream paid for it

AssumptionOn the Go side, that removing an unused internal linkname is a safe cleanup, because the tooling can see who imports it.

What happenedA Go change removed runtime.lastmoduledatap (and, a reviewer noted, moduledataverify1 and morestack_noctxt). Builds on Go 1.24 failed with "invalid reference to runtime.lastmoduledatap". The package was absent from Go's own hall-of-shame list because, as a Go maintainer observed on the thread, only three packages import sonic/loader directly. The real exposure was indirect: its maintainer replied that "sonic is dependent over 1w+ repos".

Blast radiusEvery Go 1.24.0 build of anything depending on sonic, directly or transitively. Reported 2025-02-12, fixed upstream 2025-02-18, backported to the 1.24 release branch at a Go maintainer's request.

FixUpstream restored the linknames in a commit titled "runtime: add some linknames back for github.com/bytedance/sonic". Downstream, the maintainer's interim advice was to "add compile flags -ldflags=-checklinkname=0", which is to say, switch off the check that caught it.

Design ruleDirect importers are the wrong blast-radius metric for a library that others vendor. Count transitive dependents before you delete an exported-by-accident symbol, and expect a popular reach-through to become upstream's compatibility problem whether upstream agreed to it or not.

Sourcegolang/go issue 71672, 2025-02; sonic issue 738

Postmortem

Class A: and again, on the next release, and the one after

AssumptionThat once a toolchain break is fixed, the class is closed.

What happenedBuilding against Go 1.26 rc1 failed at compile time with "internal/rt/stubs.go:33:22: undefined: GoMapIterator", a runtime type the new release no longer defines. A commenter added "Same problem with 1.25 actually". After the nominal fix landed, the same reporter built again and hit further undefined symbols in loader/internal/abi, a comment that received no reply before the issue was closed.

Blast radiusOpened 2025-12-28, closed 2026-01-23. A downstream packaging change dated 2026-09-09 bumped the dependency so that a tree "compiles on Go 1.26", nine months after the first report.

FixA per-release support pull request, which is now a recurring item rather than an incident.

Design rulePrice a reach-through as an annual subscription, not a one-off. Three of the last four Go releases have a filed build-breakage issue against this library, so the honest line item is "one engineer-week per upstream minor release, forever".

Sourcebytedance/sonic issue 895, 2025-12

Postmortem

Class B: the race detector was the thing that crashed

AssumptionThat a drop-in replacement is drop-in under test instrumentation too.

What happenedUnmarshalling a one-byte input into an int under go test -race died with "checkptr: converted pointer straddles multiple allocations", inside the JIT's stack-map builder during package initialisation. A second reporter confirmed it on a different Go and sonic version, and noted that without -race the tests pass.

Blast radiusAny dependent running its own test suite with the race detector on. Reported 2023-02-15, fixed 2023-02-20.

FixA patch to the stack-map builder. The structural version of the same problem is netpoll's, which ships race-tagged variants of its files specifically "to avoid DATA RACE detection in some code".

Design ruleWhen you replace a layer with one that uses unsafe, you are also replacing your customers' ability to test their own code. Either keep the instrumented path on the fallback implementation, or expect to be blamed for every race your users cannot see any more.

Sourcebytedance/sonic issue 363, 2023-02; netpoll explain.md

Source

Class B: the fast path has an expiry date and fails open

AssumptionThat the reason you adopted the replacement is still in force on the toolchain you are building with today.

What happenedSonic's fast path is selected by the build tag (amd64 && go1.17 && !go1.28) || (arm64 && go1.20 && !go1.28). Outside that window, compat.go compiles instead, sets apiKind = UseStdJSON and forwards every call to encoding/json. On Go 1.28, or on any other architecture, the library silently becomes the thing it replaced. No error, no warning, no build failure.

Blast radiusLatent and unbounded: a toolchain upgrade can remove the performance the capacity model was built on, and nothing in the system reports it. No public incident describes this happening, which is consistent with a failure nobody can see.

FixNone is needed in the library; the fallback is correct behaviour. The gap is observability, and the library exposes apiKind for exactly this.

Design ruleAny substitution with a fallback needs a startup assertion that the fast path is actually selected, exported as a metric. A silent fallback converts a build-time problem into a capacity-planning problem that will surface as a latency regression nobody can attribute.

Sourcebytedance/sonic compat.go, checked 2026-10-09

Postmortem

Class B: forking a server means owning its request parser

AssumptionThat forking a mature third-party HTTP server inherits its security review along with its code.

What happenedVersions of Hertz before 0.3.1 contained a path traversal in normalizePath, the function that canonicalises request paths. It was published as CVE-2022-40082 with a CVSS of 7.5 and classified CWE-22.

Blast radiusAll versions below 0.3.1, disclosed to the advisory database 2022-09-29, roughly four months after the framework's first public release on 2022-05-31.

FixPatched in 0.3.1.

Design rulePath and header normalisation is the highest-risk code in any HTTP server, and it is the code a fork is most likely to touch. If you fork a server, the first thing to port is not a feature, it is the upstream project's normalisation test corpus.

SourceGHSA-c9qr-f6c8-rgxf, 2022-09-29

Source

Class C: the published platform stopped at a 2022 release

AssumptionThat a fork carried for an internal fleet can also be the artefact other people run.

What happenedThe release table in enhanced-k8s lists exactly one row: KubeWharf v1.24.6-kubewharf.0.1 on Kubernetes v1.24.6, built with go1.18.6. Katalyst requires that distribution. Godel declares support for 1.21.4 to 1.24.6; KubeAdmiral for 1.16 to 1.24. Upstream Kubernetes is at v1.37.1, and the 1.24 line's final patch shipped 2023-08-23. The fork itself is not dead: its branch list shows sharding-1.32.3, sharding-1.33.1 and sharding-1.34.1 updated through September 2025. The newer bases exist, on a differently named line, and were never published as a distribution.

Blast radiusEvery external adopter, permanently. The components are open source and effectively unrunnable on a supported Kubernetes.

FixNone visible. KubeAdmiral's forked upstream, KubeFed, has itself published nothing since v0.10.0 on 2022-08-10, so there is no rebase target there either.

Design ruleThe version floor of your fork is the version floor of everything that depends on it, and it ratchets in one direction. Before forking a platform, write down which of your patches could instead be an extension point proposed upstream, and treat the ones that cannot as a decision to stop upgrading.

Sourcekubewharf/enhanced-k8s; branch list; upstream latest

Source

Class C: the consensus store was replaced before consistency was settled

AssumptionThat matching an interface is the hard part of replacing a store, and the guarantees follow.

What happenedKubeBrain implements the API server's storage interface and keeps watched data "in the memory of the master node", with leader election built on the storage engine's own primitives and a "Retry queue" that, in the design document's words, covers "the small number of operations that return non-deterministic errors" and "guarantees final consistency through asynchronous retry corrections". The README's TODO list still carries two unchecked items: "Guarantee consistence in critical cases" and "Jepsen Test".

Blast radiusUnknown and unpublished, which is the point. The repository's last release activity is 2024, so the public artefact has been sitting at that state for two years.

FixStated as future work. The published benchmark is honest in the other direction: "KubeBrain on TiKV can outperform etcd in read and write performance, while deletion performance needs to be further optimized".

Design ruleAn interface-compatible store is not a guarantee-compatible store. When you swap the thing that holds consensus, the acceptance test is a partition test, not a throughput test, and if the partition test is on the roadmap then so is the data loss.

Sourcekubewharf/kubebrain README and docs

Figure 4 · How one deleted symbol became a Go patch release

The sequence that matters is the last two messages: the fix landed upstream, not downstream, because by 2025 the reach-through was load-bearing for the ecosystem. Reconstructed from golang/go issue 71672 and sonic issue 738.

Diagram source

Numbers you can plan against

Everything quantitative the repositories actually state, with the date it was stated. The gaps are as informative as the figures.

Metric	Value	At	Context	As of	Source
Kubernetes stable node ceiling, as the reason for replacing etcd	5,000	KubeBrain README	Stated as "official stable operation scale", against a target of "millions of machine nodes"	checked 2026-10	README
Cluster size KubeGateway targets	>1,000 nodes	KubeGateway README	"massive large-scale kubernetes clusters"	checked 2026-10	README
TCP connection reduction per apiserver instance	≥1 order of magnitude	KubeGateway README	Claimed, not independently measured; mechanism is HTTP/2 connection pooling at the proxy	checked 2026-10	README
Thrift serialisation speedup	2.5x to 3.7x	frugal vs Apache Thrift TBinaryProtocol	Author's benchmark, go1.23.6, Xeon Gold 5118; marshal medium 3,669 ns/op vs 9,343 ns/op	checked 2026-10	README
Rust runtime throughput vs Tokio, 4 cores	~2x	Monoio	Measured "on the ByteDance production network", 10GbE, Linux 5.15	2021-12-01	benchmark.md
Rust runtime throughput vs Tokio, 16 cores	~3x	Monoio	Same run; the gap grows with core count because the model avoids work stealing	2021-12-01	benchmark.md
Where the thread-per-core model loses	1 core, few connections	Monoio	"Monoio's latency will be higher than Tokio", attributed to io_uring versus epoll	2021-12-01	benchmark.md
Shared-memory IPC vs Unix socket, 64-byte ping-pong	7,740 vs 5,523 ns/op	shmipc-go	The replacement is slower on small packets; Xeon Platinum 8260	checked 2026-10	README
Shared-memory IPC vs Unix socket, 4 KB	660.78 vs 343.44 MB/s	shmipc-go	Crossover is between 64 B and 4 KB; at 4 MB the figure is 2,686 MB/s	checked 2026-10	README
Dependent repositories cited for sonic	10,000+	sonic maintainer, to the Go team	Claim made while asking for a removed runtime symbol back; direct importers of the affected subpackage were three	2025-02	golang/go 71672
Go releases with a filed build-breakage issue	3 of the last 4	sonic issue tracker	Go 1.23 (issue 660), 1.24 (738), 1.26 (895); a comment on 895 reports 1.25 as well	2024-06 to 2026-01	issue list
Lifetime of the Thrift JIT, first release to code deletion	3 years 4 months	frugal	v0.1.0 2022-05-16; capped at Go 1.23 in v0.2.0 2024-08-08; optional v0.2.2; off by default v0.2.4; removed v0.3.0 2025-09-09	2025-09-09	releases
Kubernetes base of the published distribution	v1.24.6	kubewharf/enhanced-k8s	Released 2022-09-21; the 1.24 line's last patch, v1.24.17, shipped 2023-08-23	checked 2026-10	upstream tag
Upstream Kubernetes at time of writing	v1.37.1	kubernetes/kubernetes	13 minor versions ahead of the published fork base	2026-09-23	releases
Last release of the federation project KubeAdmiral continues	v0.10.0	sigs.k8s.io/kubefed	No later version in the module proxy, so the fork has no rebase target	2022-08-10	module proxy

Read these carefully

Every performance figure above is the author's own benchmark, published in the repository that benefits from it, and none has an independent replication in this corpus. Treat the speedups as upper bounds measured on hardware the vendor chose, and note that two of the three are honest about where the replacement loses, which is itself a reason to trust them more than a figure without a crossover point. The operational numbers are scarcer still: no service count, no queries per second, no fleet size and no cost appears anywhere in the public record for either stack, so nothing here can tell you the scale at which the trade became worth making. The only dated figure that is not self-reported is the Kubernetes version gap, and that one is arithmetic.

The lesson: substitute through contracts, fork through nothing

One company, one strategy, two outcomes. The variable that separates them is not scale, language, or engineering quality.

Put the two halves of this record side by side. On the Go side, ByteDance replaced the network layer, the JSON codec, the base64 codec, the HTTP server and the Thrift serialiser, and ten years on it still ships all of them, with a user base large enough that the Go project restored a deleted runtime symbol and backported the restoration rather than let those users break. On the Kubernetes side it replaced the metadata store, the scheduler, the client path and the federation layer, and the public artefacts are pinned to a Kubernetes release from September 2022 that upstream stopped patching in August 2023, while the newer internal bases sit on branches that were never published as a distribution.

The difference is not effort. Both halves are substantial, both are actively developed, and the Kubernetes half is arguably the more sophisticated engineering. The difference is the kind of seam each one reached through. The Go substitutions reach through a contract that a third party is obliged to keep, and when one of them reached past that contract into runtime internals, the breakage was loud, dated, fixable in a week, and eventually paid for by the upstream project. The Kubernetes substitutions reach through a patched fork, which is a contract with nobody. There is no issue to file, no symbol to ask for back, and no version of the fix that arrives from outside. The only available response to an upstream release is to do the rebase yourself, and the published record is what it looks like when that work loses to something more urgent.

So the rule an architect can carry into a design review is narrow and testable. Replacing the layer below your services is the right move when the layer above the seam you need is something somebody else has promised not to break, and it is a liability the moment your replacement needs anything that is not in that promise. Three practical consequences follow, and all three are visible in this record. Write down every reach-through you have, because it is a dependency on a decision that has not been made yet, and Go made that decision in 1.23. Ship the fallback to the original before you ship the replacement, then assert at startup that the fast path is selected, because sonic's fallback is correct and silent and that combination hides a capacity change. And before you fork a platform, sort your patches into the ones that could be an upstream extension point and the ones that cannot, because the second pile is not a patch set, it is a decision to stop upgrading, and the version floor it creates propagates to everything built on top.

The one line

A substitution is only as durable as the promise it stands on. Reach through a public contract and you can change the floor under thousands of services for a decade without touching one of them; reach through a fork and you have swapped a performance problem for a permanent, compounding upgrade debt that you will eventually pay in versions you can no longer leave.

The evidence wall

Fifty artefacts were read; the twenty-four that carry the argument are below, graded. Two tiers are empty on purpose and the reason is in the first card.

Figure 5 · Ten years, dated from the registries rather than the blog posts

The shape worth noticing is the third panel: once Go closed the seam in 1.23, the component that had reached furthest into the runtime was capped, switched off and deleted, and its successor started again inside the language. Dates from the Go module proxy and the frugal release notes, queried 2026-10-09.

Diagram source

Source code Methodology note2026-10

Why there are no blogs, talks or papers in this wall

This run's egress policy allowed GitHub, the Go module proxy, the crates index, PyPI and apache.org, and refused every other host attempted, including the company's own documentation site, InfoQ, USENIX, arXiv and the Wayback Machine. Rather than cite material that could not be fetched, the guide is built only from what could be read and re-checked.

Carry forwardA repository-only reconstruction is strong on decisions and dates and blind to operating experience. Read this guide for the first and go elsewhere for the second.

https://github.com/cloudwego

Source code CloudWeGo2021-06

netpoll: the stated case for replacing the language's network package

The README gives two reasons, both architectural rather than micro-optimising: Go's net forces "the One Conn One Goroutine design", and net.Conn "has no API to check Alive", which makes a correct RPC connection pool hard. It also declines the existing event-loop libraries as aimed at Redis-shaped and HAProxy-shaped workloads.

Carry forwardThe best argument for substituting a layer is a missing capability in its interface, not a benchmark. "No liveness check" is the kind of reason that stays true as hardware changes.

https://github.com/cloudwego/netpoll

Source code CloudWeGo2026-10

netpoll: the race detector is deliberately blinded

A documentation file titled "DATA RACE EXPLAIN" states that netpoll ships separate files behind //+build race and //+build !race "to avoid DATA RACE detection in some code", because the epoll path uses unsafe.Pointer and that "is beyond the detection range of the race detector". The design document the README links to alongside it contains the single line "# TODO".

Carry forwardAsk of any high-performance replacement which of your existing tools stop working against it. The answer is rarely in the README and is sometimes in a file called explain.md.

https://raw.githubusercontent.com/cloudwego/netpoll/main/docs/reference/explain.md

Source code CloudWeGo2022

kitex: the escape hatch back to the standard library, kept for four years

Kitex is built on netpoll, but it also carries pkg/remote/trans/gonet, whose package comment is "Package gonet contains server and client implementation for go net." The alternative transport has been in the tree since 2022 and is still there.

Carry forwardThe thing that makes a layer substitution reversible is a maintained second implementation on the original. Budget for it from the start; retrofitting one after the replacement has diverged is a rewrite.

https://raw.githubusercontent.com/cloudwego/kitex/develop/pkg/remote/trans/gonet/trans_server.go

Source code CloudWeGo2025-06

kitex PR 1767: rebuilding the standard-library transport, closed unmerged

A maintainer opened a reimplementation of the gonet transport on 2025-04-23 that, per a review comment, would "use bufiox to replace netpoll" in that path. It drew forty comments on connection-state checking and goroutine lifetime, reached about 59% patch coverage, was marked draft on 2025-06-06 and closed on 2025-06-09 with no stated reason.

Carry forwardA closed-unmerged PR on the escape hatch is a signal about how much the escape hatch is actually exercised. If nobody will finish the fallback, the fallback is decorative.

https://github.com/cloudwego/kitex/pull/1767

Source code CloudWeGo2022-08

kitex PR 585: xDS support built, milestoned, then dropped

"add xds module to manage xDS resources retrieved from control plane. Support traffic route, timeout config and service discovery based on xDS." The PR was added to the v0.4.0 milestone on 2022-08-19, removed from it on 2022-08-22, and closed the next day without comment, after an approval had been dismissed as stale.

Carry forwardA company that substitutes its own layers tends to resist the industry's shared control plane, because the shared one assumes components it has already replaced. Watch for that pattern before standardising on a mesh.

https://github.com/cloudwego/kitex/pull/585

Source code ByteDance2026-10

sonic compat.go: the build tag that is also an expiry date

The fast path compiles only for amd64 from Go 1.17 and arm64 from Go 1.20, and only below go1.28. Outside that window compat.go sets apiKind = UseStdJSON and forwards every call to encoding/json. The README adds that Go 1.24.0 specifically is unsupported and suggests -ldflags="-checklinkname=0".

Carry forwardGrep your dependencies for upper version bounds in build tags. Each one is a date on which a performance assumption quietly stops holding, with no error to alert on.

https://github.com/bytedance/sonic/blob/main/compat.go

Source code Go package index2026-10

sonic's published interface: the fallback is documented, the compatibility is opt-in

The package reference states "On non-sonic-supporting environment, the implementation will fall back to encoding/json" and that "Sonic DOES NOT ensure to support all environments, due to the difficulty of developing high-performance codes". It also shows that ConfigDefault aims "at efficiency and safety" while a separate ConfigStd aims "at being compatible with encoding/json".

Carry forwardRead which config of a drop-in replacement is the compatible one. If byte-for-byte parity with the original is a non-default option, then swapping the import is a behaviour change, not a dependency bump.

https://pkg.go.dev/github.com/bytedance/sonic

Postmortem ByteDance / Go2024-06

sonic 660: Go 1.23 rc1 broke the link to a private standard-library symbol

The reporter hit "invalid reference to encoding/json.safeSet" and noted that "Go 1.23 no longer allows //go:linkname * runtime.* link instructioins". A second reporter arrived through Hertz the following day. A maintainer replied "We are handling this. Please wait for a while" and tagged it a known issue; closed three weeks later.

Carry forwardThe first warning of a seam closing is a release-candidate build failure filed by an external packager, not by you. Test against upstream release candidates if you depend on anything that linknames.

https://github.com/bytedance/sonic/issues/660

Postmortem Go project2025-02

golang/go 71672: upstream restored a deleted symbol and backported it

A removed //go:linkname on runtime.lastmoduledatap broke Go 1.24 builds. A Go maintainer observed the package had never been listed in the hall of shame because only three packages import it directly; the sonic maintainer answered that "sonic is dependent over 1w+ repos". The symbols were added back and the fix was backported to the 1.24 branch at a Go maintainer's request.

Carry forwardPast some adoption threshold your unsanctioned dependency becomes the upstream project's compatibility problem. That is leverage, but it is leverage you only get after the breakage, which is a poor plan.

https://github.com/golang/go/issues/71672

Postmortem ByteDance2025-02

sonic 738: the downstream half of the same break

"go 1.24 build failed: link: github.com/bytedance/sonic/loader: invalid reference to runtime.lastmoduledatap". The body's diagnosis is a guess, "It seems Go teams removes the //go:linkname on runtime.lastmoduledatap", and the maintainer's interim advice was a branch plus "-ldflags=-checklinkname=0". Opened 2025-02-12, closed 2025-03-08.

Carry forwardWhen the documented workaround for a dependency is to disable a toolchain safety check, that is the real cost of the dependency, and it belongs in the decision record rather than the troubleshooting page.

https://github.com/bytedance/sonic/issues/738

Postmortem ByteDance2025-12

sonic 895: Go 1.26 removed a runtime type the library relied on

"internal/rt/stubs.go:33:22: undefined: GoMapIterator" on Go 1.26 rc1, with a commenter adding "Same problem with 1.25 actually". After the nominal fix, the same reporter hit further undefined symbols in loader/internal/abi and asked "is this relevant?"; the issue was closed the next day without an answer.

Carry forwardBreakage of this class recurs on a schedule set by somebody else's release calendar. Treat it as recurring maintenance with an owner, not as a bug that gets fixed once.

https://github.com/bytedance/sonic/issues/895

Postmortem ByteDance2023-02

sonic 363: a one-byte input crashed the test suite under the race detector

"checkptr: converted pointer straddles multiple allocations", raised inside the JIT's stack-map builder during package initialisation, triggered by unmarshalling '0' into an int under go test -race. A second reporter confirmed on a different Go and library version and noted the tests pass without -race.

Carry forwardRun your own test suite with the instrumented build before adopting a JIT-based library, because the failure appears in your CI and looks like your bug.

https://github.com/bytedance/sonic/issues/363

Decision record Go project2024-05

golang/go 67401: the decision to close the seam, with an escape hatch

Russ Cox's proposal to "prevent new //go:linkname-based dependencies and contain existing ones", on the Go 1.23 milestone. The requirement is that "all //go:linkname usage must be in the Handshake form: both sides must agree", enforced by a new -checklinkname=1 default, with -checklinkname=0 left as the opt-out. Locked as resolved in February 2025.

Carry forwardThis is the template for closing an unsanctioned extension point without an immediate flag day: make the strict mode the default, keep the opt-out, and name the packages affected. Reuse it when you need to retire an internal API other teams reached into.

https://github.com/golang/go/issues/67401

Source code Go project2025

Go's runtime names these packages in its own source as a constraint

At the go1.25.0 tag, mallocgc carries the comment "should be an internal detail, but widely used packages access it using linkname. Notable members of the hall of shame include" followed by bytedance/gopkg, bytedance/sonic and cloudwego/frugal. memmove lists sonic and cloudwego/dynamicgo; growslice and reflect_growslice list dynamicgo; procPin, noescape and the sync.Pool cleanup hook list bytedance/gopkg.

Carry forwardIf you want to know which private internals of a platform have become load-bearing for the ecosystem, read the platform's source comments rather than its documentation. Maintainers annotate what they can no longer change.

https://github.com/golang/go/blob/go1.25.0/src/runtime/malloc.go

Source code CloudWeGo2025-09

frugal releases: the JIT capped, made optional, defaulted off, deleted

Four release notes tell the whole retreat: "feat(jit): go1.23 for the last supported version" with "new reflect impl for non-amd64 arch" (v0.2.0, 2024-08-08), "feat: add NoJIT option" (v0.2.2, 2024-11-28), "refactor: disable JIT by default" (v0.2.4, 2025-01-09) and "refactor: rm JIT code & clear CI" (v0.3.0, 2025-09-09).

Carry forwardThis is the cleanest public example of retiring an over-reaching optimisation: cap the supported range, add the opt-out, flip the default, delete. Thirteen months, four releases, no flag day.

https://github.com/cloudwego/frugal/releases

Case study CloudWeGo2026-10

frugal and shmipc: what the substitutions bought, with a crossover

Frugal reports "about 2.5x to 3.7x faster than Apache Thrift (TBinaryProtocol)" on go1.23.6. Shmipc reports that it is slower than a Unix socket at 64 bytes (7,740 against 5,523 ns/op) and roughly twice as fast at 4 KB (660.78 against 343.44 MB/s), reaching 2,686 MB/s at 4 MB.

Carry forwardTrust the benchmark that names the size at which the replacement loses. Then find the equivalent crossover for your own traffic mix before adopting, because for small-message workloads this particular substitution is a regression.

https://github.com/cloudwego/shmipc-go

Case study ByteDance2021-12

monoio: the same bet, remade in Rust, measured on their own network

"Our test is carried out on the ByteDance production network." Monoio reports roughly twice Tokio's peak throughput at four cores and close to three times at sixteen, and states where it loses: "in the case of a single core and very few connections, Monoio's latency will be higher than Tokio". The README also concedes that the unstable features and new I/O abstraction "may cause some compatibility problems".

Carry forwardThread-per-core wins where connection counts are high and work is uniform, and loses on small deployments. The axis to measure is cores times connections, not requests per second.

https://raw.githubusercontent.com/bytedance/monoio/master/docs/en/benchmark.md

Decision record KubeWharf2026-10

KubeBrain: replacing etcd behind its own interface, consistency pending

The README names the forcing function, Kubernetes' "official stable operation scale is limited to 5K nodes", and the design: a stateless component that "implements the storage server interface required by the API Server" and "does not actually store the data", with watched data held in the master node's memory and final consistency reached through an asynchronous retry queue. Two TODO items remain unchecked: "Guarantee consistence in critical cases" and "Jepsen Test".

Carry forwardWhen a store advertises interface compatibility, read its consistency tests before its benchmarks. Interface compatibility is a day-one property; the guarantee is what you need on the day of the partition.

https://github.com/kubewharf/kubebrain

Case study KubeWharf2026-10

KubeBrain benchmark: faster on reads and writes, slower on deletes

Three-node clusters, 70-byte keys, 512-byte values, 300 concurrent etcd clients. The stated conclusion: "KubeBrain on TiKV can outperform etcd in read and write performance, while deletion performance needs to be further optimized", justified on the grounds that Kubernetes' storage load "has a low percentage of deletion operations".

Carry forwardA replacement that is faster on the common operation and slower on a rare one is a reasonable trade, provided you have measured your own operation mix rather than inherited the claim.

https://raw.githubusercontent.com/kubewharf/kubebrain/main/docs/benchmark.md

Decision record KubeWharf2026-10

KubeGateway: an API-server-shaped proxy in front of the API server

The design document's load-bearing sentence is about compatibility, not performance: "The control plane of KubeGateway is equivalent to a complete kube-apiserver", so "you can use client-go to make configuration changes directly without additional SDK". The README claims it "converges the number of TCP connections on a single kube-apiserver instance by at least an order of magnitude" for clusters of more than 1,000 nodes.

Carry forwardThe cheapest configuration interface for a new infrastructure component is an interface your operators already have a client for. Shaping your control plane like the system it fronts removes an entire adoption cost.

https://raw.githubusercontent.com/kubewharf/kubegateway/main/docs/en/design.md

Source code KubeWharf2026-10

The version floor: one published distribution, pinned to 2022

enhanced-k8s lists a single release, v1.24.6-kubewharf.0.1 on Kubernetes v1.24.6 with go1.18.6. Katalyst requires it. Godel supports 1.21.4 to 1.24.6, KubeAdmiral 1.16 to 1.24. The fork's own branch list shows newer bases, sharding-1.32.3 through sharding-1.34.1, maintained into September 2025 but never released as a distribution.

Carry forwardCheck the version floor of any platform component before adopting it, and check it against the fork it requires rather than the component's own release date. An actively committed repository can still be unrunnable on a supported platform.

https://github.com/kubewharf/enhanced-k8s

Source code CloudWeGo2025-03

prutal: the next generation is pure Go and says what it does not do

"Prutal is a pure Go alternative to protocol buffers", aiming "to minimize code generation as much as possible". It publishes its own incompatibility list: the Opaque API is unsupported because "field presence lives in a bitmap the runtime does not maintain", and Clone, Merge, Equal and CheckInitialized are absent. The README states it "is NOT yet ready for production use".

Carry forwardThe honest form of a drop-in replacement is one that ships a list of the calls it does not implement. Demand that list; if the project will not write it, you will discover it one call at a time.

https://github.com/cloudwego/prutal

Vendor CloudWeGo2026-10

The company's own framing of the set

"CloudWeGo is an open-source middleware set launched by ByteDance that can be used to quickly build enterprise-class cloud native and AI native architectures. The common characteristics of CloudWeGo projects are high performance, high scalability, high reliability and focusing on microservices communication and governance."

Carry forwardNote what the framing omits: no scale figure, no service count, no adoption number. The repositories assert that these systems are "widely used inside ByteDance" and never quantify it, so neither does this guide.

https://github.com/cloudwego/.github

Build a substitution, then find out what it costs you

Six rungs. The crossing from exercise to real happens at rung four, where you reach past the interface and have to decide how to bound it.

Replace one standard-library function behind its own signature

Pick something small and well specified: base64, or a single JSON encode path. Expose exactly the original signature. Run the original package's own test suite against your implementation unchanged.

Done when: the upstream test suite passes against your code with no edits to the tests. Teaches: how much of the original's behaviour is specified by its tests rather than its documentation.

Add the fallback, and prove it engages

Put your fast path behind a build tag with an upper and lower bound, and compile the original behind the complement, as sonic does. Export which one is active as a package-level value and as a metric.

Done when: a test asserts the fallback is selected on an unsupported toolchain, and a startup log line names the active path. Teaches: that a silent fallback is a capacity regression nobody can attribute.

Build the differential fuzzer

Generate random inputs, run both implementations, compare outputs byte for byte and errors by class. Add every real payload shape you can get from production.

Done when: ten million generated inputs and your whole corpus agree, including on error text where callers match on it. Teaches: that "compatible" means compatible in the error paths, which is where reimplementations actually diverge.

Reach past the interface once, and bound it in the build

Use one unsafe or reflection-internal trick that the public API does not allow. Then write the version bound into the build constraint so an unknown toolchain fails to compile rather than compiling into undefined behaviour.

Done when: building on a toolchain outside the bound either fails loudly or selects the fallback, and never silently miscompiles. Teaches: the difference between a reach-through you can date and one you cannot.

Find where your replacement loses

Sweep input size and concurrency until the original wins, the way the shmipc and monoio benchmarks do. Publish the crossover number next to the headline number.

Done when: you can state, in one sentence, the workload at which your replacement is the wrong choice. Teaches: that the crossover point, not the peak speedup, is what makes a benchmark usable by somebody else.

Carry a fork across two upstream minor releases, and time it

Fork something you depend on, apply one real patch, then rebase onto the next two upstream minor releases. Record the hours, including the test failures you had to triage and the patches that no longer applied.

Done when: you have a measured per-release rebase cost and can multiply it by your platform's release cadence. Teaches: what the Kubernetes half of this record cost, in the only unit that matters at review time.

Keep hunting

The queries that found the material above. The repository-shaped ones are the ones worth keeping, because they work when a company has no engineering blog or when you cannot reach it.

Finding the reach-throughs

"hall of shame" linkname path:src/runtime
"should be an internal detail" repo:golang/go
"-ldflags=-checklinkname=0"
"drop-in replacement" language:Go stars:>500
path:**/compat.go "go:build" "go1."

Finding the retreat, not the launch

repo:<org>/<repo> "rm JIT" OR "disable" "by default" in:title
repo:<org>/<repo> is:pr is:closed is:unmerged sort:comments-desc
repo:<org>/<repo> is:issue "go1.2" build failed
"last supported version" in:readme OR in:description

Dating a decade without a blog

https://proxy.golang.org/<module>/@v/list
https://proxy.golang.org/<module>/@v/<version>.info
https://github.com/<org>/<repo>/branches/all
https://github.com/orgs/<org>/repositories?type=all&sort=created

Finding the version floor

"supports Kubernetes versions from" in:readme
"enhanced kubernetes" OR "enhanced patches" in:readme
org:<org> "Jepsen" in:file
https://github.com/advisories?query=<org>

References

CloudWeGo, netpoll Repository README. Checked 2026-10-09.
CloudWeGo, netpoll, DATA RACE EXPLAIN In-repository documentation. Checked 2026-10-09.
CloudWeGo, Kitex Repository README. Checked 2026-10-09.
CloudWeGo, Kitex, gonet transport server Source file, copyright 2022. Checked 2026-10-09.
CloudWeGo, Kitex PR 1767, new implementation of gonet transport Closed unmerged 2025-06-09. Checked 2026-10-09.
CloudWeGo, Kitex PR 585, proxyless support via xDS api Closed unmerged 2022-08-23. Checked 2026-10-09.
CloudWeGo, Hertz Repository README. Checked 2026-10-09.
GitHub Advisory Database, GHSA-c9qr-f6c8-rgxf (CVE-2022-40082) Published 2022-09-29. Checked 2026-10-09.
ByteDance, sonic Repository README. Checked 2026-10-09.
Go package index, github.com/bytedance/sonic Published API reference, including the documented fallback. Checked 2026-10-09.
Go package index, kitex gonet transport The standard-library transport, published as part of the module. Checked 2026-10-09.
ByteDance, sonic issue 660, Incompatible with Go 1.23 Opened 2024-06-25, closed 2024-07-16. Checked 2026-10-09.
ByteDance, sonic issue 738, go 1.24 build failed Opened 2025-02-12, closed 2025-03-08. Checked 2026-10-09.
ByteDance, sonic issue 895, Build with Go 1.26 fails Opened 2025-12-28, closed 2026-01-23. Checked 2026-10-09.
ByteDance, sonic issue 363, crash in test with -race Opened 2023-02-15, closed 2023-02-20. Checked 2026-10-09.
ByteDance, sonic issue tracker Sorted by reactions. Checked 2026-10-09.
Go project, issue 67401, cmd/link: lock down future uses of linkname Opened 2024-05-15, Go 1.23 milestone, locked 2025-02-26. Checked 2026-10-09.
Go project, issue 71672, add linkname runtime.lastmoduledatap back Opened 2025-02-12, closed 2025-02-18. Checked 2026-10-09.
Go project, src/runtime/malloc.go at go1.25.0 Hall-of-shame comment on mallocgc. Checked 2026-10-09.
Go project, src/runtime/stubs.go at go1.25.0 Hall-of-shame comments on memmove, memequal and noescape. Checked 2026-10-09.
Go project, src/runtime/slice.go at go1.25.0 Hall-of-shame comments on makeslice and growslice. Checked 2026-10-09.
CloudWeGo, frugal Repository README and benchmark. Checked 2026-10-09.
CloudWeGo, frugal releases v0.1.0 2022-05-16 to v0.3.1 2025-11-07. Checked 2026-10-09.
Go module proxy, frugal v0.3.0 Timestamp 2025-09-09. Queried 2026-10-09.
Go module proxy, netpoll v0.0.1 Timestamp 2021-06-23. Queried 2026-10-09.
CloudWeGo, prutal Repository README. Checked 2026-10-09.
CloudWeGo, base64x Repository description. Checked 2026-10-09.
CloudWeGo, shmipc-go Repository README and benchmark. Checked 2026-10-09.
CloudWeGo, Volo Repository README. Checked 2026-10-09.
ByteDance, Monoio Repository README. Checked 2026-10-09.
ByteDance, Monoio performance test and comparison Dated 2021-12-01. Checked 2026-10-09.
CloudWeGo, Eino Repository README; first release 2024-12-11. Checked 2026-10-09.
CloudWeGo, organisation profile Checked 2026-10-09.
KubeWharf, KubeBrain Repository README and TODO list. Checked 2026-10-09.
KubeWharf, KubeBrain benchmark In-repository documentation. Checked 2026-10-09.
KubeWharf, KubeBrain architecture In-repository design document. Checked 2026-10-09.
KubeWharf, Godel scheduler Repository README. Checked 2026-10-09.
KubeWharf, KubeAdmiral Repository README. Checked 2026-10-09.
KubeWharf, Katalyst core Repository README. Checked 2026-10-09.
KubeWharf, KubeGateway Repository README. Checked 2026-10-09.
KubeWharf, KubeGateway design In-repository design document. Checked 2026-10-09.
KubeWharf, enhanced Kubernetes Release table. Checked 2026-10-09.
KubeWharf, Kubernetes fork branch list Checked 2026-10-09.
Kubernetes, release v1.24.17 Final 1.24 patch, 2023-08-23. Checked 2026-10-09.
Kubernetes, latest release v1.37.1, 2026-09-23. Checked 2026-10-09.
Go module proxy, sigs.k8s.io/kubefed v0.10.0 Timestamp 2022-08-10; latest version published. Queried 2026-10-09.
KubeWharf, malachite Public archive, last updated 2023-05-15. Checked 2026-10-09.