Zalando has kept the configuration of its production Kubernetes fleet in a public repository since 2017, and merged a change into it on the morning this guide was written. This is a decade of one retailer's compute platform read from that record: how a change is promoted across the fleet, which in-house components survived nine years and which were handed back to upstream, and what the two outages the company published say about shared fate inside a cluster.
A company that runs its own compute platform signs up for a change stream it does not control. This guide is about the machinery that absorbs it.
The operating system underneath needs patching. The scheduler ships a new minor version several times a year and drops an API two versions later. The credential path gets deprecated by the cloud provider. The routing layer grows a feature the network team wants. None of it can stop while several hundred delivery teams keep shipping on top. So the interesting engineering question for a platform group is not which components are in the architecture diagram. It is the change path: how do you move a modification through a fleet of production environments quickly enough to stay current, and slowly enough to catch the one change that takes everything down?
Almost every organisation answers that question privately, which is why the public
literature on platform engineering is mostly conference talks about the destination and
almost nothing about the traffic. Zalando answers it in the open, through a decision made
around 2017 that appears to have been about community goodwill and turned into something
more useful: the configuration that defines its production Kubernetes clusters lives in a
public repository, it is the operational article rather than a sanitised copy, and it is
still being changed. The newest merge on the dev branch when this guide was
compiled closed pull request 12233, on 8 September 2026.
That makes an unusual corpus available. Instead of reading what a platform team says it does, you can read the branch layout it promotes through, the decision records that argue for that layout, the pipeline definition that gates each change, the sixty-three component directories that constitute the platform today, and the pull requests that were closed without merging. Two things are missing from it, and both are stated plainly in this guide: there is no cost data, and the published failure record is four commits long and stops in June 2019.
Scope. This guide covers the platform layer at one company between 2015 and 2026: cluster lifecycle, change promotion across a fleet, the controller inventory, ingress and identity, and node provisioning. It does not cover Zalando's application architecture, its data platform, its spend, or any comparison against peers, because the session that built it could reach code hosts and nothing else. Where a claim rests on one artefact, the text says so.
Four moving parts hold the fleet: a registry of clusters, a controller that reconciles them, a repository with one branch per risk appetite, and a belt of in-house controllers hanging off the scheduler's extension points.
The shape is easiest to see from the oldest decision record in the repository. ADR-001
settles where the current state of a cluster is written down, and the answer is neither
the cluster nor the repository: the Cluster Lifecycle Manager stores three version strings
in the status field of a Cluster record in a separate Cluster
Registry, called next_version, current_version and
last_version. Each is the git commit of the channel concatenated with a hash
of that cluster's own configuration. The record explains the encoding choice in one word,
KISS, and lists the consequence that matters: because
last_version is the last working version, a rollback is a version string, not
a restore procedure.
That gives the reconciliation loop its shape. The lifecycle manager polls two sources, the registry for which clusters exist and which channel each one follows, and the repository for what that channel currently says. It is described as reentrant, so it can be killed at any point and will continue a cluster update from where it left off, because all state is in the registry and in git. Nothing durable lives in the process. That single property is what makes a fleet upgrade restartable, and it is the reason this design survived a decade of Kubernetes releases while the components inside the clusters were replaced underneath it.
What the clusters contain has changed almost completely. The admin guide still in the
repository describes the 2017 arrangement: clusters provisioned with an adapted
kube-aws, nodes running CoreOS Container Linux, a master auto-scaling group of
two nodes, three worker groups split one per availability zone, and more than eighty
production clusters. Read the configuration defaults today and none of the specifics
survive. Nodes are provisioned by Karpenter with an explicit exclusion list of instance
families, the machine image is Zalando's own Ubuntu 22.04 build carrying Kubernetes
v1.36.3, and a second provider named zalando-eks sits alongside the original
one, with its own control plane logging, IP family and IAM proxy settings.
The migration to managed control planes is visible in the configuration rather than
announced in it. There is a skipper_eks_migration_enabled flag, and a sibling
cluster concept: when a cluster has a sibling, the ingress layer publishes an additional
service so traffic can be served across the pair while workloads move. Read alongside pull
requests from 2026 that set the Karpenter maximum pods per node for EKS clusters,
the reading is straightforward and worth stating as an inference rather than a report:
after roughly nine years of operating its own control planes, the platform group is moving
clusters onto the cloud provider's managed offering, in place, one sibling pair at a time.
The layer worth naming is the third box inside the cluster. Sixty-three directories in
cluster/manifests deploy into every cluster, and they divide cleanly into two
kinds. Some are upstream or cloud-vendor software taken as-is: CoreDNS, kube-proxy,
flannel, Prometheus, metrics-server, the vertical pod autoscaler, the EBS and EFS CSI
drivers, the AWS load balancer controller, the NVIDIA device plugin, Karpenter. The rest
are controllers Zalando wrote, and every one of them sits at an extension point the
platform already offered rather than replacing a core component: an ingress controller
that reconciles AWS load balancers from Ingress and RouteGroup objects, an IAM controller
and proxy that turn pod identity into cloud credentials, a downscaler that turns
non-production workloads off out of hours, a janitor that deletes what nobody owns, a
stackset controller that expresses a deployment as a set of versioned stacks, a metrics
adapter that lets any metric drive autoscaling, and a pod disruption budget controller
that stops rolling node updates from taking a service below its floor.
Call it the controller belt. Its defining property is that no item in it forks the platform. The belt is bolted to interfaces the scheduler publishes, which is why the same belt survived the move from Container Linux to a custom image, from auto-scaling groups to Karpenter, and now from self-managed control planes to a managed one. Three of these components have outlived their original context entirely: Skipper, the layer-seven proxy that does the in-cluster routing, states in its README that it handles 350,000 route definitions and daily traffic between 500,000 and 7 million requests per second across more than one hundred production clusters, and Patroni, written at Zalando to run PostgreSQL clusters, is now described in its own README as having started as a fork of Governor and no longer carries the company's name.
The in-house controller that programs AWS load balancers is still deployed in 2026, with flags for network load balancer zone affinity, ALPN policy and container network interface target mode. The cloud vendor's own load balancer controller sits beside it. Both are present because the fleet currently spans two providers.
Evidence: ingress-controller deployment
The 2017 guide documents kube2iam, a third-party daemon, for giving pods a cloud role. The 2026 manifest set carries a Zalando-written IAM controller and an IAM proxy alongside it. This is the one place where the company took work back rather than handing it over, and the credential path is exactly where a general solution stops fitting a specific organisation.
Evidence: 2017 admin guide, 2026 manifests
The downscaler, the janitor, the pod disruption budget controller and the stackset controller do not add capability to Kubernetes. They encode company policy about cost, ownership, availability floors and what a deployment means, in a form the scheduler enforces continuously rather than a wiki page nobody reads.
Evidence: cluster/manifests, checked 2026-09-08
Five decision records, one pipeline definition and a set of closed pull requests, read for the option that lost and the condition that would bring it back.
| Decision | Chosen | Rejected or replaced | Stated reason | Evidence |
|---|---|---|---|---|
| Where cluster state lives | Three version strings in the registry record | Multiple typed fields in the registry schema | Keeps the provisioner's format out of the registry schema, and keeps it human readable for debugging | ADR-001 |
| How many channels | Three, named dev, alpha and stable | An unbounded set of per-cluster pins | Answers the questions ADR-003 left open about promotion and control | ADR-005 |
| Promotion authority | Automatic to alpha after a wait, human approval into stable | Fully automatic promotion to production | ADR-005 requires the final pull request to carry additional human approval | ADR-005 |
| Node operating system | Zalando-built Ubuntu 22.04 image, Kubernetes v1.36.3 | CoreOS Container Linux | Not stated in the corpus; the 2017 guide praises Container Linux and the 2026 configuration no longer references it | config-defaults.yaml |
| Instance selection | Karpenter with an excluded families list | AMD-based c5a, m5a, r5a and six further families, excluded by default | Not stated; the exclusion is expressed as configuration rather than argued in a record | config-defaults.yaml |
| Control plane ownership | A second provider, zalando-eks, with sibling clusters for migration | Continuing to self-manage every control plane | Not stated in the corpus; inferred from the provider switch, the migration flags and 2026 pull requests scoped to EKS clusters | config-defaults.yaml |
ADR-005 decided on three channels. The pipeline definition triggers on four branch names, and the branch listing shows dev, alpha, beta and stable all updated on 8 September 2026, plus two standing promotion branches called dev-to-alpha and beta-to-stable. The record was never amended. This is the ordinary fate of decision records, and it is an argument for reading the pipeline next to the record rather than instead of it: the record tells you what was argued, the pipeline tells you what is true.
The company published exactly two postmortems, both in the first half of 2019, both about the same underlying mistake in two different disguises.
Read together, both incidents are the same failure class in different clothing: a component that everything depends on was configured for the average case, and the average case was not the failure case. Neither was a bug in the software. The DNS outage came from a memory limit derived from historical usage, and the build outage from an upstream default nobody had revisited. Both fixes changed a number; only one of them, the move to a per-node resolver, changed the architecture. That distinction is the most useful thing an architect can take from this pair.
The third failure class in this corpus is not written up anywhere, because it is a class of near-miss that the change path is designed to absorb. It is visible instead in the closed pull requests. One from March 2025 is titled as an update of Karpenter to a specific version with the parenthetical note that this should fail, which is a deliberate negative test of the gate rather than a change anyone wanted merged. Others, such as a February 2026 attempt to enable a new instance family for the default node pool and a March 2026 proposal to disable AMD instance types in production by default, were opened, tested and closed without merging. The record of what a platform team decided not to do is thin everywhere in public engineering writing; here it is a filter on a pull request list.
The postmortem directory has received four commits in its life, between 4 February and 29 June 2019, and none since. Zalando has run this fleet for another seven years and has certainly had incidents; it has simply stopped publishing them, while continuing to publish every configuration change. Treat the two write-ups here as a sample of one year's operational reality, not as the decade's. If you are evaluating a platform approach on public evidence, this asymmetry is the trap: the change record survives because it is a by-product of doing the work, and the failure record does not, because writing it is separate work that no pipeline enforces.
Everything quantitative in the corpus, with the date attached, because three of these figures are stale and the page should say which.
| Metric | Value | Kind | Context | As of | Source |
|---|---|---|---|---|---|
| Production clusters | more than 80 | Measured | Stated in the admin guide during the early migration | 2017 era | Admin guide |
| Clusters operated by the lifecycle manager | 200+ | Claimed | README, whose upgrade claim stops at Kubernetes v1.24 | v1.24 era | Lifecycle manager README |
| Clusters running the layer-seven proxy | more than 100 | Claimed | Skipper README, undated in the document itself | Checked 2026-09-08 | Skipper README |
| Route definitions in the proxy | 350,000 | Claimed | Shop frontend routing | Checked 2026-09-08 | Skipper README |
| Daily request rate through the proxy | 500k to 7M per second | Claimed | Serving 15,000 ingress objects and 3,750 route groups | Checked 2026-09-08 | Skipper README |
| Unit cost of routing | under 5 cents per million requests | Claimed | The single cost figure in the entire corpus, method not published | Checked 2026-09-08 | Skipper README |
| Kubernetes minor versions crossed in place | 32 | Derived | v1.4 named as the oldest cluster's starting point, v1.36.3 built into the current node image | 2026-09-08 | config-defaults.yaml |
| Staleness of the lifecycle manager README | 12 minor versions | Derived | v1.24 claimed there against v1.36.3 in the configuration | 2026-09-08 | README |
| Node agent request rate limit | 5 raised to 50 per second | Measured | Default caused delays over five minutes when twenty pods start together | 2019-06-21 | Pull request 2247 |
| Resolver memory limit at failure | 100Mi raised to 2000Mi | Measured | Set from historical usage, killed under a query spike | 2019-01-07 | Postmortem |
| Promotion time, merge to stable | 4 days | Measured | The node agent fix, merged 21 June and stable on 25 June 2019 | 2019-06 | Pull request 2247 |
| Platform components per cluster | 63 | Measured | Directory count under cluster/manifests | 2026-09-08 | Manifests |
Only four rows here are measurements from an incident or a merged change. The cluster counts and the proxy figures are self-reported in README files that carry no date, and the routing cost figure names no method, no denominator and no accounting boundary, so it is a claim rather than a benchmark. The two derived rows are arithmetic on version strings and are labelled as such. What nobody has published anywhere in this corpus: the current cluster count, node count or spend, the promotion interval actually configured between channels, and any incident frequency after 2019. If you plan against this platform's numbers, plan against the four measured rows and treat the rest as direction.
Every source behind this page, graded and dated. Two tiers are absent and the absence is explained below rather than hidden.
A complete incident write-up with timeline, six contributing factors, lessons and action items. Names the memory limit, the query amplification from the default search domain setting, and the monitoring path that shared the failure domain.
Short and unusually precise: a default rate limit on the node agent, exposed by pod churn on a single node, surfacing as a credential failure three components away. Links the permanent fix as a configuration change.
Defines the three version strings and argues for an opaque concatenated format so the registry never learns the provisioner's internals. Lists rollback via the last working version as a consequence rather than a feature.
Maps configuration channels onto git branches and draws the promotion graph. Ends with three explicit open questions about when promotion happens, how it is triggered and what approval it needs, which the later record answers.
Settles three channels, requires end-to-end tests on a freshly provisioned cluster per pull request, and puts a human approval on the final promotion. Also states that the first cluster updated after merge must be one with applications already running.
Shows the build triggering on dev, alpha, beta and stable plus pull requests, and a step that creates a cluster on the cloud provider for end-to-end tests using a named lifecycle-manager role.
Around 1,540 lines of fleet defaults: Karpenter feature gates and excluded instance families, a second provider named zalando-eks, control plane logging and IP family settings, proxy migration flags, and the machine image carrying Kubernetes v1.36.3.
The whole platform inventory in one listing: upstream components taken as-is next to in-house controllers for ingress, identity, downscaling, cleanup, disruption budgets, stacksets and metrics adaptation.
Deployment arguments show network load balancer zone affinity, ALPN policy, deregistration delay, container network interface target mode and a pod label selector pointing at the proxy. Nine years of accumulated operational detail.
dev, alpha, beta and stable all updated on the same day, alongside standing dev-to-alpha and beta-to-stable promotion branches and several component update branches named after container images.
The structural fix from the June 2019 incident, with the author's description of the default as ridiculously low. Merged on 21 June and promoted through alpha and beta to stable by 25 June.
Titled as an update of the node provisioner to a specific version with the note that this should fail. Closed without merging. Evidence that the gate is itself tested rather than assumed.
Opened and closed without merging in March 2026, in the same period as several other node provisioning experiments. The exclusion list in the defaults file shows which of these arguments ended up as configuration.
Four commits in total, between 4 February and 29 June 2019, by two authors. Nothing since, while the repository as a whole took a change today.
The team's own operational account from the early migration: more than eighty production clusters, one cluster per cloud account, an adapted provisioning tool, Container Linux nodes, and five stated goals including no manual operations and no pet clusters. Now a historical document contradicted by the configuration beside it.
States that the component has been developed internally since January 2017, operates more than two hundred clusters, and has carried the oldest cluster from Kubernetes v1.4 to v1.24 by continuous update. Describes the reentrant design and non-disruptive rolling node updates.
Self-reported production figures: 350,000 route definitions, more than one hundred production clusters, daily traffic between 500,000 and 7 million requests per second, 15,000 ingress objects and 3,750 route groups, at a stated cost below five cents per million requests.
The PostgreSQL high-availability template written at Zalando now describes itself as formerly known as Zalando's Patroni and as a fork of Governor, with no company ownership language and support for several coordination stores.
The command line tool of the pre-Kubernetes in-house platform, carrying a shouted notice that the project is not actively maintained any more. Its final tag is dated March 2023, years after the successor platform was in production.
The tool that generated and executed CloudFormation templates for the pre-Kubernetes platform still describes what it does at length, under a single shouted line saying the project is not actively maintained any more. No date, no successor named, no migration guidance.
There are no papers and no conference talks in this wall, and no engineering blog posts from any organisation. That is a property of the session rather than of the topic: the network policy in force reached code hosts and nothing else, so material on video platforms, conference sites and company blogs could not be fetched, and this skill forbids citing anything not retrieved. The lifecycle manager README names a 2018 conference talk about exactly this system, which is the obvious next thing to watch and is deliberately not cited here. Nor is there any independent measurement or peer comparison; every number above comes from the company that runs the system.
Six rungs. The line between exercise and production sits between rungs three and four, where the change gate stops being advisory.
Take one non-production environment and move every component it runs into a single directory of manifests plus one defaults file. Nothing clever, no promotion yet.
Done when: you can delete the environment and rebuild it from the repository alone. Teaches: which parts of your platform exist only in someone's shell history.
Write the smallest loop that reads a list of environments and the repository, computes a version string from both, applies the difference and records the version it reached plus the last one that worked.
Done when: killing the loop mid-apply and restarting it converges without human help. Teaches: why the version string is a concatenation and why rollback is a string rather than a procedure.
Create a second branch, point one environment at each, and promote by merge. Write down, as a record, who may promote and after how long.
Done when: a change reaches the second environment only via the first. Teaches: that risk appetite is a property of an environment, not of a change.
Wire the pull request to create an environment from scratch, run the platform's own conformance suite plus one test per policy controller you rely on, and destroy it.
Done when: a deliberately broken change is rejected without a human noticing it. Teaches: the cost of the gate, which is the number that decides whether this model is available to you at all.
Point the first automatic update at an environment that has real workloads and has been upgraded before, not a fresh one, and watch service level indicators across the update rather than the apply exit code.
Done when: an upgrade that passes on a clean environment and fails on a used one is caught here. Teaches: that upgrade defects live in accumulated state.
Take the single alarm that says the environment is serving traffic and run it from somewhere that shares no dependency with the environment, name resolution included. Then run a game day that kills the cluster-wide resolver.
Done when: the outside alarm fires before anyone reports the outage. Teaches: the January 2019 lesson at first hand, for the price of one afternoon.
This page was built without a single engineering blog post. These are the queries and paths that produced it, which work on any organisation that operates in the open.
path:docs/postmortemspath:docs/dev-guide/arch filename:adr"this should fail" is:pr is:closed is:unmergedfilename:delivery.yaml OR filename:config-defaults.yaml/commits/<branch>/<path> for the first and last commit to a directory/branches/all to see which channels are still moving/tags on the superseded project, for the real end-of-life dategrep the defaults file for a version string, then diff it against every README claim