Evidence ledger
20 sources
Checked 08 Sep 2026
Evidence ledger
One row per claim in Changing a platform without stopping it: a decade of Zalando's cluster fleet: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
Field guide: Changing a platform without stopping it (2026-09-08). One row per claim. Quotes are copied, not paraphrased. Every URL was fetched during the session that produced the guide, on 2026-09-08.
Access note. The session's network policy permitted code hosts and nothing else. Every other host tried, including engineering blogs, conference sites, video platforms, paper archives and the Internet Archive, was refused at the egress proxy with a 403 on CONNECT. The corpus is therefore entirely first-party repository material from one company, with no independent measurement, no talks and no papers. That limitation is stated in the guide itself.
| # | Org | Title | Tier | Published | Checked | URL | Claim taken from it | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 1 | Zalando | kubernetes-on-aws repository | source | live | 2026-09-08 | https://github.com/zalando-incubator/kubernetes-on-aws | The production cluster configuration is public and still changing | Landing page: "17,765 Commits" on branch dev; newest merge commit dated Sep 8, 2026 closes "pull request #12233" |
| 2 | Zalando | kubernetes-on-aws, README description | source | live | 2026-09-08 | https://github.com/zalando-incubator/kubernetes-on-aws | The repository is the real configuration, not a sample | README: "configuration templates to provision Kubernetes clusters on AWS using Cloud Formation and Ubuntu Linux" |
| 3 | Zalando | Total DNS outage in Kubernetes cluster | postmortem | 2019-02-04 (commit) | 2026-09-08 | https://github.com/zalando-incubator/kubernetes-on-aws/blob/dev/docs/postmortems/jan-2019-dns-outage.md | One application's retry spike took out cluster-wide DNS for over an hour | "On Monday 7 January 2019, all web product and outfit pages of the Zalando Fashion Store were returning a high amount of errors to customers for over 1 hour." |
| 4 | Zalando | Total DNS outage in Kubernetes cluster | postmortem | 2019-02-04 | 2026-09-08 | https://github.com/zalando-incubator/kubernetes-on-aws/blob/dev/docs/postmortems/jan-2019-dns-outage.md | The memory limit was set from historical usage and was the binding constraint | "The CoreDNS pods were configured with a very small memory limit of 100Mi based on historical resource usage data." Manual fix: "changed the Memory requests/limits from 100Mi to 2000Mi" |
| 5 | Zalando | Total DNS outage in Kubernetes cluster | postmortem | 2019-02-04 | 2026-09-08 | https://github.com/zalando-incubator/kubernetes-on-aws/blob/dev/docs/postmortems/jan-2019-dns-outage.md | The design rule is locality, not headroom | "It should not be possible for a single application to \"DoS\" our DNS infrastructure in a way that impacts everything running in the cluster. The blast radius ... should at least be isolated to a single node in the cluster, ideally to single pods." |
| 6 | Zalando | Total DNS outage in Kubernetes cluster | postmortem | 2019-02-04 | 2026-09-08 | https://github.com/zalando-incubator/kubernetes-on-aws/blob/dev/docs/postmortems/jan-2019-dns-outage.md | Monitoring shared the failure domain | "As a side effect of the DNS outage our internal monitoring for the cluster was also completely down as it needs to talk to external services to trigger alerts and push metrics." |
| 7 | Zalando | Total DNS outage, query amplification | postmortem | 2019-02-04 | 2026-09-08 | https://github.com/zalando-incubator/kubernetes-on-aws/blob/dev/docs/postmortems/jan-2019-dns-outage.md | Default search-domain configuration multiplied every external lookup | "This results in 10 DNS queries for every single DNS lookup of an cluster-external name by the application." |
| 8 | Zalando | Build Errors of Continuous Delivery Platform | postmortem | 2019-06-28 (commit) | 2026-09-08 | https://github.com/zalando-incubator/kubernetes-on-aws/blob/dev/docs/postmortems/jun-2019-kubelet-qps.md | A default rate limit on the node agent became a capacity ceiling | "On 11 June 2019, CDP builds were failing for almost an hour ... The default Kubelet configuration has an extremely low rate limit on the requests to the API server" |
| 9 | Zalando | Build Errors of Continuous Delivery Platform | postmortem | 2019-06-28 | 2026-09-08 | https://github.com/zalando-incubator/kubernetes-on-aws/blob/dev/docs/postmortems/jun-2019-kubelet-qps.md | The permanent fix was a global configuration change | "the Kubelet QPS was increased from 5 (default) to 50 via a global cluster configuration" |
| 10 | Zalando | Commit history of docs/postmortems | source | 2019 | 2026-09-08 | https://github.com/zalando-incubator/kubernetes-on-aws/commits/dev/docs/postmortems | The published failure record is four commits long and stops in June 2019 | Commits: "Add Cluster DNS Outage PM" (Feb 4, 2019), "remove \" |
| 11 | Zalando | ADR-001: Store cluster versions in Cluster Registry | adr | undated, status accepted | 2026-09-08 | https://github.com/zalando-incubator/kubernetes-on-aws/blob/dev/docs/dev-guide/arch/cluster-lifecycle/adr-001.rst | Fleet state is three opaque version strings held outside the reconciler | "CLM will store the current configuration state under the status field of the Cluster resource in the Cluster Registry ... next_version ... current_version ... last_version"; rationale ends with "KISS" |
| 12 | Zalando | ADR-001 | adr | undated | 2026-09-08 | https://github.com/zalando-incubator/kubernetes-on-aws/blob/dev/docs/dev-guide/arch/cluster-lifecycle/adr-001.rst | Rollback is a version string | "The last version is also used for rolling back a cluster in case the new version is broken." |
| 13 | Zalando | ADR-003: Organize cluster versions in branches | adr | undated | 2026-09-08 | https://github.com/zalando-incubator/kubernetes-on-aws/blob/dev/docs/dev-guide/arch/cluster-lifecycle/adr-003-organize-cluster-versions.rst | Configuration channels are git branches, chosen for differing SLOs | "Cluster configuration channels will map to git branches in the configuration repository." The record leaves three questions open, each marked "TBD" |
| 14 | Zalando | ADR-005: How to use channels | adr | undated | 2026-09-08 | https://github.com/zalando-incubator/kubernetes-on-aws/blob/dev/docs/dev-guide/arch/cluster-lifecycle/adr-005-channels-usage.rst | Three channels were decided, with a human gate only at the last hop | "We decided to use the following branches/channels: dev ... alpha ... stable"; "this PR will not be automatically merged, but will require additional human approval" |
| 15 | Zalando | ADR-005 | adr | undated | 2026-09-08 | https://github.com/zalando-incubator/kubernetes-on-aws/blob/dev/docs/dev-guide/arch/cluster-lifecycle/adr-005-channels-usage.rst | Every pull request provisions a real cluster and destroys it | "The e2e testing infrastructure will create a new cluster ... The cluster will be deleted as soon as the tests finish" |
| 16 | Zalando | ADR-005 | adr | undated | 2026-09-08 | https://github.com/zalando-incubator/kubernetes-on-aws/blob/dev/docs/dev-guide/arch/cluster-lifecycle/adr-005-channels-usage.rst | The first post-merge target must be a used cluster, not a fresh one | "Testing the update on an updated cluster and not on a fresh cluster as this might show some different behavior." |
| 17 | Zalando | delivery.yaml | source | schema dated 2017-09-20 | 2026-09-08 | https://github.com/zalando-incubator/kubernetes-on-aws/blob/dev/delivery.yaml | The pipeline runs on four channel branches, not three | when: list contains branch: dev, branch: alpha, branch: beta, branch: stable, plus event: pull_request |
| 18 | Zalando | delivery.yaml | source | 2026-09-08 | 2026-09-08 | https://github.com/zalando-incubator/kubernetes-on-aws/blob/dev/delivery.yaml | Pull requests create a cluster for end-to-end tests | Step create-cluster-aws, desc: "Create cluster for Kubernetes e2e tests", with IAM_ROLE_ARN naming role/cluster-lifecycle-manager-entrypoint |
| 19 | Zalando | Branch listing | source | 2026-09-08 | 2026-09-08 | https://github.com/zalando-incubator/kubernetes-on-aws/branches/all | Four channels plus standing promotion branches are all live | Branches dev, alpha, beta updated Sep 8, 2026, stable Sep 7, 2026; also dev-to-alpha and beta-to-stable |
| 20 | Zalando | cluster/config-defaults.yaml | source | 2026-09-08 | 2026-09-08 | https://github.com/zalando-incubator/kubernetes-on-aws/blob/dev/cluster/config-defaults.yaml | Nodes are provisioned by Karpenter with an explicit exclusion list | karpenter_excluded_instance_families: "c5a,m5a,r5a,c5ad,m5ad,r5ad,c6a,m6a,r6a,c6ad,m6ad,r6ad"; karpenter_ami_family_alias: "custom" |
| 21 | Zalando | cluster/config-defaults.yaml | source | 2026-09-08 | 2026-09-08 | https://github.com/zalando-incubator/kubernetes-on-aws/blob/dev/cluster/config-defaults.yaml | The current node image carries Kubernetes v1.36.3 | kuberuntu_image_v1_36_new_amd64: {{ amiID "zalando-ubuntu-jammy-22.04-kubernetes-production-v1.36.3-amd64-master-436" ... }} |
| 22 | Zalando | cluster/config-defaults.yaml | source | 2026-09-08 | 2026-09-08 | https://github.com/zalando-incubator/kubernetes-on-aws/blob/dev/cluster/config-defaults.yaml | A second provider and an in-place migration exist | {{ else if eq .Cluster.Provider "zalando-eks" }}; skipper_eks_migration_enabled: "true"; eks_control_plane_logging: "true"; sibling-cluster condition (ne .Cluster.SiblingCluster nil) |
| 23 | Zalando | cluster/manifests directory | source | 2026-09-08 | 2026-09-08 | https://github.com/zalando-incubator/kubernetes-on-aws/tree/dev/cluster/manifests | The platform is 63 component directories, split between upstream and in-house | Directory listing includes skipper, stackset-controller, kube-downscaler, kube-janitor, pdb-controller, kube-metrics-adapter, kube-aws-iam-controller, z-karpenter, aws-load-balancer-controller, prometheus, flannel, coredns-local (63 entries counted) |
| 24 | Zalando | ingress-controller deployment manifest | source | 2026-09-08 | 2026-09-08 | https://github.com/zalando-incubator/kubernetes-on-aws/blob/dev/cluster/manifests/ingress-controller/deployment.yaml | The in-house AWS ingress controller is still deployed in 2026 | name: kube-ingress-aws-controller, with args including --nlb-zone-affinity=, --nlb-alpn-policy=, --target-access-mode=AWSCNI |
| 25 | Zalando | Running Kubernetes in Production (admin guide) | casestudy | early migration era, undated | 2026-09-08 | https://github.com/zalando-incubator/kubernetes-on-aws/blob/dev/docs/admin-guide/kubernetes-in-production.rst | The 2017-era fleet had 80+ clusters, Container Linux nodes and kube-aws provisioning | "We have more than 80 production Kubernetes clusters."; "We chose to adapt kube-aws_"; "CoreOS' Container Linux perfectly matches our understanding of the node OS" |
| 26 | Zalando | Running Kubernetes in Production | casestudy | undated | 2026-09-08 | https://github.com/zalando-incubator/kubernetes-on-aws/blob/dev/docs/admin-guide/kubernetes-in-production.rst | The channel model and its automatic update behaviour were stated policy from the start | "Clusters are automatically updated as soon as changes are merged into the respective branch."; "The channel concept is similar to how CoreOS manages releases of Container Linux." |
| 27 | Zalando | Running Kubernetes in Production | casestudy | undated | 2026-09-08 | https://github.com/zalando-incubator/kubernetes-on-aws/blob/dev/docs/admin-guide/kubernetes-in-production.rst | Pod identity was originally a third-party daemon | "The kube2iam_ daemon conveniently allows to assign an AWS IAM role to a pod by adding an annotation." |
| 28 | Zalando | Cluster Lifecycle Manager README | source | undated, claim ceiling v1.24 | 2026-09-08 | https://github.com/zalando-incubator/cluster-lifecycle-manager/blob/master/README.md | 200+ clusters, continuously upgraded in place since v1.4 | "It's currently used to operate 200+ clusters on AWS where the oldest clusters has been continuously updated all the way from Kubernetes v1.4 to Kubernetes v1.24 by the CLM." |
| 29 | Zalando | Cluster Lifecycle Manager README | source | undated | 2026-09-08 | https://github.com/zalando-incubator/cluster-lifecycle-manager/blob/master/README.md | The reconciler is reentrant and holds no state of its own | "It is designed in a reentrant way meaning it can be killed at any point in time and it will just continue any cluster updates from where it left off. All state is stored in the Cluster Registry and the git configuration repository." |
| 30 | Zalando | Cluster Lifecycle Manager README | source | undated | 2026-09-08 | https://github.com/zalando-incubator/cluster-lifecycle-manager/blob/master/README.md | Development started January 2017 | "The CLM has been developed internally at Zalando since January 2017." |
| 31 | Zalando | Pull request 2247 | source | merged 2019-06-21 | 2026-09-08 | https://github.com/zalando-incubator/kubernetes-on-aws/pull/2247 | The postmortem's fix reached production in four days via the channels | Title "Increase QPS for kubelet, controller-manager"; merged June 21, 2019; "promoted through alpha and beta channels, reaching stable status by June 25, 2019"; author: "The default for kubelet is ridiculously low, causing 5+ minute long delays when 20 pods are started at the same time." |
| 32 | Zalando | Pull request 9078 | source | closed unmerged, 2025-03 | 2026-09-08 | https://github.com/zalando-incubator/kubernetes-on-aws/pull/9078 | The change gate is itself tested with a deliberately bad change | Title: "Update Karpenter to v1.3.0 (this should fail)", listed among closed-but-unmerged pull requests |
| 33 | Zalando | Pull request 10710 | source | closed unmerged, 2026-03-10 | 2026-09-08 | https://github.com/zalando-incubator/kubernetes-on-aws/pull/10710 | Node-provisioning policy is argued through pull requests, several of which are closed | Title: "Disable amd instance types in production by default" (Mar 10, 2026), closed without merge |
| 34 | Zalando | Skipper README | source | undated | 2026-09-08 | https://github.com/zalando/skipper | Self-reported scale and the only unit cost figure in the corpus | "We use it as Kubernetes ingress controller in more than 100 production clusters. With every day traffic between 500k and 7M RPS serving 15000 ingress and 3750 RouteGroups at less than ¢5/1M requests."; "designed to handle >300k HTTP route definitions"; "Zalando used this project as shop frontend http router with 350000 routes" |
| 35 | Community | Patroni README | source | undated | 2026-09-08 | https://github.com/zalando/patroni | A component built at the company was handed over and lost its company identity | "Patroni (formerly known as Zalando's Patroni) started as a fork of Governor" |
| 36 | Zalando | Senza README | source | undated | 2026-09-08 | https://github.com/zalando-stups/senza | The previous in-house platform is explicitly unmaintained | "THIS PROJECT IS NOT ACTIVELY MAINTAINED ANYMORE!" |
| 37 | Zalando | Senza tags | source | last tag 2023-03-27 | 2026-09-08 | https://github.com/zalando-stups/senza/tags | The superseded platform kept shipping releases years into the successor's life | Tags: 2.1.143 on Mar 27, 2023; 2.1.141 on Sep 23, 2020; 2.1.134 on May 13, 2019 |
Derived figures, with the arithmetic
| Figure | Derivation | Inputs |
|---|---|---|
| 32 Kubernetes minor versions crossed | 1.36 minus 1.4 | Row 28 (v1.4 starting point) and row 21 (v1.36.3 in the current image) |
| Lifecycle manager README is 12 minor versions stale | 1.36 minus 1.24 | Rows 28 and 21 |
| 4 days from merge to stable | 25 June minus 21 June 2019 | Row 31 |
Tiers present and absent
| Tier | Count | Note |
|---|---|---|
| postmortem | 2 | Both from 2019; the only two the company has published in this repository |
| adr | 3 | ADR-001, ADR-003, ADR-005, all in the repository's dev guide |
| source | 13 | Repository files, directory listings, branch and commit history, three pull requests |
| casestudy | 1 | The team's own admin guide, now historical |
| blog, paper, talk, vendor | 0 | Unreachable from this session; see the access note above |
What could not be checked
- The Cluster Registry itself is not public, so its schema is known only through ADR-001.
- No public evidence exists for the promotion intervals actually configured between channels (ADR-005 leaves them as "X hours" and "Y days").
- The move to managed control planes is an inference from configuration and pull request titles. No design record for it was found in the public repository.
- No incident, availability or cost data after 2019 exists in this corpus.