Gardener is the system SAP started in 2017 to give its own teams Kubernetes clusters on five infrastructures, and nearly every architectural decision it took is public: thirty-five enhancement proposals, a feature gate table that dates every mechanism it adopted and every one it withdrew, 759 released versions with timestamps, and three 2025 advisories that price the architecture's one structural risk. This guide reconstructs what changed between 2019 and 2026 and why, so that a reader about to build or buy a fleet control plane can argue the same trade-offs with numbers instead of taste.
One team has to hand hundreds of other teams their own compute environment, on whichever infrastructure each product is sold on, and keep the unit cost low enough that asking for one is not a budget conversation.
Three groups have solved this in production. The hyperscalers solved it and published a price list rather than a design: Google's flat fee of "$0.10 per cluster per hour (charged in 1 second increments) applies to all GKE clusters irrespective of the mode of operation, cluster size, or topology", which is roughly $73 a month for a control plane you never see. The Kubernetes community solved a narrower problem with Cluster API, which Gardener's own documentation characterises precisely: "Cluster API primarily harmonizes how to get to clusters, while Gardener goes a significant step further by also harmonizing the clusters themselves." And SAP solved it for itself, starting in 2017 according to the copyright line that still ships in the repository, and then wrote the decisions down in public.
That last part is why this guide exists. Gardener is the rare case where a large company's fleet platform can be read rather than guessed at: the enhancement proposals state the motivation, the goals, the non-goals and the rejected alternatives; the feature gate table records when each mechanism entered and when it was taken out again; the module proxy timestamps every release; and the vulnerability record states, in the vendor's own words, what a tenant could reach. Set against that, the public record of how AWS, Google, Microsoft and the other managed-Kubernetes operators run their own fleets is a marketing page and a status history.
The sharpened question for this guide: what does a decade of operating other people's control planes actually teach, and which of those lessons transfer to a reader who is not SAP? The answer that the evidence keeps pointing at is a selection rule. Gardener's durable parts are the ones that have a one-to-one analogue in Kubernetes itself, and the parts it deleted are the ones that had no analogue or that fused two upstream concepts into one. The system's own vocabulary makes the rule visible: a tenant cluster is a pod, a host cluster is a node, the agent on the host cluster is a kubelet. Where that mapping held, the component survived seven years of releases. Where SAP invented something outside it, the code is gone, and the feature gate table gives the date.
This guide covers the architecture of the fleet control plane: where tenant control planes run, how they are exposed, scheduled, scaled, upgraded, isolated and recovered. It does not cover SAP's wider estate (HANA, ABAP, the Business Technology Platform application layer), Gardener's dashboard and CLI surface, or the product and commercial history. It also cannot quote SAP's own operational scale: this session's network policy resolved code hosts, package registries and one vendor price list, and refused every engineering blog, conference site, paper archive and status page, so there are no talks, papers or third-party measurements here, and no published incident review. Section 04 says what that absence does and does not permit a reader to conclude.
Kubernetes managing Kubernetes, with every operational concept renamed rather than reinvented. The naming is not a joke; it is the design method.
The core move is stated in the repository's own README: "The shoot clusters do not have dedicated master VMs. Instead, the control plane is deployed as a native Kubernetes workload into the seeds (the architecture is commonly referred to as kubeception or inception design). This does not only effectively reduce the total cost of ownership but also allows easier implementations for day-2 operations (like cluster updates or robustness) by relying on all the mature Kubernetes features and capabilities." Read that as a bet: if a tenant's control plane is an ordinary deployment, then every tool that already exists for deployments applies to it, including rolling updates, horizontal and vertical autoscaling, pod disruption budgets, topology spread and the scheduler.
The bet is carried through to the vocabulary. The README publishes the mapping directly, and it is worth reading as an architecture diagram in table form.
| Kubernetes concept | Gardener concept | What the analogue buys |
|---|---|---|
| Pod | Shoot cluster (a tenant's cluster) | Scheduling, eviction, capacity accounting |
| Node | Seed cluster (a host for control planes) | Capacity and allocatable, drain, cordon |
| Kubelet | Gardenlet, running in each seed | Pull-based reconciliation, no inbound path to the host |
| Scheduler | Gardener scheduler | Placement policy as data, not code |
| Controller manager, API server | Gardener controller manager, extension API server | Declarative spec and status for a cluster |
Three consequences of that mapping are visible in the current code and documents, and they are the parts a reader should copy rather than the naming.
Availability is a scheduling property, not a replica count. The architecture document is explicit that control plane components "can be deployed with a replica count of 1 and only need to be scaled out when the control plane gets under pressure, but no longer for HA reasons", because the host cluster restarts them. That held until it did not: GEP-20, which first appears in the release of 11 August 2022, records that "many of the other critical control plane components including etcd are only offered with a single replica, making them susceptible to both node failure as well as zone failure causing downtimes", and introduces a per-tenant choice of failure domain. The design is still that redundancy is bought deliberately, per tenant, rather than applied uniformly.
The expensive shared resources are partitioned by name. Each tenant gets two etcd
instances rather than one: the concepts document states the split exists so that "the critical
etcd-main is not flooded by Kubernetes Events, as well as backup
space is not occupied by non-critical data". At the documented default of 200 tenants per host
cluster, that is 400 etcd instances in one Kubernetes cluster, each with a backup sidecar that
ships snapshots to object storage and, per GEP-6, performs "data corruption checks ... prior to
starting etcd". The network path in front of them is shared in the opposite direction: one load
balancer per host cluster serves every tenant API server, using server name indication to route
without terminating the tenant's TLS, with a deliberately stripped Istio installation as the
gateway ("the default profile which is recommended for production deployment, is not suitable
for the Gardener use case, as it offers more functionality than desired").
Infrastructure knowledge lives outside the core. GEP-1, the oldest proposal in the set and already implemented when version 1.0.0 shipped in February 2020, moved every cloud-specific and operating-system-specific behaviour into separate controllers, for a reason stated in one sentence: "Every change must be done centrally, requires to completely rebuild Gardener, and cannot be deployed individually. Similar to the motivation for Kubernetes to extract their cloud-specifics into dedicated cloud-controller-managers or to extract the container/storage/network/... specifics into CRI/CSI/CNI/..., we aim to do the same right now." The modern extension set covers five infrastructures that are continuously conformance-tested, and the extension contract is where the provider teams work without touching the core.
What left the platform is as instructive as what stayed. Version 1.0.0 in February
2020 shipped opinionated extras into every tenant cluster: an add-on chart bundle carrying a
Kubernetes dashboard and an nginx ingress controller, plus a second bundle described in its own
chart metadata as "temporarily experimental support for out-of-the-box installation of Kyma",
another SAP project. Comparing the 2020 and 2026 release archives dates the retreat: the Kyma
bundle was gone by release 1.9.0 on 25 August 2020, the bundled seed monitoring chart by
1.15.0 that January, after GEP-19 moved observability onto operators, and the last ingress
add-on is being switched off behind feature gates named
DisableNginxIngressInShoot, DisableNginxIngressInSeed and
DisableNginxIngressInGarden, which appeared in release 1.142 in 2026. Six years to
put down four add-ons: the platform kept the contracts and gave up the opinions.
Two divergence points matter when transferring this design. The first is that hosting control planes elsewhere is not always allowed: GEP-28, carried in the release of 13 November 2025, adds tenant clusters whose control plane runs on their own nodes, for environments "where the control plane must run side-by-side and cannot run in seed clusters", naming air-gapped operation, edge, firewalling and compliance. The proposal is careful to say it is not a kubeadm or k3s replacement, and it also fixes an awkwardness the team names itself: Gardener needed a conformant Kubernetes cluster to run in, "but so far there was no way of setting up or managing this initial cluster via Gardener itself", which the document calls "somewhat paradoxical". The second divergence is the recursion: host clusters can themselves be managed tenant clusters, which is how the fleet grows without a separate provisioning path.
Nine forks in the road, each taken in public, each with the condition that would make the rejected option right. Four are worth the long form.
Shoot API, which fixes the make-up of the cluster:
same version, same add-ons, same update and rotation behaviour on every infrastructureprovider: foo with provider: bar",
managed-service support is partly experimental, and the structure "doesn't always
align naturally with how fully managed services architect their offerings"| Decision | Chosen | Rejected | Because | Evidence |
|---|---|---|---|---|
| Provider and operating system knowledge | Out-of-tree controllers behind a contract | In-tree branches per cloud | Central changes forced a full rebuild and could not be deployed individually | GEP-1 |
| Host-to-tenant connectivity | A tunnel opened from the tenant side, built in-house | Upstream Konnectivity, adopted as alpha in release 1.6 and removed in 1.27 | The older design needed an extra load balancer per tenant and blocked private-only estates | GEP-14, gate table |
| etcd lifecycle | A dedicated operator reconciling an Etcd resource | A mutating webhook that rewrote the StatefulSet to inject provider details | The webhook approach "restricts the operations on etcd, such as scale-up and upgrade" | GEP-6 |
| Placement of tenants onto hosts | Capacity and allocatable on the host, enforced by the scheduler | Unbounded placement | "Seeds have a practical limit of how many shoots they can accommodate" | GEP-13 |
| Who decides when a version dies | Classifications plus a machine-readable expiry date per version | Manual promotion through stages | Upstream ships a minor "roughly every three months" and maintains three; manual movement "is cumbersome" | GEP-5, GEP-32 |
| Credentials used to call the infrastructure | Short-lived federated tokens issued by the platform | Long-lived service keys supplied by the tenant | Such credentials are "often non-expiring", widely copied, and their accidental invalidation stops reconciliation | GEP-26 |
| Changing a worker node's configuration | In-place update as an alternative strategy | Rolling replacement only | Bare metal has long boot and local disks; and virtual machine types can be "scarce ... because of an ongoing capacity crunch", naming GPUs | GEP-31 |
| Log storage for every control plane | OpenTelemetry collection into VictoriaLogs | Staying on a fork of the previous engine | The fork "maintains only security updates, thus leading to no new features or improvements getting integrated" | GEP-35 |
| Recovering a tenant whose host cluster is unreachable | Ownership passed by a DNS text record, checked by every participant | Synchronisation objects written to the backup bucket | With owner records "such sync objects are no longer needed", and split brain is prevented without the source host participating | GEP-17 |
Read the table column by column rather than row by row and the selection rule appears. Every chosen option on the left has a Kubernetes analogue: a contract with out-of-tree controllers is the cloud controller manager, capacity and allocatable is the node status, an operator reconciling a custom resource is the operator pattern, independent horizontal and vertical autoscalers are upstream components. Every rejected option is either an invention with no analogue (a fused two-dimensional autoscaler, synchronisation files in a bucket, a webhook that rewrites another controller's output) or an upstream component used against the grain. The two mechanisms Gardener took out of its own feature gate table without ever promoting them to general availability, Konnectivity and HVPA, are one of each kind.
No public incident review exists for this fleet. What does exist is a vulnerability record, a set of design documents written against named failure modes, and a component whose entire purpose is to stop a recovery mechanism from making things worse.
SAP publishes no postmortems for the landscapes it runs on Gardener, and this session could not reach any engineering blog that might carry one. Three of the four cards below are built from vendor security advisories, which are incident records with a known root cause and a known blast radius but no timeline and no detection story. The fourth is built from design documents that state the failure they exist to prevent. That means a reader can trust the mechanisms and the design rules here, and should treat every claim about frequency, duration or detection time as unavailable rather than as reassuring.
providerConfig into the infrastructure tooling they drive.The four cards group into two failure classes, and both are properties of the architecture rather than bugs in it. The first class is shared-host escalation: packing two hundred tenants' control planes into one cluster converts an input-validation defect into a cross-tenant compromise, which is why all three 2025 advisories end with the same clause about obtaining control over the host. The second class is a controller acting on stale or missing information, which appears as machines being replaced during a network fault, as split brain during a control plane migration, and in GEP-20 as a single etcd replica deciding a whole tenant's availability. Gardener's answers to the second class are consistent and worth stealing: a second probe before acting, an ownership record that no participant can forge, and an explicit per-tenant choice of failure domain.
Everything quantitative in the corpus, with the date attached and the derivation shown where the number is mine rather than theirs.
| Metric | Value | Kind | Context | As of | Source |
|---|---|---|---|---|---|
| Tenant clusters per host cluster | 200 | Measured in config | Default capacity.shoots in the shipped agent configuration; the scheduler refuses to place more | 2026-09 | gardenlet config |
| etcd instances per host cluster at default capacity | 400 | Derived | Two per tenant, main and events, times 200 tenants; each with a backup sidecar | 2026-09 | etcd concepts |
| Load balancer cost avoided by sharing one gateway | ~$3,380/mo | Derived | 199 tenants times the proposal's figure of 17 USD per month for a classic load balancer; the figure predates 2020 and current prices differ | 2019 | GEP-8 |
| Managed-service price for one control plane | $0.10/hr | Vendor list price | GKE's flat cluster management fee, about $73 per 730-hour month, independent of cluster size or topology | 2026-10 | GKE pricing |
| Penalty for staying on an unsupported version | +$0.50/hr | Vendor list price | GKE's extended support fee, taking the total to $0.60 per cluster per hour; Gardener instead expires the version | 2026-10 | GKE pricing |
| Minor releases of the control plane | 152 | Measured | From 1.0.0 on 6 February 2020 to 1.152.0 on 24 September 2026, about one every 16 days, against a stated target of "roughly every other week" | 2026-09 | module proxy, process doc |
| Released versions in total | 759 | Measured | Including patches, and including the 0.x line back to 29 November 2019 | 2026-10 | module proxy |
| Release lines kept patchable | 3 | Stated policy | "Hotfixes are usually maintained for the latest three minor releases"; the May 2025 advisories were fixed on four lines | 2026-09 | process doc |
| Feature gates documented | 69 | Derived | Distinct feature names in the gate table, counted by parsing it; 39 carry a general-availability row | 2026-09 | gate table |
| Median time from alpha to general availability | 15 releases | Derived | About 225 days, over the features with both an alpha and a general-availability row; slowest was 57 releases and 904 days | 2026-09 | gate table, release dates |
| Upstream Kubernetes cadence the fleet absorbs | 3 minors/yr | Stated | "The Kubernetes community releases minor versions roughly every three months and usually maintains three minor versions"; control plane and workers are kept on the same version | 2020 | GEP-5 |
| Infrastructures under continuous conformance test | 5 | Stated | AWS, Azure, GCP, OpenStack and Alibaba Cloud, certified up to Kubernetes 1.35 | 2026-09 | README |
| Variation in control plane resource needs | 100x | Stated | "Compute resources required by ShootKapis of different shoots vary by two orders of magnitude", which is why one fixed size per tenant is not an option | 2023 | GEP-23 |
Two of these numbers do the most work in a design review. The first is the packing ratio: a host cluster configured for 200 tenant control planes is the difference between a fleet whose cost scales with clusters and one whose cost scales with load, and the managed-service price list is the alternative you are being measured against. The second is the cadence pair, three upstream minor versions a year arriving into a platform that ships its own minor every sixteen days and keeps three lines patchable; that ratio, not the cluster count, is what sets the size of the team. Everything in the cost column is a list price or a figure inside a proposal, not an audited bill, and the oldest of them is seven years old.
Every source behind this page, graded and dated. The shape of this wall is itself a finding: nineteen decision records, eleven repository artefacts, three advisories, one price list, and no blog post, talk or paper, because the network policy in this session resolved code hosts and registries only.
A tenant project administrator could obtain control over the host clusters running their control planes, in every installation regardless of cloud provider. Fixed on four maintenance lines simultaneously.
The same escalation through a second path, this time in the host-cluster agent and scoped to installations using the GCP provider extension. Identical severity vector, with a changed scope.
Where Terraform drives infrastructure provisioning, tenant-supplied configuration could be injected, again ending in control over the host cluster. Four provider extensions each needed their own fixed version.
The founding decision: provider and operating-system behaviour leaves the core for separate controllers behind a contract, explicitly modelled on Kubernetes extracting its own cloud, container, storage and network specifics.
Routes every tenant API server in a host cluster through one gateway using server name indication, with the tenant's API server still terminating TLS. Prices the rejected option and names the quota limit behind it.
Documents the pre-2020 design, a tunnel needing a load balancer in every tenant cluster, and proposes adopting the upstream proxy instead. The gate it added was removed sixteen months later.
Supersedes both the old tunnel and the adopted upstream proxy, and states the intention to delete the upstream code once validated. The non-goals are unusually honest: not a low latency path, no availability promise.
Makes placing tenants onto hosts a scheduling problem using the fields a node publishes, because "seeds have a practical limit of how many shoots they can accommodate".
Replaces a mutating webhook that rewrote the etcd StatefulSet to inject provider details, because that "restricts the operations on etcd, such as scale-up and upgrade".
Turns the most dangerous manual operation in a cluster into a staged, declarative one, which is the only form that survives being multiplied by a fleet.
States plainly that critical components including etcd ran with a single replica, "making them susceptible to both node failure as well as zone failure", and makes isolation a per-tenant selection.
The clearest cost document in the set: tenant API server compute is "a major part of Gardener's overall compute cost", requirements vary by two orders of magnitude, and the home-grown fused autoscaler had "severe algorithmic limitations".
Replaces long-lived infrastructure credentials with short-lived tokens the cloud provider trusts through federation, naming the operational failure as well as the security one.
Rolling replacement assumes interchangeable machines; the proposal lists what breaks that, including long bare-metal boot, local disks, and machine types scarce during a capacity crunch, with GPUs named.
Adds the model the architecture was built to avoid, for air-gapped, edge and compliance cases, and so that Gardener can bootstrap the cluster it runs in, which the document calls "somewhat paradoxical" to have needed third-party tools for.
Records why the fleet sat on a fork of a relicensed log engine, what that cost, and why collection was standardised first so the storage engine became swappable.
Ownership passes through a DNS text record that every participant checks, including the etcd backup sidecar, so the source host need not cooperate for split brain to be avoided. An earlier design using files in the backup bucket is recorded as no longer needed.
Sets the policy for marking versions preview, supported, deprecated or expired, against an upstream that ships a minor roughly quarterly and maintains three.
Extends classifications with scheduled transitions, because moving versions through stages by hand "is cumbersome".
A standing comparison: Cluster API harmonises getting to clusters, Gardener harmonises the clusters themselves. Lists the blockers, including provider-specific control plane resources and experimental managed-service providers.
Project-wide proposals now live in a dedicated repository under a technical steering committee, modelled on the upstream Kubernetes process. The pointer files left in the code repository date the move.
States the central decision (no dedicated master machines, control planes as workload, lower total cost of ownership) and publishes the concept mapping the rest of the architecture follows.
The sizing statement ("hundreds or even thousands of clusters"), the placement rule, and the claim that single replicas are acceptable because the host cluster watches them.
Sixty-nine documented features with the release each entered and left, including the adopted upstream tunnel (alpha in 1.6, removed in 1.27) and the in-house fused autoscaler (alpha since 0.31, removed in 1.109).
The default packing ratio is a line of YAML in the example component configuration, and the scheduler treats it as the Kubernetes scheduler treats a node's capacity.
Events are split away from the critical store so neither the hot path nor the backup bucket is flooded by low-value data; a sidecar handles snapshots, defragmentation and validation.
Lists everything deliberately not deployed from the upstream default profile: telemetry, the egress gateway, the sidecar injector, the addons.
Carries the cadence target, the hotfix window, and a rota naming a responsible engineer for each version through late 2026. Fourteen names rotate.
A component built to stop cascading failure by "conservatively scaling down dependent configured resources", documenting the scenario, the two probes, the threshold and the meltdown-protection annotation.
"Copyright 2017-2019 SAP SE or an SAP affiliate company" still ships in the current release, alongside the record that the project was seeded from Kubernetes' sample API server.
The proxy lists every published version with its tag time, which dates this guide's timeline, and serves the full source archive of any version, which is how the 2020 and 2026 trees were compared.
The versions tab dates every release: 49 in 2020, then 70, 77, 79, 75, 92 and 79 so far in 2026.
Puts an OpenTelemetry collector in every tenant control plane because the stack "still relies on vendor specific format and protocols", and refuses a flag day: the migration is phased and the old components are not decommissioned immediately.
"A flat cluster management fee of $0.10 per cluster per hour ... applies to all GKE clusters irrespective of the mode of operation, cluster size, or topology", plus $0.50 per hour more for clusters left on a version in extended support.
Six rungs. The first three fit in an evening on a laptop; the line from toy to production-shaped is crossed at rung four, where failure handling starts.
In a local cluster, run a second Kubernetes API server and its own etcd as ordinary pods in a namespace, with their own certificate authority, and point a kubeconfig at it through a service.
Done when: kubectl --kubeconfig tenant.yaml get ns answers from the pod-hosted API server. Teaches: why the inception model makes day-two operations ordinary.
Join one node to that control plane from outside the host cluster, with the tunnel opened from the worker side so no inbound path to the worker network is required.
Done when: the node is Ready and a pod on it is reachable, with no inbound firewall rule on the worker side. Teaches: the connectivity inversion in GEP-14, and why it is explicitly not a high throughput path.
Add a gateway that routes to the right tenant API server by server name indication, without terminating TLS, and give each tenant its own hostname.
Done when: twenty kubeconfigs with twenty hostnames and one IP address all authenticate with client certificates. Teaches: the load balancer and quota arithmetic from GEP-8.
Model the host's capacity and allocatable tenant count, and write a placement loop that refuses to exceed it and spreads across hosts. Then fill a host and watch the next request stay pending.
Done when: the twenty-first tenant stays pending with a reason, rather than degrading the twenty already there. Teaches: that a packing ratio is a scheduling constraint, which is the lesson of GEP-13.
Cut connectivity between the workers and their control plane, and watch what your node lifecycle controller does. Then add the second probe: count expired leases, and scale the replacing controller to zero past a threshold.
Done when: a thirty-minute network fault destroys no healthy machines, and the controllers come back by themselves afterwards. Teaches: the meltdown-protection pattern, and that absence of signal is not evidence of failure.
Back the tenant's etcd to object storage, then restore it on a second host without the first host cooperating. Add an ownership record that both sides check before writing, and test the case where the old host comes back.
Done when: the tenant works on the new host, and the revived old host refuses to reconcile because it lost the ownership check. Teaches: why GEP-17 chose a record no participant can forge over files in a shared bucket.
This page was assembled without access to a single engineering blog. These are the techniques that replaced them, and they work on any company that ships Go.
curl -sL https://proxy.golang.org/<module>/@v/listcurl -s https://proxy.golang.org/<module>/@v/v1.0.0.infocurl -sL https://proxy.golang.org/<module>/@v/v1.0.0.zip -o old.ziphttps://pkg.go.dev/<module>?tab=versionsraw.githubusercontent.com/<org>/<repo>/<tag>/<path>bisect the tag list for the first release containing a design docgrep the feature gate or deprecation table for Since and Until columnsdiff the docs tree of the oldest and newest release archiveraw.githubusercontent.com/github/advisory-database/main/advisories/github-reviewed/<year>/<month>/<GHSA>/<GHSA>.json"obtain control over" OR "privilege escalation" <product> advisorypkg.go.dev/vuln/ for the Go vulnerability database entry<project> enhancement proposal OR GEP OR KEP "alternatives considered"docs/proposals OR geps/ OR docs/adr in the project repository"non-goals" <project> design proposal