A critical CVE in a widely-used C library is announced at 09:00. You have 200 services in containers, each built by its own team's pipeline. Which approach patches the fleet fastest?
Show the full answer Hide the answer
The deciding property
Fleet-wide patch speed is decided long before the CVE, by whether a rebuild is something the platform can initiate or something 200 teams must each choose to do. Everything else in this question is a distraction from that one property.
If services build FROM a platform-owned base whose tag the platform controls, patching is:
update the base, trigger rebuilds, open a pull request per service with the new digest, and
track merge rate as a single number. The slow step becomes review and deploy, which is
measured in hours across teams that are already awake.
The pull request per service, rather than an automatic push to production, is deliberate: it keeps the service's own tests and deploy gate in the path, so a bad base image does not become 200 simultaneous incidents.
Why the other options fail
- Asking every team to rebuild makes the tail unbounded. The median team responds within a day; the last 10% include a service whose owner is on leave, one whose pipeline has not run in six months and has broken in the meantime, and one nobody can name. Fleet patch time is set by the slowest service, not the median, and this approach optimises the median.
- Scan and open exemption tickets produces an accurate picture of your exposure and changes nothing about it. It is the right second activity, to find the images the automated rebuild missed, and it is not a patch mechanism.
- Switching to distroless is a sound long-term reduction in attack surface and is a migration, not an incident response. It also does not help here: the vulnerable library is the C library the application links against, which distroless images still contain.
What would flip the decision
| If this changes | Choose | Because |
|---|---|---|
| Services pin base images by digest with no central control | team-by-team rebuild, then fix the structure | there is no lever to pull; the work is making one exist |
| Fewer than 10 services | ask the teams | coordination is cheaper than automation at that size |
| The CVE is in the application's own dependency tree | dependency bot per repository | the base image is not where the vulnerable code lives |
| Regulatory deadline in hours | rebuild plus a blocking admission policy | you need to prevent old images from running, not only patch them |
What a strong answer adds
The measurement that makes this manageable: base image currency across the fleet, as a distribution. Not "are we patched" but "what fraction of running workloads are on a base image older than 30 days", visible before an incident. Teams with that number already know their patch time; teams without it discover it on the day.
Also the honest constraint: automated rebuild is only safe if base image updates are tested. A platform that can rebuild 200 services in an hour can also break 200 services in an hour, which is why the per-service pipeline stays in the path.
When not to build this machinery
Below roughly 20 services, or where every service is owned by one of three teams in one room, the automation costs more than it returns and a shared chat message patches the fleet by lunchtime. Build the rebuild pipeline when the service count exceeds what one person can chase, which in practice is somewhere between 20 and 50.