Policy Failure Mode

The policy engine is unreachable. A deployment is in flight. The behaviour at this moment is a design decision, and in most installations it is a default nobody chose.

Both uniform answers are defensible on a whiteboard and both fail in practice, in opposite and predictable ways.

Why it matters

Fail closed for everything makes the policy engine's availability the ceiling on your ability to change production. During a correlated outage - shared cluster, shared identity provider, shared network - you cannot deploy the fix. A control that prevents remediation during an incident has converted a security mechanism into an availability risk, and the human response is entirely predictable: a break-glass path, used often enough to become routine. Break-glass bypasses all policies, including the strict ones, so the strictest posture produces the weakest outcome.

Fail open for everything makes every policy advisory at exactly the moment an attacker would prefer, and it does so silently: the deployment succeeds and nothing records that it was unchecked. An open failure that nobody can see is indistinguishable from having no policy.

Implementation patterns

• Split policies into two classes and declare the failure mode per class. Security-critical (no privileged containers, no public storage, no unsigned images, no secrets in environment variables) fails closed. Advisory (naming, labels, resource hints, cost tags) fails open.
• Record every open failure with the change, the policy, the reason and the identity, and reconcile the records on a schedule. This is what makes failing open acceptable.
• Cache compiled policy at each admission point with a bounded staleness window, so a control-plane blip does not immediately mean no policy at all.
• Remove the shared dependency. A policy engine that depends on the cluster it admits to will be unavailable precisely when it is needed.
• Match the engine's availability to the strictest policy's requirement. If some policies fail closed, the engine is in the critical path of deployment and needs the redundancy that implies.
• Make break-glass narrow, logged and noisy - it should page someone, not merely write a line.
• Roll out new policies in warn mode first for at least 14 days, then enforce. A rule that blocks on its first day is a rule that gets removed rather than fixed, and 14 days of warnings also sizes the exemption list before it blocks anything.

Industry example

The pattern is general to every admission control system, and its clearest expression is in Kubernetes admission webhooks, where the failure policy is a first-class configuration field with exactly these two values - a design choice made because the platform's authors recognised that no single answer is correct for all policies. Kubernetes has carried that field since admission webhooks became generally available in 2019, and the recurring operational lesson published across platform-engineering accounts since then is the same: organisations that set every webhook to fail closed discover it during their first control-plane incident, and organisations that set every webhook to fail open discover it during an audit.

Failure scenarios

• Deployment blocked during an incident by a policy engine affected by the same incident.
• Break-glass normalisation, where the emergency path becomes the path of least resistance and all controls are routinely bypassed.
• Silent open failures, so post-incident nobody can enumerate which changes were unchecked.
• Stale cached policy enforcing a rule that was revoked, or missing one that was added.
• A new policy enforced from day one, blocking dozens of legitimate changes and being disabled wholesale rather than corrected.
• Re-evaluation treated as equivalent to admission. Checking afterwards is a detective control; the change is already running.

Trade-offs

The real cost of the split is classification discipline: someone must decide which class each policy is in, and re-decide as policies are added. A classification nobody maintains drifts until everything is advisory.

When not to use it

With a small number of policies and no workload that can bypass the pipeline, run the checks as a pipeline stage instead. That is fail-closed by construction, needs no engine in the request path, and removes the entire question. The admission-time engine earns its place when changes can reach production without passing through your pipeline - which is common in shared clusters, and is the same gap described by the unmanaged change surface.

It also does not apply to detective controls, which by their nature run after the fact and have no admission decision to make. Do not conflate the two: a nightly scan and an admission policy for the same rule are different controls with different guarantees, and only one of them prevents anything.

Interview question

Q: Your platform team wants every admission webhook set to fail closed, arguing that no unchecked workload should ever run. The SRE team objects. Facilitate that argument and propose a resolution.

What a strong answer covers: stating both positions fairly - the security position that an unchecked workload is unacceptable, and the reliability position that a control preventing remediation is itself a risk · identifying the shared failure domain, since the objection is strongest when the engine and the cluster fail together · proposing the split by policy class with a written classification and an audit record for open failures · the supporting work that makes fail-closed affordable for the strict class - engine redundancy outside the admitted cluster and locally cached policy with bounded staleness · break-glass that pages rather than merely logs · and the observation that a uniform fail-closed posture predictably produces routine break-glass use, which is a weaker outcome than the split for both teams.

Quick check

Quiz: Why does setting every admission policy to fail closed often produce weaker security than splitting by class? Because it makes the engine's availability a ceiling on deployment, so an emergency bypass gets built and normalised - and break-glass skips every policy, including the ones that mattered.

Flashcard: What makes failing open acceptable for a policy? — An audit record of every bypassed evaluation, reconciled afterwards. Without it, an open failure is indistinguishable from having no policy.