Your admission-time policy engine becomes unavailable during a deployment window. Should admission fail open - allow the change - or fail closed - block it?

Second by second, what happens under each choice

Fail closed for everything. The policy engine's availability becomes the ceiling on your ability to change production. During an incident that also affects the policy engine - a shared cluster, a shared identity provider, a shared network - you cannot deploy the fix. A control that prevents remediation during an incident has converted a security mechanism into an availability risk, and the predictable human response is a break-glass path used often enough to become the normal path.

Fail open for everything. Every policy is now advisory, and an attacker who can make the engine unavailable can deploy anything. Worse, this is silent: the deployment succeeds and nothing records that it was unchecked.

Why the split is the answer

The two classes of policy have genuinely different risk profiles, and treating them identically is the error.

• Security-critical policies - no privileged containers, no public storage buckets, no unsigned images, no secrets in environment variables - guard against outcomes that are expensive and hard to reverse. Blocking a deployment is cheap by comparison. Fail closed.
• Advisory policies - naming conventions, required labels, resource-request hints, cost tags - guard against untidiness. Blocking a deployment over a missing label during an outage is indefensible. Fail open, and record it.

The audit record is what makes the split safe. An open failure that nobody can see is indistinguishable from no policy at all, so every bypassed evaluation is logged with the change, the policy and the reason, and reconciled afterwards.

Why the other options fail

• "Fail closed for everything." Defensible on a whiteboard and it produces a break-glass procedure that is used weekly, which is worse than the split - because break-glass bypasses all policies, including the security-critical ones.
• "Fail open always." Makes the control's effectiveness depend on the engine's uptime in the direction an attacker prefers, and removes the guarantee that the strict policies exist to provide.
• "Fail open but queue for re-evaluation." Sounds rigorous and is mostly theatre: the change is already running. Re-evaluation afterwards is a detective control, useful as a supplement and not a substitute for admission. It is the right answer to a different question - what to do about changes that did bypass.

What stops the failure in the first place

• Make the engine's availability match the strictest policy's requirement. If some policies fail closed, the engine is in the critical path of deployment and needs the corresponding redundancy.
• Cache compiled policy locally on each admission point, so a control-plane outage does not immediately mean no policy - with a bounded staleness after which the fail behaviour applies.
• Remove the shared dependency. A policy engine that depends on the same cluster it admits to will fail exactly when it is needed.

When this is over-thinking it

With fewer than a handful of policies, and no regulated workload, run them in the pipeline rather than at admission and let a pipeline failure block the change. That is fail-closed by construction, needs no engine in the request path, and is right until you have workloads that can reach production without passing through your pipeline.