An AI feature needs guardrails on inputs and outputs. Where should they run, and what is the latency and reliability consequence?

Where they should run

At a gateway rather than in each application, for the same reasons any cross-cutting control belongs there: consistency, one place to update, one place to measure, and no dependence on every team implementing them correctly.

Implementing guardrails in ten places produces ten different levels of protection, and the weakest one defines the platform's actual posture.

Input guardrails

• Redaction of sensitive data before it leaves the boundary, which is a data-protection requirement and frequently a contractual one.
• Injection detection as a mitigation rather than a control — it is probabilistic and must not be the thing standing between an attacker and an irreversible action. The actual control is authorisation around tools.
• Length and cost limits, which are also a denial-of-service control.

Output guardrails

• Structural validation — is the output valid against the expected schema — which is cheap, deterministic and catches a large class of failures.
• Content checks for the categories that matter to the product.
• Grounding checks for retrieval-based features: does the output's claim appear in the retrieved context. Imperfect and one of the more valuable checks available for factual features.
• Treating output as untrusted wherever it is rendered, executed or used in a query, since the conventional injection defences apply and are frequently forgotten in this context.

The latency and reliability consequence

Every guardrail adds latency and a failure mode. A synchronous output check on a streaming response either blocks the stream or checks after the user has read it, and neither is satisfactory.

The practical resolutions: run cheap deterministic checks synchronously and expensive model-based ones asynchronously with the ability to retract or annotate; stream with a short buffer so a check can run on each chunk; and decide explicitly what happens when a guardrail is unavailable — fail open or fail closed — rather than letting the timeout handler decide.

For a safety-critical category, fail closed; for a quality check, fail open. Applying one policy to both is wrong in one direction or the other.