Guardrail Classifiers and Content Filtering
Putting a separate classifier in front of and behind the model gives you a safety layer updatable in hours rather than retrained over weeks, and it buys that agility with latency, false refusals, and a second model to keep honest.
Refusal training bakes safety behaviour into the weights. That is slow to change, hard to audit, and unavoidably entangled with capability: making a model refuse more also makes it help less. Guardrails take the opposite approach, wrapping the model in classifiers that inspect input and output as separate, replaceable components.
The tradeoff is explicit. You get a policy you can version, test, and roll back without touching the model, in exchange for extra latency, extra compute, and a new component that can be wrong in both directions.
The two-sided architecture
A production guardrail stack is rarely one classifier.
Input classification runs before the model sees the prompt. It catches policy-violating requests and, in agent settings, injected instructions arriving inside retrieved documents or tool output; see prompt injection and injection-aware prompt design.
Output classification runs on generated text. It exists because input filtering cannot catch everything: a harmless-looking prompt can produce a harmful completion, and multi-turn attacks distribute the harm across turns so no single input is objectionable.
Streaming changes the shape of the problem. A token-by-token stream forces the output classifier either to buffer, destroying perceived latency, or to classify partial completions and be prepared to retract. Both options are visible to users.
Llama Guard made the input/output split concrete as an open artefact: a Llama 2 7B model fine-tuned for safety classification against a customisable taxonomy, applied to both prompts and responses, matching or exceeding available moderation tools on the OpenAI Moderation and ToxicChat benchmarks (Inan et al., 2023, arXiv:2312.06674).
Rules that generate their own training data
The scaling problem with classifiers is data. A new policy needs a new labelled corpus, and labelling is slow.
Constitutional Classifiers address this by writing the policy as natural-language rules and using a model to generate synthetic training data from those rules, adversarial variants included (Sharma et al., 2025, Constitutional Classifiers, arXiv:2501.18837). The reported evaluation is unusually informative because it publishes both sides of the ledger:
- Over 3,000 estimated hours of red teaming produced no universal jailbreak against the protected system.
- Production refusals rose by 0.38 percentage points in absolute terms.
- Inference compute rose by 23.7%.
That last pair is the number to remember. Roughly a quarter more compute per request is what a strong guardrail layer costs, and it is a permanent tax on every token, not a one-time training expense.
Calibrating the threshold is the actual engineering
A guardrail is a binary decision with a tunable threshold, so it has the ROC curve of any detector, and choosing the operating point is a product decision rather than a safety one.
The asymmetry that catches teams out is base rate. If 1 request in 10,000 is genuinely harmful and the classifier runs at a 1% false positive rate, then 100 legitimate requests are blocked for every real catch. False refusals are not a cosmetic problem. They are the dominant failure mode by volume, they concentrate on unusual but legitimate users in medicine, security research and non-English languages, and they are the fastest route to users routing around the safety layer entirely.
When it breaks
The classifier is a model, so it has a model's vulnerabilities. It can be jailbroken, injected into, and confused by adversarial encoding. A guardrail that reads untrusted text is itself a target.
Two models, two policies, one incoherent product. When the guardrail's notion of harm and the model's refusal training disagree, users see inconsistent behaviour that neither team owns.
Distribution shift is invisible until it is not. A classifier trained on last year's attack corpus degrades silently as attack styles move. Guardrails need the same production monitoring and drift discipline as any deployed classifier, and the same red teaming cadence.
Latency lands on the critical path. Input classification adds directly to time to first token. Small classifiers, or speculative overlap with prefill, are the usual mitigations, and both add architectural complexity.
Coverage is not policy. Passing a taxonomy of eleven categories says nothing about harms outside it. The taxonomy is the policy, and writing it well is harder than training the classifier that enforces it.
12 flashcards for this concept
Click a card to reveal the answer.