Input and output guardrails
Checks around the model: classifiers, programmable rails and validators. Where they sit, what the main tools do, and why they reduce risk without being a boundary.
What guardrails are
Guardrails are checks wrapped around a model: before it runs, between the model and its tools, and after it responds. Each check can allow the request, block it, redact part of it, rewrite it, or send it to a person.
Diagram: my own.
Kinds of check
Rules and validators. Cheap, deterministic checks such as schema validation, length limits, regular expressions for card numbers, allow lists of URLs, and PII redaction. Use these first. They are predictable and easy to test (Structured outputs).
Classifier models. A small model labels inputs or outputs. Meta’s Llama Guard is an instruction-tuned Llama 2 7B model that classifies prompts and responses against a safety taxonomy, and its authors report it matched or beat existing moderation tools on the benchmarks they used1. The taxonomy can be adapted, because the policy is supplied in the prompt1.
Programmable rails. NVIDIA’s NeMo Guardrails is an open-source runtime that sits between the application and the model and applies user-defined, model-independent rules2. Input rails can reject or alter input, for example to mask sensitive data3. Output and dialogue rails shape what the application returns2.
Provider-trained classifiers. Anthropic’s Constitutional Classifiers monitor inputs and outputs for harmful content, trained on synthetic data generated from a written constitution of permitted and restricted content4. In their tests, a guarded Claude’s jailbreak success rate fell from 86 percent to 4.4 percent4. A later generation aims at similar protection with lower compute cost and fewer needless refusals5.
Guardrails in agent SDKs. OpenAI’s Agents SDK runs guardrails in parallel with the agent and stops early if a check fails6. See OpenAI Agents SDK.
Output handling is a security issue
Model output is untrusted data when it reaches other systems. OWASP’s 2025 list includes improper output handling as its own entry8. If the output is rendered as HTML, put into a SQL query or run as code, validate and escape it as you would user input (my own restatement of the standard advice).
Limits
- They are probabilistic. Simon Willison argues that products which stop 95 percent of attacks deserve a failing grade, because an attacker only needs the 5 percent that gets through7. For prompt injection, use permissions and design limits as the real defence (Prompt injection).
- They cost latency and money. Each check is another call or model. Run cheap checks first and heavy ones only when needed.
- They produce false positives. A guardrail that blocks legitimate work trains users to route around it. Measure both directions.
- They need maintenance. Attackers adapt, so retest (Red teaming).
Where they are the right tool
Guardrails suit content policy, PII handling, topic restriction, format enforcement and screening for known attack patterns. They are a poor fit as the only barrier between an agent and a destructive action. For that, rely on permissions, a sandbox and approval.
Practical advice
- Start with deterministic checks, then add a classifier where rules cannot express the policy.
- Check tool calls and arguments, not only text.
- Decide in advance what each failure does: block, redact, retry or escalate.
- Log every trigger with the input, so you can tune thresholds with real data.
- Evaluate guardrails like any model component (Building eval sets).
Sources
- [1]Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations — Inan et al., Meta · 2023PARAPHRASED
- [2]NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails — Rebedea et al., NVIDIA · 2023PARAPHRASED
- [3]NeMo Guardrails library — NVIDIA, GitHubPARAPHRASED
- [4]Constitutional Classifiers: Defending against universal jailbreaks — Anthropic · 2025PARAPHRASED
- [5]Next-generation Constitutional Classifiers: More efficient protection against universal jailbreaks — Anthropic · 2026PARAPHRASED
- [6]OpenAI Agents SDK documentation — OpenAIPARAPHRASED
- [7]Willison names 'lethal trifecta' behind AI agent data leaks — AI Weekly · 2025REPORT OF WILLISON'S POSITION, SECONDARY
- [8]OWASP Top 10 risks for LLMs (2025 update) — Zbigniew Banach, Invicti · 2025LIST OF ENTRIES, AS SUMMARISED BY A THIRD PARTY