NOTES / PART 3 · RUNNING IN PRODUCTION

09Safety and guardrails

Keeping models and agents within safe limits.

THIS SECTION COVERS

Prompt injection, permission design, sandboxing, input and output filtering, human approval steps, red-teaming.

Related: Harnesses · Evals and observability

TOPICS · 7/7 WRITTEN

  1. 09.1Threat modelWhere an agent can be attacked or go wrong: untrusted text, sensitive access and outward actions, plus the public risk catalogues to check yourself against.
  2. 09.2Prompt injectionWhy models obey instructions hidden in content, what attackers do with that, and which defences change the outcome rather than just lowering the odds.
  3. 09.3PermissionsDecide what an agent may attempt, outside the model: least privilege, allow, ask and deny rules, modes, and why rules are a policy rather than a boundary.
  4. 09.4SandboxingContain what an agent's code and commands can reach: filesystem and network isolation together, the isolation primitives, and what a sandbox does not protect.
  5. 09.5Input and output guardrailsChecks around the model: classifiers, programmable rails and validators. Where they sit, what the main tools do, and why they reduce risk without being a boundary.
  6. 09.6Human approvalWhen a person should approve an agent's action, how to tier actions by risk, and how to design prompts people can actually judge instead of rubber-stamping.
  7. 09.7Red teamingAttack your own agent before others do: scope, manual and automated methods, adaptive attackers, and turning every finding into a regression test.