NOTES / SAFETY AND GUARDRAILS

Human approval

When a person should approve an agent's action, how to tier actions by risk, and how to design prompts people can actually judge instead of rubber-stamping.

UPDATED 18 NOV 2026#safety#human-in-the-loop#approval#oversight

Approval is a control, not a ritual

A person approving an action is the strongest check you can add, because the person can use context the system lacks. It is also the most expensive and the easiest to degrade. Use it where it changes outcomes.

OWASP’s guidance on excessive agency, one of its LLM risks, is about agents holding more autonomy than they should6. Approval gates are how you give autonomy back in small, controlled amounts. See also Levels of autonomy and the pattern note Human in the loop.

Tier actions by risk

Diagram: read-only lookups auto-allowed, reversible edits allowed within scope, hard-to-undo or external actions asked, never-acceptable actions denied

Diagram: my own proposal for tiers. Adjust to your domain.

A simple test for the ask tier: could the action be hard to reverse, visible to other people, costly, or involve sensitive data? If so, ask. Examples are sending a message, spending money, deleting data, changing access, and publishing.

Claude Code lets you express tiers as rules: ask rules prompt each time, allow rules run without approval and deny rules block outright1. Some actions are never auto-approved by any mode, such as tools matched by an explicit ask rule2. Meta’s Rule of Two suggests that when an agent must combine untrusted input, sensitive access and outward action, human supervision or another reliable check should be added4.

Keep the number of prompts low

Prompts that appear constantly get approved without reading. Anthropic’s sandboxing work is aimed at this: automatically allow safe operations inside a boundary, block clearly bad ones, and ask only when needed3. So the best way to improve approval quality is to remove the low-risk prompts, using permissions and a sandbox, so the remaining prompts mean something.

Design the prompt

Diagram: a risk check lets low-risk actions run and shows high-risk actions to a person, who approves or rejects with a reason

Diagram: my own.

A good approval request shows (my own checklist):

  1. The exact action, with real arguments: the recipient and message body, the file path, the amount.
  2. Why the agent wants it, in a sentence, tied to the user’s goal.
  3. The effect and whether it can be undone.
  4. The source of the instruction, if the plan was influenced by content the agent read.
  5. Clear options: approve, edit, or reject with a reason that goes back to the agent.

Show the action itself, not only the model’s description of it. A summary written by a model that has been manipulated can hide what the action does.

Pausing and resuming

Approvals often arrive minutes or hours later. Runtimes with checkpointing, such as LangGraph, let a run pause for a person to inspect or edit state, then continue5. Design the approval step as a stored request, not a blocking call (State and checkpoints, LangGraph).

Failure modes

  • Rubber-stamping. Too many prompts, or prompts nobody can judge. Measure approval rate and time-to-approve; near-100 percent and near-zero seconds is a warning sign.
  • Approval bypass. The model finds another tool that does the same thing without a prompt. Gate by effect, not by tool name.
  • Bad defaults. If no one answers, the safe default is to not act.
  • Responsibility gaps. Decide who may approve what, and keep an audit trail (Tracing and monitoring).

In regulated settings, human review may be a legal requirement, not only a safety choice (Privacy and regulation).

Sources

  1. [1]Configure permissions — Anthropic, Claude Code docsPARAPHRASED
  2. [2]Permission modes — Anthropic, Claude Code docsPARAPHRASED
  3. [3]Beyond permission prompts, making Claude Code more secure and autonomous — Anthropic Engineering · 2025PARAPHRASED
  4. [4]Agents Rule of Two: A Practical Approach to AI Agent Security — Meta AI · 2025PARAPHRASED
  5. [5]LangGraph overview — LangChain docsPARAPHRASED
  6. [6]OWASP Top 10 for LLM Applications 2025 — AI Governance Library (summary of the OWASP guidance)EXCESSIVE AGENCY, AS SUMMARISED BY A THIRD PARTY

© Ashish Kumar. All rights reserved. Quoted and cited material belongs to its owners. How I attribute sources.

Product, company and framework names are trademarks of their respective owners. Mentioning them does not imply affiliation or endorsement.