NOTES / AGENTIC DESIGN PATTERNS
Reflection
Have the model critique its own output and try again. Self-Refine, Reflexion, and why the critique works best when something outside the model checks the work.
The idea
A first draft is rarely the best one. Reflection builds that observation into the system: produce an output, assess it, and use the assessment to produce a better one. There are two well-known versions, one that improves a single output and one that improves the agent across attempts.
Self-Refine: improve one output
Self-Refine generates an initial output, has the same model give feedback on it, then has the model refine the output using that feedback. It iterates between feedback and refinement until a desired condition is met1. It needs a suitable language model and three prompts (generation, feedback, refinement), and no extra training1.
Diagram: my own, following the loop described in 1.
The authors tested it on seven tasks, from dialogue response generation to mathematical reasoning, with GPT-3.5, ChatGPT and GPT-4, and report that outputs were preferred by humans and by automatic metrics over one-step generation1. Treat that as evidence from those tasks and models, not a guarantee for yours.
Reflexion: learn across attempts
Reflexion targets agents that act in an environment such as a game, compiler or API. Instead of updating model weights, it reinforces the agent through linguistic feedback2. It converts a binary or scalar signal from the environment into a short textual summary and adds that to the context for the next episode2. Agents verbally reflect on the feedback and keep that reflective text in an episodic memory buffer to make better decisions in later trials2.
Diagram: my own, using the actor, evaluator and self-reflection roles from 2.
The paper reports 91% pass@1 on the HumanEval coding benchmark, against 80% for GPT-4 at the time of writing2. As with any benchmark number, that reflects one setup in 2023.
The evaluator-optimizer workflow
Anthropic describes the same shape as a workflow: one LLM call generates a response while another provides evaluation and feedback, in a loop. It fits when there are clear evaluation criteria and iteration gives measurable value3. This is the version most production systems use, because it keeps the generator and the critic as separate steps you can test.
Make the critique concrete
My own working rule, not a claim from the papers: a critic is only as good as the signal it can see. A model judging its own answer can share the blind spots that produced it. A critic that can run the tests, execute the code, search a source or compare against a rubric has something to push against.
Diagram: my own illustration of the working rule above.
Reflexion’s design agrees in spirit, because its evaluator turns an environment signal into the lesson2.
Practical notes
- Bound the loop. Two or three rounds usually capture most of the gain; set a cap and a stop condition (Stopping conditions).
- Give the critic criteria, not just “improve this”. Specific rubrics make feedback actionable.
- Check that it helps. Compare with and without the reflection step on your eval set, since the extra calls cost latency and money.
- Pair with a judge when scoring. Evaluating outputs is its own skill: see LLM as judge.
Sources
- [1]Self-Refine: Iterative Refinement with Self-Feedback — Aman Madaan and 15 co-authors · arXiv, NeurIPS 2023 · 2023PARAPHRASED
- [2]Reflexion: Language Agents with Verbal Reinforcement Learning — Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, Shunyu Yao · arXiv, NeurIPS 2023 · 2023PARAPHRASED
- [3]Building effective agents — Erik Schluntz and Barry Zhang · Anthropic · 2024PARAPHRASED