NOTES / LLM FUNDAMENTALS

Capabilities and failure modes

What models are good at, and the failures to design around: hallucination, drifting over long inputs, prompt sensitivity, reasoning slips and unsafe actions.

UPDATED 18 NOV 2026#llm#hallucination#long-context#reliability

What they are good at

Trained on next-token prediction at scale, language models handle fluent writing, summarising, translating, classifying, extracting structure from text, and writing and explaining code. They can perform new tasks from instructions or a few examples in the prompt, without retraining4. Combined with tools, they can act on the world. The same training that makes them flexible also produces predictable weaknesses.

Diagram: five common failure modes mapped to habits that address them: verify and ground, trim and place context, test variants, let it think and check, limit tools and add approval

Diagram: my own map from failure to habit.

1. Hallucination

Models produce plausible but false statements. The OpenAI and Georgia Tech paper argues this happens because training and evaluation procedures reward guessing over acknowledging uncertainty, like students guessing on exam questions1. They analyse hallucination as an ordinary statistical error: if invalid statements cannot be told apart from facts, errors will arise through natural statistical pressure in pretraining1. They also argue it persists because most benchmarks are graded in a way that makes guessing pay1.

Diagram: an exam rewarding guesses scores a guess at least as well as I don't know, so training learns to guess

Diagram: my own illustration of the incentive argument in 1.

Their suggested fix is socio-technical: change the scoring of existing influential benchmarks so uncertainty is not penalised, rather than add more hallucination tests1. For builders: ground answers in retrieved or tool-provided sources (RAG overview), ask for citations, allow “I don’t know” in the prompt, and verify. For a wider survey, see Ji et al.3

2. Weak use of long context

Bigger windows do not mean uniform attention. Liu and colleagues tested multi-document question answering and key-value retrieval and found performance can degrade significantly when the position of the relevant information changes. It is best at the start or end of the input and worst in the middle, and it decreases as context grows longer, even for models built for long contexts2.

Diagram: an illustrative U-shaped curve of accuracy against the position of the relevant passage, weakest in the middle

Diagram: my own and illustrative, showing the shape reported in 2. Models have changed since 2023, so measure your own.

Implications: keep prompts lean, put key instructions and evidence where they will be used, and retrieve rather than dump. See Long-context trade-offs.

3. Sensitivity to wording and format

Small changes in phrasing, ordering or examples can shift results. Treat prompts as code: version them and test variants against an eval set. Prompt design covers structure.

4. Reasoning slips

Even strong models make arithmetic and logic mistakes, and can talk themselves into wrong answers. Letting the model think helps (Reasoning models), but verifying with code, tests or a checker helps more. GPT-3’s authors already noted tasks where few-shot learning struggled4.

5. Wrong or unsafe actions

When models act through tools, errors become side effects. Limit permissions, sandbox execution, and require approval for irreversible steps (Human in the loop, Permissions).

A habit that covers all five

Assume the model will sometimes be wrong, and build the system so that wrongness is detected (evals, tracing), contained (limits, approvals) and recoverable (error handling).

Sources

  1. [1]Why Language Models Hallucinate — Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, Edwin Zhang · OpenAI and Georgia Tech · 2025PARAPHRASED
  2. [2]Lost in the Middle: How Language Models Use Long Contexts — Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang · arXiv, TACL · 2023PARAPHRASED
  3. [3]Survey of Hallucination in Natural Language Generation — Ziwei Ji and co-authors · arXiv · 2022FURTHER READING
  4. [4]Language Models are Few-Shot Learners — Tom B. Brown and co-authors · arXiv, NeurIPS 2020 · 2020PARAPHRASED

© Ashish Kumar. All rights reserved. Quoted and cited material belongs to its owners. How I attribute sources.

Product, company and framework names are trademarks of their respective owners. Mentioning them does not imply affiliation or endorsement.