NOTES / PART 1 · FOUNDATIONS

01LLM fundamentals

How language models work and what they can and can't do.

THIS SECTION COVERS

Tokens and tokenisation, transformers and attention, pre-training vs post-training, sampling and temperature, reasoning models, capabilities and failure modes.

Related: Prompting and context engineering · Inference and serving

TOPICS · 6/6 WRITTEN

  1. 01.1Tokens and tokenisationHow text becomes the integers a model reads: subword tokens, byte pair encoding, and why token counts differ between models and from word counts.
  2. 01.2Transformers and attentionThe architecture under every current LLM: attention as a weighted lookup, the repeated transformer block, and the causal mask that makes generation left to right.
  3. 01.3How models are trainedFrom next-token prediction to a useful assistant: pretraining, scaling laws, instruction tuning with human feedback, and reinforcement learning for reasoning.
  4. 01.4Sampling and decodingHow logits become a chosen token: temperature, top-k and top-p, why pure sampling degenerates, and what to do when providers remove the knobs.
  5. 01.5Reasoning modelsModels that think before answering: chain of thought, spending compute at inference time, reinforcement learning, and how thinking is controlled and paid for.
  6. 01.6Capabilities and failure modesWhat models are good at, and the failures to design around: hallucination, drifting over long inputs, prompt sensitivity, reasoning slips and unsafe actions.