NOTES / LLM FUNDAMENTALS

Transformers and attention

The architecture under every current LLM: attention as a weighted lookup, the repeated transformer block, and the causal mask that makes generation left to right.

UPDATED 18 NOV 2026#llm#transformer#attention#architecture

The shift

Before 2017, strong sequence models used recurrent or convolutional networks, often with attention added on top. Vaswani and colleagues proposed the Transformer, an architecture based solely on attention mechanisms, dispensing with recurrence and convolutions entirely1. Because positions are processed in parallel rather than one step at a time, training scales far better, which is the practical reason this design took over.

Attention in one idea

The paper describes attention as mapping a query and a set of key-value pairs to an output. The output is a weighted sum of the values, where each value’s weight comes from how well the query matches the corresponding key1.

Diagram: a query is scored against four keys, softmax turns scores into weights, and the output is a weighted sum of values

Diagram: my own, following the description in 1. The weights are invented for illustration.

In the paper’s “scaled dot-product attention”, the query is compared with all keys by dot product, each score is divided by the square root of the key dimension, and a softmax turns the scores into weights1. The scaling matters: for large dimensions the dot products grow large, pushing softmax into regions with tiny gradients, so dividing keeps training stable1.

Plainly: each token asks “which other tokens matter for me?”, and gathers a mix of their information in proportion.

Many heads

Multi-head attention runs several attention operations in parallel, each on its own learned projection of queries, keys and values, so different heads can attend to different kinds of relationship at once1.

The block

A transformer layer combines attention with a position-wise feed-forward network, wrapped in residual connections and layer normalisation1. A model stacks many of these blocks. Position information is supplied separately, since attention itself does not know order1.

Diagram: token embeddings with positions feed a repeated block of self-attention, add and normalise, feed-forward, add and normalise, then logits over the vocabulary

Diagram: my own simplification of the architecture in 1. Details such as where normalisation sits vary between models.

The final layer produces a score (a logit) for every token in the vocabulary. Turning those scores into a chosen token is the subject of Sampling and decoding.

The causal mask

The original Transformer was an encoder-decoder for translation, and its decoder uses masked self-attention so a position cannot see later positions1. Today’s chat and agent models are, in broad terms, decoder-style models: GPT-3, for example, is an autoregressive language model2, meaning it predicts each token from the ones before it. The mask enforces that.

Diagram: a lower-triangular grid showing each token attends to itself and earlier tokens only

Diagram: my own.

This explains two practical facts:

  • Generation is sequential. Each new token needs the previous ones, so output is produced one token at a time. Reusing past computation is what a KV cache is for.
  • Longer inputs cost more. Every token can attend to every earlier token, so cost grows with context length, which is one reason long prompts are slower and pricier. See Long-context trade-offs.

What to take away

  • Attention is a differentiable lookup: queries retrieve information from keys and values.
  • Everything else in the block is there to make that lookup trainable at scale.
  • The model’s “knowledge” lives in weights; the context window holds only what you put in the prompt, which is why context engineering matters.

Next: How models are trained.

Sources

  1. [1]Attention Is All You Need — Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin · arXiv, NeurIPS 2017 · 2017PARAPHRASED
  2. [2]Language Models are Few-Shot Learners — Tom B. Brown and co-authors · arXiv, NeurIPS 2020 · 2020PARAPHRASED

© Ashish Kumar. All rights reserved. Quoted and cited material belongs to its owners. How I attribute sources.

Product, company and framework names are trademarks of their respective owners. Mentioning them does not imply affiliation or endorsement.