NOTES / FINE-TUNING AND ADAPTATION
Supervised fine-tuning
Training on example conversations with a next-token loss: data formats, loss masking on assistant tokens, how much data you need, and the common mistakes.
What it is
Supervised fine-tuning (SFT) shows the model example inputs with the outputs you want and trains it to reproduce them. It is the same next-token objective as pretraining, applied to a small, curated dataset. TRL’s documentation describes the loss as token-level cross-entropy1.
SFT is the first stage of the InstructGPT recipe: labelers wrote demonstrations of the desired behaviour, and the model was tuned on them before any preference learning3 (Preference tuning). For most application work it is also the last stage.
What an example looks like
Most chat models are tuned on conversations: a list of messages with roles. TRL accepts conversational data and prompt-completion data, and applies the model’s chat template to turn messages into one token sequence1. Using the wrong template is a classic silent failure, because the model is then trained on a format it will never see at inference.
Diagram: my own.
Train on the answer, not the question
By default you may be training on every token, including the user’s. TRL offers two ways to restrict the loss: completion-only loss for prompt-completion data, and assistant-only loss for conversations1. Assistant-only loss depends on the chat template marking where assistant text starts and ends, using generation markers; TRL patches the templates of some known model families, but for others you must check12. If your template lacks the markers, training can fail or, worse, silently mask nothing. Print a batch of labels and look.
Whether to mask the prompt is a choice, not a law. Masking focuses learning on the behaviour you want. I usually mask and verify by inspection (my own habit).
How much data
Less than people expect, if it is good. LIMA used 1,000 carefully curated prompt-response pairs and no reinforcement learning4. In the paper’s human study, its outputs beat or matched a reproduction of Alpaca trained on 52,000 examples4. The lesson I take is quality and diversity over volume, with the caveat that LIMA tuned a strong base model for general style, not a narrow skill that the base lacks.
For narrow tasks such as classification or extraction, a few hundred to a few thousand clean examples is a common starting range in practice (my own rule of thumb; run a learning curve on your task to see where gains flatten).
Evaluate on behaviour, not loss
Validation loss and perplexity are weak guides. LIMA’s authors observed that held-out perplexity worsened as training continued, even while generation quality improved5. So choose checkpoints by scoring generations on a held-out eval set, using the same checks as in production (Building eval sets).
Full tuning or adapters
You can update all weights or only small adapters. Biderman and colleagues compared them on code and maths, with about 100,000 prompt-response pairs for instruction tuning, and found that LoRA substantially underperformed full tuning in the standard low-rank settings, though it forgot less of the base model’s other skills6. See LoRA and QLoRA for the trade-off.
Common mistakes
- Wrong chat template or end-of-sequence token, so the model never learns to stop.
- Training on user text and then reproducing it.
- Duplicates or leakage between train and test (Preparing training data).
- Over-training on a small set, which makes outputs narrow and brittle.
- Judging by loss instead of by task metrics.
- No baseline. Compare against a good prompt on the same eval (When to fine-tune).
Tool-using agents
If the goal is reliable tool calls, include full traces in the training data: the tool schema, the call, the result, and the final answer, with realistic errors and recoveries (Tool calling). Synthetic data helps but should be checked against real traces.
Sources
- [1]SFT Trainer — Hugging Face TRL documentationPARAPHRASED
- [2]Chat templates in TRL — Hugging Face TRL documentationMIRROR OF THE TRL DOCS PAGE ON TEMPLATES AND GENERATION MARKERS
- [3]Training language models to follow instructions with human feedback — Ouyang et al. · 2022PARAPHRASED
- [4]LIMA: Less Is More for Alignment — Zhou et al. · 2023PARAPHRASED
- [5]LIMA discussion of perplexity versus quality — LessWrongQUOTES THE PAPER'S FINDING ON PERPLEXITY, AS SUMMARISED BY A THIRD PARTY
- [6]LoRA Learns Less and Forgets Less — Biderman et al. · 2024PARAPHRASED