Quantisation
Storing weights in fewer bits to cut memory and bandwidth: what the common methods do, where accuracy can slip, and how to check on your own task.
The idea
Model weights are normally stored as 16-bit numbers. Quantisation stores them with fewer bits, such as 8 or 4, which shrinks the model and reduces how much data must move from memory per token. Since decoding is often limited by memory bandwidth (Latency and cost), smaller weights can also mean faster generation.
The saving is simple arithmetic: parameters times bytes per parameter. A 70-billion-parameter model needs about 140 GB at 16 bits, 70 GB at 8 bits and 35 GB at 4 bits, counting weights only.
Diagram: my own arithmetic; it excludes the KV cache and activations.
Kinds of quantisation
Surveys split methods into post-training quantisation, applied to a finished model, and quantisation-aware training, which accounts for the lower precision while training. Post-training methods divide further into weight-only quantisation and methods that quantise weights and activations together4.
Weight-only methods store weights in low precision and convert them back to higher precision for the arithmetic. The speed benefit comes mostly from reading fewer bytes, rather than from faster math5.
Weight and activation methods also quantise the activations so the matrix multiplications themselves run in low precision on supporting hardware5.
Common methods
- LLM.int8(). Dettmers and colleagues showed that 8-bit inference can halve memory for large models without degrading results. The key obstacle was a small set of systematic outlier features in hidden states. Their method multiplies most values in 8 bits and handles the outlier dimensions in 16 bits, with more than 99.9 percent of values in 8-bit1.
- GPTQ. A one-shot, weight-only method that supports 3-bit and 4-bit quantisation and was used to run a 175-billion-parameter model on one 80 GB GPU3.
- AWQ. Activation-aware weight quantisation rests on the observation that weights are not equally important: protecting a small fraction of salient weights, around 0.1 to 1 percent, reduces error. It chooses which channels to protect by looking at activation statistics, not weight statistics23.
- FP8 and other low-precision float formats are supported on newer hardware; vLLM’s documentation lists GPTQ, AWQ, INT4, INT8 and FP8 among supported options6.
Costs and risks
- Accuracy can fall, especially at 4 bits and below, on small models, and on tasks needing precise reasoning, long context or rare knowledge. Aggregate benchmark scores can hide task-specific loss.
- Outliers are hard. Naive rounding to 8 or 4 bits can destroy information in large-magnitude features, which is why the methods above exist1.
- Speed is not guaranteed. Gains depend on kernels, hardware and batch size. At large batches, compute may dominate over memory reads, which narrows the benefit (my understanding; measure).
- Calibration data matters for methods that use it, as they tune to whatever data you provide.
How to adopt it safely
- Start from a published quantised checkpoint or an engine’s supported method, rather than writing your own.
- Run your own eval set on the full-precision and quantised model and compare per-category results (Building eval sets). Include your hardest cases, tool-call formatting and long inputs.
- Check structured outputs. Quantised models can slip on strict formats, so test schema validity (Structured outputs).
- Pick the least aggressive setting that meets your memory goal. 8-bit is usually safer than 4-bit.
- Re-measure latency and cost, not just memory.
If you use a hosted API, the provider decides the precision; you cannot choose it. Quantisation matters for self-hosting and local use (Serving stacks). It is also a different thing from distillation, which trains a smaller model.
Sources
- [1]LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale — Dettmers, Lewis, Belkada, Zettlemoyer · 2022PARAPHRASED
- [2]AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration — Lin et al. · 2023PARAPHRASED
- [3]Resource-Efficient Language Models: Quantization for Fast and Accessible Inference — 2025DESCRIPTION OF GPTQ AND AWQ IN A SURVEY
- [4]Efficient Large Language Models, a survey — 2023TAXONOMY OF QUANTISATION METHODS
- [5]Quantization for Inference: GPTQ, AWQ, SmoothQuant, and FP8 — General Compute · 2026WEIGHT-ONLY VERSUS WEIGHT-AND-ACTIVATION, AS SUMMARISED BY A THIRD PARTY
- [6]Welcome to vLLM — vLLM project docsPARAPHRASED