NOTES / PART 3 · RUNNING IN PRODUCTION

10Inference and serving

Running models fast and affordably.

THIS SECTION COVERS

Latency and cost, KV cache, batching, quantisation, speculative decoding, mixture-of-experts, serving stacks, caching.

Related: LLM fundamentals · Fine-tuning and adaptation

TOPICS · 8/8 WRITTEN

  1. 10.1Latency and costThe two numbers that describe speed, how a request's price is built, and the levers that move each. Measure at your real prompt lengths before optimising.
  2. 10.2The KV cacheWhat the key-value cache stores, why it dominates serving memory, and the techniques that shrink or manage it: paging, grouped-query attention and prefix reuse.
  3. 10.3BatchingWhy serving many requests together raises throughput, how continuous batching refills the batch at every step, and how the trade-off with latency shows up.
  4. 10.4QuantisationStoring weights in fewer bits to cut memory and bandwidth: what the common methods do, where accuracy can slip, and how to check on your own task.
  5. 10.5Speculative decodingA small draft model proposes several tokens and the large model verifies them in one pass, giving faster decoding without changing the output distribution.
  6. 10.6Mixture of expertsSparse models route each token to a few expert sub-networks, so compute per token is lower than total size suggests, while memory still holds every expert.
  7. 10.7Prompt cachingReuse the computed state of an identical prompt prefix to cut input cost and latency: how provider caches are priced, how to lay out prompts, and what breaks a hit.
  8. 10.8Serving stacksThe layers between your app and the GPU, the main open-source inference engines, and a framework for choosing between a managed API and self-hosting.