Serving stacks
The layers between your app and the GPU, the main open-source inference engines, and a framework for choosing between a managed API and self-hosting.
The layers
Between your agent and the hardware there is more than one piece of software. Which pieces you run yourself depends on whether you call a hosted API or host a model.
Diagram: my own.
- Gateway or router. Authentication, rate limits, retries, routing between models, logging and sometimes a response cache.
- Inference engine. The server that loads the model and schedules requests.
- Engine features. Continuous batching, KV-cache management, quantisation, speculative decoding and prefix caching.
- Hardware. GPUs, or CPUs for small models. Memory capacity and bandwidth decide a lot.
Common open-source engines
These descriptions come from each project’s own material or from third-party summaries, and the projects change quickly, so confirm current features before you commit.
vLLM. Built on PagedAttention, which manages KV-cache memory in blocks2. Its docs list continuous batching, chunked prefill, prefix caching, speculative decoding, several quantisation formats (GPTQ, AWQ, INT4, INT8, FP8), tensor and pipeline parallelism, streaming and an OpenAI-compatible API server1. A common default for GPU serving.
SGLang. A frontend language and a runtime. The runtime adds RadixAttention for KV-cache reuse across calls and compressed finite state machines for faster structured-output decoding3. The authors report up to 6.4 times higher throughput than state-of-the-art systems on their benchmarks, including agent-style, JSON-decoding and RAG workloads3; treat that as their result. It suits workloads with many calls that share prefixes or need constrained output.
TensorRT-LLM. NVIDIA’s library for its own GPUs, with in-flight batching and a paged KV cache with block reuse. A summary describes the focus as speed on NVIDIA hardware at the cost of more setup4.
llama.cpp and Ollama. llama.cpp is a lightweight C/C++ runtime that runs on consumer hardware, using GGUF quantised files and supporting CPU-GPU hybrid inference45. Ollama wraps llama.cpp to simplify downloading and running models locally6. Red Hat’s comparison frames llama.cpp as suited to local and consumer hardware and vLLM to higher-throughput serving5.
Managed API or self-hosted?
Diagram: my own.
Questions that usually decide it (my own framework):
- Data. Can prompts leave your network? Regulated data may force self-hosting or a private deployment (Privacy and regulation).
- Quality. Do you need the strongest models? Hosted frontier models are often ahead of open weights, but check on your own eval set.
- Volume. At steady high volume, owning GPUs can be cheaper per token; at spiky or low volume, pay-per-token usually wins. Include engineering time in the comparison.
- Customisation. Need a fine-tuned or distilled model? Open weights give you control.
- Operations. Do you have people to run capacity, upgrades, monitoring and on-call?
- Latency. Self-hosting can reduce network hops and give predictable latency; hosted providers may offer fast modes.
A middle path is a managed host for open models, which gives open weights without running GPUs yourself.
If you self-host
- Benchmark your workload. Together AI’s guidance applies: measure TTFT and token rate at your real prompt lengths and concurrency7.
- Plan memory: weights plus KV cache for your target concurrency and context length.
- Set limits: maximum context, output length and queue depth, to protect the tail.
- Observe it: export metrics for queue time, TTFT, TPOT, cache hit rate and GPU memory (Tracing and monitoring).
- Plan for upgrades. Engines and model formats change quickly; pin versions and test before rolling out.
- Keep an API fallback for outages and traffic spikes.
Either way, put a thin gateway in front so that you can change models or providers without rewriting your agent.
Sources
- [1]Welcome to vLLM — vLLM project docsPARAPHRASED
- [2]Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al. · 2023PARAPHRASED
- [3]SGLang: Efficient Execution of Structured Language Model Programs — Zheng et al. · 2024PARAPHRASED
- [4]8 LLM Inference Frameworks You Should Try in 2026 — KDnuggets · 2026DESCRIPTIONS OF TENSORRT-LLM AND LLAMA.CPP, AS SUMMARISED BY A THIRD PARTY
- [5]llama.cpp vs. vLLM: Choosing the right local LLM inference engine — Cedric Clyburn, Red Hat · 2026PARAPHRASED
- [6]LLM Inference Provider Frameworks — ALC docs (talk notes) · 2025DESCRIPTION OF OLLAMA, AS SUMMARISED BY A THIRD PARTY
- [7]What is LLM inference — Together AIPARAPHRASED