Serving engine
The software that loads a model's weights and serves inference requests, handling batching, KV cache memory, multi-GPU splitting and an HTTP API. Examples include vLLM, SGLang, TensorRT-LLM and llama.cpp.
Why it matters. When self-hosting, the engine affects throughput, latency, supported hardware, and which models and formats you can run.
How it works
A serving engine turns weight files into an endpoint. It manages GPU memory for weights and KV cache, schedules requests with continuous batching, runs optimized attention kernels, splits large models across GPUs, and usually exposes an OpenAI-compatible API.
Engines differ in focus: vLLM and SGLang are open-source engines aimed at high-throughput GPU serving; TensorRT-LLM is NVIDIA's, optimized for its GPUs; llama.cpp runs GGUF files on CPUs, Apple silicon and consumer GPUs. Support for new model architectures, quantization formats and features such as tool-call parsing, structured outputs and speculative decoding arrives at different times in each.
Results depend on engine version, settings and hardware, so benchmark your own model, prompts and concurrency rather than relying on published comparisons. Check that the engine applies the model's chat template and tool-call format correctly; a mismatch hurts quality without raising errors.
Worked example
Illustrative: on the same GPU at 64 concurrent users, engine A produces 1,500 output tokens per second and engine B 1,000. At $3 per GPU-hour, A costs $3 / (1,500 x 3,600 / 1,000,000) = about $0.56 per 1M tokens and B about $0.83.
Common mistakes
- Comparing engines with different settings (precision, batch limits, context length) and crediting the engine for the difference.
- Using a chat template or tool-call parser that doesn't match the model, which degrades answers silently.
- Assuming a newly released model runs on every engine on day one; architecture support often lags.
Across the catalog today
Who serves the open-weights models in the catalog
- DeepInfrahosts open-weights models10 models
- Together AIhosts open-weights models9 models
- Amazon Bedrockhosts open-weights models7 models
- Replicatehosts open-weights models5 models
- Azure AI Foundryhosts open-weights models4 models
- fal.aihosts open-weights models4 models
- Fireworks AIhosts open-weights models3 models
- Groqhosts open-weights models2 models
Published prices from each host's own pricing page.
See it on a real card: Where to run it on Llama 4 Maverick.
Related terms
- Inference29
Continuous batching
Requests join and leave the batch at every step
Explained - Inference28
KV cache
Per-request attention memory that grows with context
Explained - Pricing14
Self-hosting cost
What it costs to run open weights yourself
Explained - Weights & training48
Weight file formats
How weights are packaged: safetensors, GGUF and more
Explained - Inference37
Tensor and pipeline parallelism
Splitting one model across several GPUs
Explained