vLLM
A Python library and OpenAI-compatible server for high-throughput LLM inference and serving on GPUs and other accelerators.
Embedding models run in pooling mode; vLLM's docs say to pass `--runner pooling` for architectures that support both generation and pooling (https://github.com/vllm-project/vllm/blob/v0.32.0/docs/models/supported_models.md).
At a glance
- Built by
- vLLM project — open-source community of 2000+ contributors; originally developed in UC Berkeley's Sky Computing Lab
- Status
- Active
- Latest release
- v0.32.0 (2026-10-18)
- License
- Apache-2.0
- Written in
- Python
- Serves
- Language, Embedding
- Hardware
- NVIDIA GPU, AMD GPU, Intel GPU, x86 / ARM / PowerPC CPU, Google TPU (plugin), Intel Gaudi (plugin), Huawei Ascend (plugin), Apple Silicon (plugin)
- API
- OpenAI-compatible HTTP server (also Anthropic Messages API and gRPC)
- Quantization
- FP8, MXFP8, MXFP4, NVFP4, INT8, INT4, GPTQ, AWQ, GGUF, compressed-tensors, ModelOpt, TorchAO
From vLLM's own repo and docs, checked 21 Oct 2026.
Get started
uv pip install vllmvllm serve {model}Replace {model} with the model ID below.
What it's good at
- PagedAttention and continuous batching
KV-cache memory is managed with PagedAttention; requests are continuously batched, with chunked prefill and prefix caching. source ↗
- 200+ architectures
Decoder-only and MoE LLMs, hybrid/state-space models, multimodal models, and embedding, retrieval, reward and classification (pooling) models. source ↗
- Distributed inference
Tensor, pipeline, data, expert and context parallelism, plus disaggregated prefill, decode and encode. source ↗
- Speculative decoding
N-gram, suffix, EAGLE and DFlash speculative decoding. source ↗
- Tool calling and multi-LoRA
Tool-calling and reasoning parsers, structured outputs via xgrammar or guidance, and multi-LoRA for dense and MoE layers. source ↗
Catalog models it runs 32
vllm serve Qwen/Qwen3-Embedding-8BQwen's card gives a vLLM example (requires vllm>=0.8.5); Qwen3-Embedding is listed on vLLM's embedding page.
vllm serve google/embeddinggemma-2Architecture EmbeddingGemma2Model is registered as a multimodal embedding model in vLLM v0.32.0's model registry (not yet in the docs table).
vllm serve jinaai/jina-embeddings-v5-omni-smallJina's card validates vllm==0.20.1; needs --trust-remote-code and one task adapter per instance via --hf-overrides.
vllm serve jinaai/jina-embeddings-v5-text-smallListed under JinaEmbeddingsV5Model; vLLM merges one task LoRA at load time, chosen with --hf-overrides '{"jina_task": "<task>"}' (default retrieval).
vllm serve nomic-ai/nomic-embed-text-v2-moeListed under NomicBertModel on vLLM's embedding page.
vllm serve nvidia/Nemotron-3.5-Embed-1B-BF16NVIDIA's card gives vLLM 0.32.0 examples (mean-pooling config, transformers>=5.16.1,<5.20.0) for text, image and image+text embeddings.
vllm serve nvidia/Nemotron-3.5-Embed-8B-BF16NVIDIA's card gives vLLM 0.32.0 examples (mean-pooling config, transformers>=5.16.1,<5.20.0) for text, image and image+text embeddings.
vllm serve Qwen/Qwen3.8-27BQwen's card links a vLLM recipe; architecture Qwen3_5ForConditionalGeneration is on vLLM's supported list.
vllm serve Qwen/Qwen3.8-Flash-NextQwen's card links a vLLM recipe; architecture Qwen4ExpForConditionalGeneration is registered in vLLM v0.32.0.
vllm serve Qwen/Qwen3.8-2.4T-A95BQwen's card says the weights are compatible with vLLM and links a recipe; architecture Qwen3_5MoeForCausalLM is registered in vLLM v0.32.0.
vllm serve CohereLabs/command-a-plus-05-2026-bf16Cohere's card requires vllm>=0.21.0 and its cohere-melody library for accurate response parsing; also listed on vLLM's supported-models page (Cohere2VisionForConditionalGeneration).
vllm serve deepseek-ai/DeepSeek-V4-Pro-0813DeepSeek's card gives a vllm serve command (with DSpark speculative decoding); DeepseekV4ForCausalLM is on vLLM's supported list.
vllm serve deepseek-ai/DeepSeek-V4.1-FlashArchitecture DeepseekV41ForCausalLM is registered in vLLM v0.32.0's model registry (not yet in the docs table).
vllm serve google/gemma-3-27b-itListed under Gemma3ForConditionalGeneration (text + image).
vllm serve google/gemma-4-31B-itArchitecture Gemma4ForConditionalGeneration is on vLLM's supported list; vLLM notes audio input only works on the E2B/E4B variants.
vllm serve meta-llama/Llama-3.1-8B-InstructLlama 3.1 is listed under LlamaForCausalLM.
vllm serve meta-llama/Llama-3.3-70B-InstructArchitecture LlamaForCausalLM is on vLLM's supported list.
vllm serve meta-llama/Llama-4-Maverick-17B-128E-InstructListed under Llama4ForConditionalGeneration (text + image); an FP8 checkpoint is also listed.
vllm serve meta-llama/Llama-4-Scout-17B-16E-InstructListed under Llama4ForConditionalGeneration (text + image).
vllm serve meta-models/Muse-Glimmer-30BListed under MuseGlimmerForConditionalGeneration (text + image + video); speculative decoding via the -assistant checkpoint (DFlash).
vllm serve meta-models/Muse-Spark-Open-816B-A42B-NVFP4Not in vLLM v0.32.0 (support PR #61879 still open); vLLM's recipe says to use a Docker image until the code lands and lists vLLM 0.33.0+. Meta's guide gives a vllm serve command.
vllm serve mistralai/Ministral-3-14B-Instruct-2512Mistral recommends vLLM (>= 0.12.0) and serves it with --tokenizer_mode mistral --config_format mistral --load_format mistral.
vllm serve mistralai/Ministral-3-3B-Instruct-2512Mistral recommends vLLM (>= 0.12.0) and serves it with --tokenizer_mode mistral --config_format mistral --load_format mistral.
vllm serve mistralai/Ministral-3-8B-Instruct-2512Mistral recommends vLLM (>= 0.12.0) and serves it with --tokenizer_mode mistral --config_format mistral --load_format mistral.
vllm serve mistralai/Mistral-Large-3-675B-Instruct-2512Listed under MistralLarge3ForCausalLM; Mistral's card recommends vLLM with tensor parallel 8.
vllm serve mistralai/Mistral-Medium-3.5-128BMistral's card names vLLM the recommended deployment and links an EAGLE draft model for speculative decoding.
vllm serve mistralai/Mistral-Small-4-119B-2603Mistral's card recommends vLLM for production inference and gives a vllm serve command.
vllm serve moonshotai/Kimi-K3Listed under KimiK3ForConditionalGeneration (text + image); Moonshot's card links a vLLM recipe.
vllm serve nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4NVIDIA's card recommends the vllm/vllm-openai:v0.22.0 container; up to 1M context with VLLM_ALLOW_LONG_MAX_MODEL_LEN=1.
vllm serve openai/gpt-oss-120bOpenAI's card serves it with vllm==0.10.1+gptoss; listed on vLLM's supported-models page.
vllm serve openai/gpt-oss-20bOpenAI's card serves it with vllm==0.10.1+gptoss; listed on vLLM's supported-models page.
vllm serve zai-org/GLM-5.3Z.ai's card lists vLLM with a recipe; architecture GlmMoeDsaForCausalLM is on vLLM's supported list.
Only models with an official source are listed. vLLM may run others with the same architecture; see its supported-models page.