SGLang
An open-source serving framework for large language, vision-language and diffusion models, aimed at large-scale serving, agentic workloads and RL rollouts.
Several day-0 models ship first in preview Docker images (e.g. lmsysorg/sglang:dev-dsv41) before a tagged release; the per-model cookbook pages say which image to use (https://cookbook.sglang.io/).
Decoder-style embedding models need --is-embedding (https://github.com/sgl-project/sglang/blob/v0.5.22/docs/docs/supported-models/embedding_models.mdx).
Image and video models (SGLang Diffusion) were not checked against this catalog.
At a glance
- Built by
- SGLang project, hosted by LMSYS (a non-profit open-source organization)
- Status
- Active
- Latest release
- v0.5.22 (2026-10-19)
- License
- Apache-2.0
- Written in
- Python
- Serves
- Language, Embedding, Image, Video
- Hardware
- NVIDIA GPU, AMD GPU, Google TPU, Intel GPU, Intel Xeon CPU, Apple Silicon, Huawei Ascend NPU, Moore Threads GPU
- API
- OpenAI-compatible HTTP server
- Quantization
- FP8, MXFP4, MXFP8, NVFP4, W8A8 INT8, W8A8 FP8, AWQ, GPTQ (Marlin), GGUF, compressed-tensors, ModelOpt, bitsandbytes
From SGLang's own repo and docs, checked 21 Oct 2026.
Get started
uv pip install --prerelease=allow sglangsglang serve {model}Replace {model} with the model ID below.
What it's good at
- RadixAttention prefix caching
Built for low-latency, high-throughput inference with RadixAttention, prefix caching and multi-GPU parallelism. source ↗
- Per-model cookbook
The Cookbook gives a ready-to-run launch command per model and hardware, including Docker images and speculative-decoding settings. source ↗
- Broad hardware
NVIDIA (Ampere to Blackwell, DGX Spark, Jetson), AMD Instinct MI300X–MI355X, Google TPU, Intel GPU and Xeon, Apple Silicon, Huawei Ascend and Moore Threads. source ↗
- Image and video generation built in
SGLang Diffusion, shipped in the same Python package, serves image and video diffusion models. source ↗
- Many quantization paths
FP8, FP4 (NVFP4, MXFP4), AWQ, GPTQ-Marlin, GGUF, ModelOpt and more, with selectable FP8/FP4 GEMM backends. source ↗
Catalog models it runs 26
sglang serve Qwen/Qwen3-Embedding-8BQwen3-Embedding is on SGLang's embedding models page; serve with --is-embedding.
sglang serve google/embeddinggemma-2Cookbook page; SGLang auto-detects EmbeddingGemma2Model, so --is-embedding is not needed.
sglang serve Qwen/Qwen3.8-27BCookbook page (BF16, FP8, NVFP4) on lmsysorg/sglang:latest; Qwen's card links it.
sglang serve Qwen/Qwen3.8-Flash-NextCookbook says v0.5.20 or later; single-DGX-Spark serving with the offloaded PLE table needs the dev-qwen38-next-local image.
sglang serve Qwen/Qwen3.8-2.4T-A95BCookbook page; NVIDIA uses the lmsysorg/sglang:qwen38 image, AMD pinned v0.5.17 builds.
sglang serve CohereLabs/command-a-plus-05-2026-bf16Architecture Cohere2VisionForConditionalGeneration is implemented in SGLang v0.5.22 (models/cohere2_vision.py); no cookbook page.
sglang serve deepseek-ai/DeepSeek-V4-Pro-0813Cookbook covers Pro Official (0813) on lmsysorg/sglang:latest; DeepSeek's card gives an sglang serve command with DSpark.
sglang serve deepseek-ai/DeepSeek-V4.1-FlashNeeds the preview image lmsysorg/sglang:dev-dsv41 on NVIDIA (not in the v0.5.22 release); AMD MI355X uses a v0.5.21 ROCm build.
sglang serve google/gemma-3-27b-itGemma 3 4B/12B/27B multimodal is supported, but bidirectional attention between image tokens only works with the Triton attention backend (CUDA graph and chunked prefill off).
sglang serve google/gemma-4-31B-itCookbook page lists gemma-4-31B-it and its -assistant draft model.
sglang serve meta-llama/Llama-3.1-8B-InstructLlama 3.1 cookbook covers the 8B, 70B and 405B sizes.
sglang serve meta-llama/Llama-3.3-70B-InstructCookbook page (written for AMD GPUs).
sglang serve meta-llama/Llama-4-Maverick-17B-128E-InstructLlama 4 cookbook covers Scout and Maverick.
sglang serve meta-llama/Llama-4-Scout-17B-16E-InstructLlama 4 cookbook covers Scout and Maverick.
sglang serve meta-models/Muse-Glimmer-30BMuseGlimmerForConditionalGeneration is in SGLang v0.5.22; the cookbook's Docker path uses the image lmsysorg/sglang:dev-muse-glimmer.
sglang serve meta-models/Muse-Spark-Open-816B-A42B-NVFP4Not in the v0.5.22 release; the code is on SGLang main and the cookbook uses the preview image lmsysorg/sglang:dev-muse-spark-open. Meta's guide gives an sglang serve command (BF16 and NVFP4).
sglang serve mistralai/Ministral-3-14B-Instruct-2512Ministral-3 cookbook page uses this checkpoint.
sglang serve mistralai/Ministral-3-3B-Instruct-2512Architecture Mistral3ForConditionalGeneration is implemented in SGLang v0.5.22 (models/mistral.py); the cookbook's example uses the 14B.
sglang serve mistralai/Ministral-3-8B-Instruct-2512Architecture Mistral3ForConditionalGeneration is implemented in SGLang v0.5.22 (models/mistral.py); the cookbook's example uses the 14B.
sglang serve mistralai/Mistral-Medium-3.5-128BCookbook page on lmsysorg/sglang:latest; Mistral's card also documents SGLang and an EAGLE draft model.
sglang serve mistralai/Mistral-Small-4-119B-2603Cookbook page; an NVFP4 checkpoint and an EAGLE draft are also listed.
sglang serve moonshotai/Kimi-K3Cookbook page; NVIDIA uses lmsysorg/sglang:latest. Tool calling is not yet supported on the Ascend NPU (A3/A5) recipes.
sglang serve nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4Cookbook page (BF16 and NVFP4, up to 1M context); NVIDIA's card also documents SGLang.
sglang serve openai/gpt-oss-120bCookbook page; listed on SGLang's generative models page.
sglang serve openai/gpt-oss-20bCookbook page; listed on SGLang's generative models page.
sglang serve zai-org/GLM-5.3Cookbook page (H200, B200, B300, GB300, MI300X/MI325X/MI355X) on lmsysorg/sglang:latest.
Only models with an official source are listed. SGLang may run others with the same architecture; see its supported-models page.