LMDeploy
A toolkit for compressing, deploying and serving LLMs and VLMs, with a C++/CUDA engine (TurboMind) and a pure-Python PyTorch engine.
The installation guide lists NVIDIA Volta through Ada Lovelace GPUs and says LMDeploy runs on Linux and Windows (https://github.com/InternLM/lmdeploy/blob/main/docs/en/get_started/installation.md); since v0.13.0 the PyPI wheels are built against CUDA 12.8 and cover RTX 50 series (README).
Many newer models (DeepSeek-V4, GLM-5.x, Llama 4, Gemma 3) are PyTorch-engine only, with no KV or weight quantization marked as supported.
At a glance
- Built by
- MMRazor and MMDeploy teams (InternLM)
- Status
- Active
- Latest release
- v0.18.0 (2026-09-28)
- License
- Apache-2.0
- Written in
- Python
- Serves
- Language
- Hardware
- NVIDIA GPU, Huawei Ascend NPU, MetaX Maca GPU, Cambricon MLU
- API
- OpenAI-compatible HTTP server (lmdeploy serve api_server); also an offline Python pipeline
- Quantization
- AWQ (W4A16), GPTQ (W4A16, inference), W8A8 INT8 (SmoothQuant), W8A8 FP8, KV cache INT8/INT4, MXFP4 (gpt-oss), llm-compressor 4-bit
From LMDeploy's own repo and docs, checked 21 Oct 2026.
Get started
pip install lmdeploylmdeploy serve api_server {model}Replace {model} with the model ID below.
What it's good at
- Two engines
TurboMind (C++/CUDA) targets peak performance; the PyTorch engine is pure Python and covers more architectures. Support differs per engine, so check the table for the one you plan to use. source ↗
- Quantization built in
lmdeploy lite quantizes to 4-bit AWQ and 8-bit SmoothQuant (INT8, or FP8 on GPUs such as H100); TurboMind also runs GPTQ 4-bit models and online INT8/INT4 KV cache. source ↗
- MXFP4 gpt-oss on older GPUs
TurboMind runs MXFP4 on NVIDIA GPUs from V100 onward (news item, Sep 2025). source ↗
- Non-NVIDIA accelerators
The PyTorch engine has support tables for Huawei Ascend (Atlas 800T A2/A3, 300I Duo), MetaX Maca C500 and Cambricon. source ↗
Catalog models it runs 10
lmdeploy serve api_server Qwen/Qwen3.8-27BArchitecture Qwen3_5ForConditionalGeneration is in LMDeploy's PyTorch model map (Qwen3.5 family listed); TurboMind does not support the Qwen3.5-series vision encoder.
lmdeploy serve api_server deepseek-ai/DeepSeek-V4-Pro-0813DeepSeek-V4 (284B, 1.6T) listed for the PyTorch engine, BF16 only (no KV or weight quantization).
lmdeploy serve api_server google/gemma-3-27b-itGemma3 1B-27B listed for the PyTorch engine only.
lmdeploy serve api_server meta-llama/Llama-3.1-8B-InstructLlama3.1 8B is listed for both TurboMind and the PyTorch engine, and for Ascend, Maca and Cambricon.
lmdeploy serve api_server meta-llama/Llama-3.3-70B-InstructArchitecture LlamaForCausalLM is in LMDeploy's PyTorch model map (Llama3.1 70B is listed in the table; 3.3 is not named).
lmdeploy serve api_server meta-llama/Llama-4-Maverick-17B-128E-InstructLlama4 Maverick listed for the PyTorch engine only (BF16, KV INT8/INT4).
lmdeploy serve api_server meta-llama/Llama-4-Scout-17B-16E-InstructLlama4 Scout listed for the PyTorch engine only (BF16, KV INT8/INT4).
lmdeploy serve api_server openai/gpt-oss-120bgpt-oss 20B/120B listed for TurboMind (BF16, KV INT8/INT4, W4A16); GptOssForCausalLM is also in the PyTorch engine's model map.
lmdeploy serve api_server openai/gpt-oss-20bgpt-oss 20B/120B listed for TurboMind (BF16, KV INT8/INT4, W4A16); GptOssForCausalLM is also in the PyTorch engine's model map.
lmdeploy serve api_server zai-org/GLM-5.3Architecture GlmMoeDsaForCausalLM is in LMDeploy's PyTorch model map (the table names GLM-5 and GLM-5.2, not 5.3).
Only models with an official source are listed. LMDeploy may run others with the same architecture; see its supported-models page.