TensorRT-LLM
NVIDIA's open-source library and server for fast LLM (and visual generation) inference on NVIDIA GPUs, with a Python API and an OpenAI-compatible trtllm-serve.
GitHub's 'latest' stable release is v1.2.1 (20 Apr 2026); newer builds are release candidates (v1.3.0rc1-rc29, v1.4.0rc0/rc1). Rows marked partial below are listed in release-candidate docs but not in v1.2.1's supported-models page.
The default pip wheel is Linux-only and needs a matching CUDA Toolkit and PyTorch (https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/installation/installation-guide.md); NVIDIA also ships NGC containers.
At a glance
- Built by
- NVIDIA
- Status
- Active
- Latest release
- v1.2.1 (2026-04-20)
- License
- Apache-2.0
- Written in
- Python
- Serves
- Language, Embedding, Image, Video
- Hardware
- NVIDIA GPU
- API
- OpenAI-compatible HTTP server (trtllm-serve); also a Python LLM API
- Quantization
- NVFP4/FP4, FP8 (per-tensor, block scaling, rowwise), FP8 KV cache, NVFP4 KV cache, W4A16 GPTQ, W4A8 GPTQ, W4A16 AWQ, W4A8 AWQ
From TensorRT-LLM's own repo and docs, checked 21 Oct 2026.
Get started
pip3 install tensorrt_llmtrtllm-serve {model}Replace {model} with the model ID below.
What it's good at
- NVIDIA GPUs only
Supported architectures are Blackwell (B200, GB200, B300, GB300, DGX Spark), Hopper (H100, H200, GH200), Ada Lovelace (L20, L40/L40S) and Ampere (A100). source ↗
- Low-precision formats
FP4 and FP8 recipes on Blackwell and Hopper, plus GPTQ/AWQ 4-bit weights; runs pre-quantized checkpoints from NVIDIA Model Optimizer directly. source ↗
- Fast-moving release candidates
The last stable release is v1.2.1 (Apr 2026); most 2026 model support (DeepSeek-V4, Kimi K3, GLM-5.3, Qwen3.8, Gemma 4, Nemotron 3 Ultra) is only in v1.3/v1.4 release candidates. source ↗
- Per-model deployment guides
Tuned trtllm-serve recipes for gpt-oss, Llama 3.3 70B, Llama 4 Scout, Kimi K3, GLM-5, Qwen3.8 and Nemotron 3, among others. source ↗
- Visual generation (beta)
Diffusion image and video models such as FLUX.2-dev, Wan 2.x, LTX-2 and Qwen-Image through the VisualGen API and trtllm-serve. source ↗
Catalog models it runs 21
trtllm-serve Qwen/Qwen3.8-27BArchitecture Qwen3_5ForConditionalGeneration is in the multimodal matrix of release-candidate docs only, not v1.2.1.
trtllm-serve Qwen/Qwen3.8-Flash-NextQwen4ExpForConditionalGeneration is listed only in release-candidate docs; most multimodal features are Untested.
trtllm-serve Qwen/Qwen3.8-2.4T-A95BQwen3_5MoeForCausalLM with this repo is listed only in release-candidate docs, not v1.2.1.
trtllm-serve deepseek-ai/DeepSeek-V4-Pro-0813DeepseekV4ForCausalLM is listed only in release-candidate docs (not v1.2.1) and is supported only on Blackwell GPUs (SM100+).
trtllm-serve deepseek-ai/DeepSeek-V4.1-FlashDeepseekV41ForCausalLM (text decoder) first appears in the v1.4.0rc1 docs; not in any stable release.
trtllm-serve google/gemma-3-27b-itArchitecture Gemma3ForConditionalGeneration is in the multimodal support matrix (language + image).
trtllm-serve google/gemma-4-31b-itGemma4ForConditionalGeneration with google/gemma-4-31B-it is listed only in release-candidate docs; disaggregated serving and KV cache reuse are marked No.
trtllm-serve meta-llama/Llama-3.1-8B-InstructLlamaForCausalLM (Llama 3.1) is on the supported list.
trtllm-serve meta-llama/Llama-3.3-70B-InstructLlamaForCausalLM is on the supported list, and there is a dedicated Llama 3.3 70B deployment guide.
trtllm-serve meta-llama/Llama-4-Maverick-17B-128E-InstructArchitecture Llama4ForConditionalGeneration is on the supported list.
trtllm-serve meta-llama/Llama-4-Scout-17B-16E-InstructLlama4ForConditionalGeneration listed with this repo as the example; has a deployment guide.
trtllm-serve mistralai/Ministral-3-14B-Instruct-2512Architecture Mistral3ForConditionalGeneration is in the multimodal support matrix.
trtllm-serve mistralai/Ministral-3-3B-Instruct-2512Architecture Mistral3ForConditionalGeneration is in the multimodal support matrix.
trtllm-serve mistralai/Ministral-3-8B-Instruct-2512Architecture Mistral3ForConditionalGeneration is in the multimodal support matrix.
trtllm-serve mistralai/Mistral-Large-3-675B-Instruct-2512Has an official example (multimodal and LLM-only runs, Mistral checkpoint format, EAGLE-3) in v1.2.1 and later.
trtllm-serve mistralai/Mistral-Medium-3.5-128BArchitecture Mistral3ForConditionalGeneration is in the multimodal support matrix.
trtllm-serve moonshotai/Kimi-K3Listed only in release-candidate docs and supported only on Blackwell (SM100 family); validated on 16x GB200.
trtllm-serve nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4NVIDIA's model card gives a TRT-LLM recipe using the 1.3.0rc17 release-candidate container; not in v1.2.1's list.
trtllm-serve openai/gpt-oss-120bGptOssForCausalLM listed with openai/gpt-oss-120b as the example; has a deployment guide.
trtllm-serve openai/gpt-oss-20bArchitecture GptOssForCausalLM is on the v1.2.1 supported list; openai/gpt-oss-20b is named in later docs.
trtllm-serve zai-org/GLM-5.3GlmMoeDsaForCausalLM with zai-org/GLM-5.3 is listed only in release-candidate docs, not v1.2.1.
Only models with an official source are listed. TensorRT-LLM may run others with the same architecture; see its supported-models page.