ModelCards

Text Generation Inference (TGI)

A Rust, Python and gRPC server for serving open LLMs, used by Hugging Face for HuggingChat and Inference Endpoints; now archived, with vLLM and SGLang recommended instead.

GitHub ↗Docs ↗Supported models ↗

Maintenance mode: the README states 'text-generation-inference is now in maintenance mode' and that only minor bug fixes, documentation and lightweight maintenance PRs are accepted (https://github.com/huggingface/text-generation-inference, added in v3.3.7 via PR #3344).

The GitHub repository was archived by the owner on 21 Mar 2026 and is now read-only (https://github.com/huggingface/text-generation-inference). The last release is v3.3.7 (19 Dec 2025), so models released after that are not covered.

The README's Docker example pins image tag 3.3.5, not the latest release 3.3.7.

At a glance

Built by
Hugging Face
Status
Archived
Latest release
v3.3.7 (2025-12-19)
License
Apache-2.0
Written in
Rust, Python
Serves
Language
Hardware
NVIDIA GPU, AMD GPU, AWS Inferentia, Intel GPU, Intel Gaudi, Google TPU
API
HTTP server: native /generate API plus an OpenAI Chat Completions-compatible Messages API
Quantization
bitsandbytes, GPTQ, EETQ, AWQ, Marlin, FP8

From Text Generation Inference (TGI)'s own repo and docs, checked 21 Oct 2026.

Get started

Install
docker pull ghcr.io/huggingface/text-generation-inference:3.3.5
Run a model
docker run --gpus all --shm-size 1g -p 8080:80 -v $PWD/data:/data ghcr.io/huggingface/text-generation-inference:3.3.5 --model-id {model}

Replace {model} with the model ID below.

What it's good at

  • No longer developed

    The README says TGI is in maintenance mode and recommends vLLM, SGLang, llama.cpp or MLX going forward; the GitHub repository was archived (read-only) on 21 Mar 2026. source ↗

  • OpenAI-compatible Messages API

    Besides its own /generate and /generate_stream endpoints, TGI serves /v1/chat/completions compatible with the OpenAI Chat Completion API. source ↗

  • Broad hardware list

    Official images for NVIDIA and AMD (ROCm), plus Inferentia, Intel GPU, Gaudi and Google TPU backends. source ↗

  • Serving features

    Continuous batching, tensor parallelism, SSE token streaming, Flash/Paged Attention, speculation and JSON/grammar-guided output. source ↗

Catalog models it runs 5

  • Gemma 3 27BGoogle · Languagesince v3.2.0source ↗
    docker run --gpus all --shm-size 1g -p 8080:80 -v $PWD/data:/data ghcr.io/huggingface/text-generation-inference:3.3.5 --model-id google/gemma-3-27b-it

    Gemma3 is on the supported-models list; v3.2.0 release notes say 'Added Gemma 3 support'.

  • Llama 3.1 8BMeta · Languagesource ↗
    docker run --gpus all --shm-size 1g -p 8080:80 -v $PWD/data:/data ghcr.io/huggingface/text-generation-inference:3.3.5 --model-id meta-llama/Llama-3.1-8B-Instruct

    Llama (3.1 collection) is on TGI's supported-models list.

  • Llama 3.3 70BMeta · Languagesource ↗
    docker run --gpus all --shm-size 1g -p 8080:80 -v $PWD/data:/data ghcr.io/huggingface/text-generation-inference:3.3.5 --model-id meta-llama/Llama-3.3-70B-Instruct

    Architecture LlamaForCausalLM; 'Llama' is on TGI's supported-models list.

  • Llama 4 MaverickMeta · Languagesince v3.2.2source ↗
    docker run --gpus all --shm-size 1g -p 8080:80 -v $PWD/data:/data ghcr.io/huggingface/text-generation-inference:3.3.5 --model-id meta-llama/Llama-4-Maverick-17B-128E-Instruct

    Llama4 is on the supported-models list; added in v3.2.2 ('Add llama4', PR #3145).

  • Llama 4 ScoutMeta · Languagesince v3.2.2source ↗
    docker run --gpus all --shm-size 1g -p 8080:80 -v $PWD/data:/data ghcr.io/huggingface/text-generation-inference:3.3.5 --model-id meta-llama/Llama-4-Scout-17B-16E-Instruct

    Llama4 is on the supported-models list; added in v3.2.2 ('Add llama4', PR #3145).

Only models with an official source are listed. Text Generation Inference (TGI) may run others with the same architecture; see its supported-models page.