llama.cpp
A dependency-free C/C++ engine for running GGUF models locally or in the cloud, from laptops and phones to multi-GPU servers.
Since v0.7.0 the server listens on port 9931 by default instead of 8080 (Docker images keep 8080 inside the container) — release notes for v0.7.0.
Besides tagged v releases, prerelease builds (bNNNNN) are published several times a day on the GitHub releases page.
Rows that point at a safetensors repo instead of a GGUF repo mean no official GGUF exists yet: convert it yourself with convert_hf_to_gguf.py; support is based on the converter's registered architectures.
At a glance
- Built by
- ggml-org
- Status
- Active
- Latest release
- v0.7.0 (2026-10-15)
- License
- MIT
- Written in
- C++
- Serves
- Language, Embedding
- Hardware
- CPU (x86, ARM, RISC-V), Apple Silicon (Metal), NVIDIA GPU, AMD GPU, Intel GPU, Moore Threads GPU, Vulkan GPUs, Ascend NPU, Snapdragon (Hexagon, Adreno), IBM Z
- API
- OpenAI-compatible HTTP server (llama serve)
- Quantization
- 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, 8-bit integer (GGUF)
From llama.cpp's own repo and docs, checked 21 Oct 2026.
Get started
curl -LsSf https://llama.app/install.sh | shllama serve -hf {model}Replace {model} with the model ID below.
What it's good at
- Runs almost anywhere
Backends include CUDA, HIP, Metal, Vulkan, SYCL, MUSA, CANN, OpenCL, Hexagon and WebGPU, plus optimized CPU paths (AVX/AVX512/AMX, ARM NEON, RISC-V). source ↗
- Models larger than your VRAM
CPU+GPU hybrid inference partially accelerates models that don't fit in VRAM; v0.7.0 adds a GPU cache for MoE experts kept in host memory (--moe-cache-mib), up to about 2.2x faster decode. source ↗
- GGUF only, 1.5- to 8-bit
Models must be in GGUF; other formats are converted with the repo's convert_hf_to_gguf.py, and integer quantization goes from 1.5-bit to 8-bit. source ↗
- Pull straight from Hugging Face
`llama serve -hf <user>/<model>[:quant]` downloads a GGUF repo and starts an OpenAI-compatible server with a built-in web UI. source ↗
Catalog models it runs 31
llama serve -hf Qwen/Qwen3-Embedding-8B-GGUFOfficial GGUF from Qwen; serve with --embedding --pooling last.
llama serve -hf ggml-org/embeddinggemma-2-GGUFText, vision and audio embeddings.
llama serve -hf jinaai/jina-embeddings-v5-omni-small-retrieval-GGUFJina's GGUF needs its feat-v5-omni llama.cpp fork (multimodal patches not yet upstream); one GGUF repo per task.
llama serve -hf jinaai/jina-embeddings-v5-text-small-retrieval-GGUFJina publishes one GGUF per task (retrieval, text-matching, classification, clustering) and builds against upstream llama.cpp.
llama serve -hf nomic-ai/nomic-embed-text-v2-moe-GGUFOfficial GGUF from Nomic; needs task instruction prefixes (e.g. search_query) on each input.
- nvidia/Nemotron-3.5-Embed-1B-BF16
No official GGUF; the converter registers Mistral3Model as a Nemotron Embed model and names this repo as its example.
- nvidia/Nemotron-3.5-Embed-8B-BF16
No official GGUF; architecture Mistral3Model (Nemotron Embed) is registered in the converter.
llama serve -hf ggml-org/Qwen3.8-27B-GGUFllama serve -hf ggml-org/Qwen3.8-Flash-Next-GGUFThese are the open Qwen3.8-Flash-Next weights the API model is based on; includes vision mmproj, MTP and DFlash2 draft sidecars.
- Qwen/Qwen3.8-2.4T-A95B
No official GGUF; architecture Qwen3_5MoeForCausalLM is registered in the converter. The open weights are text-only.
- CohereLabs/command-a-plus-05-2026-bf16
No official GGUF; the converter registers Cohere2VisionForConditionalGeneration and Cohere2MoeForCausalLM and names this repo as its example.
- deepseek-ai/DeepSeek-V4-Pro-0813
No official GGUF; architecture DeepseekV4ForCausalLM is registered in the converter (ggml-org only publishes V4-Flash GGUFs).
llama serve -hf ggml-org/gemma-3-27b-it-GGUFGoogle also publishes a QAT Q4_0 GGUF (google/gemma-3-27b-it-qat-q4_0-gguf).
llama serve -hf ggml-org/gemma-4-31B-it-GGUFGoogle also publishes a QAT Q4_0 GGUF (google/gemma-4-31B-it-qat-q4_0-gguf).
llama serve -hf ggml-org/Meta-Llama-3.1-8B-Instruct-Q4_0-GGUFThe ggml-org repo has a Q4_0 file only.
- meta-llama/Llama-3.3-70B-Instruct
No official GGUF; architecture LlamaForCausalLM is registered in the converter.
- meta-llama/Llama-4-Maverick-17B-128E-Instruct
No official GGUF; architecture Llama4ForConditionalGeneration is registered in the converter.
llama serve -hf ggml-org/Llama-4-Scout-17B-16E-Instruct-GGUFllama serve -hf meta-models/Muse-Glimmer-30B-GGUFOfficial GGUF from Meta; ggml-org's copy includes a Q8_0 mmproj for image input.
llama serve -hf ggml-org/Muse-Spark-Open-816B-A42B-GGUFIncludes mmproj files for the vision and audio encoders.
llama serve -hf mistralai/Ministral-3-14B-Instruct-2512-GGUFOfficial GGUF from Mistral AI; ggml-org has one too.
llama serve -hf mistralai/Ministral-3-3B-Instruct-2512-GGUFOfficial GGUF from Mistral AI; ggml-org has one too.
llama serve -hf mistralai/Ministral-3-8B-Instruct-2512-GGUFOfficial GGUF from Mistral AI; ggml-org has one too.
- mistralai/Mistral-Large-3-675B-Instruct-2512
No official GGUF; weights ship in Mistral format, so convert with --mistral-format (converter support merged in PR #17730).
- mistralai/Mistral-Medium-3.5-128B
No official GGUF; architecture Mistral3ForConditionalGeneration (ministral3 text model) is registered in the converter.
- mistralai/Mistral-Small-4-119B-2603
No official GGUF; the converter handles Mistral3ForConditionalGeneration with a mistral4 text model.
- moonshotai/Kimi-K3
No official GGUF; architecture KimiK3ForConditionalGeneration is registered in the converter with this repo as its example.
- nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16
No official GGUF; architecture NemotronHForCausalLM is registered in the converter (convert from NVIDIA's BF16 repo).
llama serve -hf ggml-org/gpt-oss-120b-GGUFllama serve -hf ggml-org/gpt-oss-20b-GGUF- zai-org/GLM-5.3
No official GGUF; architecture GlmMoeDsaForCausalLM is registered in the converter.
Only models with an official source are listed. llama.cpp may run others with the same architecture; see its supported-models page.