Quantization stores model weights with fewer bits. Fewer bits mean less memory and faster generation, at some cost in output quality that grows as bits shrink.
| Format | Approx. bits/weight | 8B model weights | 70B model weights |
|---|---|---|---|
| Q4_K_M (≈4.89 bits/weight) | 4.89 | 4.9 GB | 42.8 GB |
| Q5_K_M (≈5.70 bits/weight) | 5.7 | 5.7 GB | 49.9 GB |
| Q8_0 (≈8.50 bits/weight) | 8.5 | 8.5 GB | 74.4 GB |
| FP16 / BF16 (16 bits/weight) | 16 | 16.0 GB | 140.0 GB |
Bits-per-weight values are llama.cpp's published figures for Llama 3.1 8B; other models vary slightly. Source: llama.cpp: quantize README (bits/weight table). Weight sizes are computed (parameters × bits ÷ 8), excluding KV cache and overhead.
Some models ship natively quantized: OpenAI post-trained gpt-oss with MXFP4 quantization of its mixture-of-experts weights (Hugging Face: openai/gpt-oss-120b model card).
Which level should you use?
Q4_K_M is a common default for local use because it roughly quarters FP16 memory. If you have spare memory, Q5 or Q8 trades memory for quality. Quality impact varies by model and task; we have not published our own quality measurements (TBD).