Measured results: TBD. We will add a row only when we have run the test ourselves or can cite a reproducible public run.
Planned method
- llama.cpp release noted per run; same GGUF file per model
- Prompt processing and generation tokens/s at fixed context
- Power draw where measurable
Theoretical ceilings (not benchmarks)
| Hardware | Bandwidth | Ceiling: 8B Q4 | Ceiling: 70B Q4 |
|---|---|---|---|
| GeForce RTX 5090 | 1792 GB/s | ~366 tok/s | ~41 tok/s (does not fit) |
| GeForce RTX 4090 | 1008 GB/s | ~206 tok/s | ~23 tok/s (does not fit) |
| GeForce RTX 5080 | 960 GB/s | ~196 tok/s | ~22 tok/s (does not fit) |
| NVIDIA DGX Spark (128GB) | 273 GB/s | ~56 tok/s | ~6 tok/s |
| GMKtec EVO-X2 (128GB / 1TB) | 256 GB/s | ~52 tok/s | ~6 tok/s |
| Mac Studio (M5 Max 40-core GPU, 128GB) | 614 GB/s | ~125 tok/s | ~14 tok/s |