What GPU do you need to run Gemma 2 27B?
27.2B params 8K max context
No single GPU in our database runs Gemma 2 27B comfortably; 47 can load it with very limited context.
MMLU 75.2
VRAM needed per quantization
| Quantization | Weights | 16K context | Full 8K |
|---|---|---|---|
| Q4_K_M | 16.6 GB | 19.5 GB | 19.5 GB |
| Q8_0 | 28.9 GB | 31.7 GB | 31.7 GB |
Weights plus fp16 KV cache. Leave ~1 GB headroom for the inference engine.
GPUs that can run Gemma 2 27B
Best quantization that fits on each GPU, with the context and speed it gets. For chips with several memory sizes, the smallest that works is shown.
NVIDIA
- RTX 4000 Ada 20 GB Q4_K_M 8K context · ~20 tok/s tight fit
- RTX 3090 24 GB Q4_K_M 8K context · ~53 tok/s tight fit
- RTX 3090 Ti 24 GB Q4_K_M 8K context · ~57 tok/s tight fit
- RTX 4090 24 GB Q4_K_M 8K context · ~57 tok/s tight fit
- RTX 4500 Ada 24 GB Q4_K_M 8K context · ~24 tok/s tight fit
- RTX A5000 24 GB Q4_K_M 8K context · ~44 tok/s tight fit
- RTX PRO 4000 Blackwell 24 GB Q4_K_M 8K context · ~38 tok/s tight fit
- RTX Spark 24 GB Q4_K_M 8K context · ~17 tok/s tight fit
- Tesla P40 24 GB Q4_K_M 8K context · ~20 tok/s tight fit
- RTX 5090 32 GB Q8_0 8K context · ~60 tok/s tight fit
- RTX 5000 Ada 32 GB Q8_0 8K context · ~19 tok/s tight fit
- RTX PRO 4500 Blackwell 32 GB Q8_0 8K context · ~30 tok/s tight fit
- Tesla V100 32GB 32 GB Q8_0 8K context · ~30 tok/s tight fit
- L40S 48 GB Q8_0 8K context · ~29 tok/s tight fit
- RTX 6000 Ada 48 GB Q8_0 8K context · ~32 tok/s tight fit
- RTX A6000 48 GB Q8_0 8K context · ~26 tok/s tight fit
- RTX PRO 5000 Blackwell 48 GB Q8_0 8K context · ~45 tok/s tight fit
- RTX PRO 5000 Blackwell 72GB 72 GB Q8_0 8K context · ~45 tok/s tight fit
- RTX PRO 5500 Blackwell 84 GB Q8_0 8K context · ~47 tok/s tight fit
- RTX PRO 6000 Blackwell 96 GB Q8_0 8K context · ~60 tok/s tight fit
- DGX Spark (GB 10) 128 GB Q8_0 8K context · ~9 tok/s tight fit
AMD
- Radeon RX 7900 XT 20 GB Q4_K_M 8K context · ~45 tok/s tight fit
- Radeon RX 7900 XTX 24 GB Q4_K_M 8K context · ~54 tok/s tight fit
- Radeon AI PRO R9700 32 GB Q8_0 8K context · ~21 tok/s tight fit
- Ryzen AI Max (Strix Halo) 32 GB Q8_0 8K context · ~9 tok/s tight fit
- Ryzen AI Max 400 (Gorgon Halo) 64 GB Q8_0 8K context · ~9 tok/s tight fit
Intel
- Arc Pro B65 32 GB Q8_0 8K context · ~20 tok/s tight fit
- Arc Pro B70 32 GB Q8_0 8K context · ~20 tok/s tight fit
Apple
- M3 Pro 18 GB Q4_K_M 3K context · ~8 tok/s tight fit
- M2 24 GB Q4_K_M 8K context · ~6 tok/s tight fit
- M3 24 GB Q4_K_M 8K context · ~6 tok/s tight fit
- M4 24 GB Q4_K_M 8K context · ~7 tok/s tight fit
- M4 Pro 24 GB Q4_K_M 8K context · ~15 tok/s tight fit
- M5 24 GB Q4_K_M 8K context · ~9 tok/s tight fit
- M5 Pro 24 GB Q4_K_M 8K context · ~17 tok/s tight fit
- M6 24 GB Q4_K_M 8K context · ~10 tok/s tight fit
- M1 Max 32 GB Q8_0 8K context · ~13 tok/s tight fit
- M1 Pro 32 GB Q8_0 8K context · ~7 tok/s tight fit
- M2 Max 32 GB Q8_0 8K context · ~13 tok/s tight fit
- M2 Pro 32 GB Q8_0 8K context · ~7 tok/s tight fit
- M3 Max 36 GB Q8_0 8K context · ~10 tok/s tight fit
- M4 Max 36 GB Q8_0 8K context · ~14 tok/s tight fit
- M5 Max 36 GB Q8_0 8K context · ~15 tok/s tight fit
- M1 Ultra 64 GB Q8_0 8K context · ~27 tok/s tight fit
- M2 Ultra 64 GB Q8_0 8K context · ~27 tok/s tight fit
- M3 Ultra 96 GB Q8_0 8K context · ~27 tok/s tight fit
- M5 Ultra 96 GB Q8_0 8K context · ~40 tok/s tight fit