What GPU do you need to run Gemma 2 9B?

9.24B params 8K max context

No single GPU in our database runs Gemma 2 9B comfortably; 120 can load it with very limited context.

MMLU 71.3

VRAM needed per quantization

Quantization	Weights	16K context	Full 8K
Q4_K_M	5.8 GB	8.3 GB	8.3 GB
Q8_0	9.8 GB	12.4 GB	12.4 GB

Weights plus fp16 KV cache. Leave ~1 GB headroom for the inference engine.

GPUs that can run Gemma 2 9B

Best quantization that fits on each GPU, with the context and speed it gets. For chips with several memory sizes, the smallest that works is shown.

NVIDIA

GTX 1070 8 GB Q4_K_M 6K context · ~38 tok/s tight fit
GTX 1070 Ti 8 GB Q4_K_M 6K context · ~38 tok/s tight fit
GTX 1080 8 GB Q4_K_M 6K context · ~47 tok/s tight fit
RTX 2060 SUPER 8 GB Q4_K_M 6K context · ~66 tok/s tight fit
RTX 2070 8 GB Q4_K_M 6K context · ~66 tok/s tight fit
RTX 2070 SUPER 8 GB Q4_K_M 6K context · ~66 tok/s tight fit
RTX 2080 8 GB Q4_K_M 6K context · ~66 tok/s tight fit
RTX 2080 SUPER 8 GB Q4_K_M 6K context · ~73 tok/s tight fit
RTX 3050 8 GB Q4_K_M 6K context · ~33 tok/s tight fit
RTX 3060 Ti 8 GB Q4_K_M 6K context · ~66 tok/s tight fit
RTX 3060 Ti GDDR6X 8 GB Q4_K_M 6K context · ~90 tok/s tight fit
RTX 3070 8 GB Q4_K_M 6K context · ~66 tok/s tight fit
RTX 3070 Ti 8 GB Q4_K_M 6K context · ~90 tok/s tight fit
RTX 4060 8 GB Q4_K_M 6K context · ~40 tok/s tight fit
RTX 4060 Ti 8 GB Q4_K_M 6K context · ~43 tok/s tight fit
RTX 5050 8 GB Q4_K_M 6K context · ~47 tok/s tight fit
RTX 5060 8 GB Q4_K_M 6K context · ~66 tok/s tight fit
RTX 5060 Ti 8 GB Q4_K_M 6K context · ~66 tok/s tight fit
T1000 8GB 8 GB Q4_K_M 6K context · ~24 tok/s tight fit
RTX 3080 10 GB Q4_K_M 8K context · ~112 tok/s tight fit
GTX 1080 Ti 11 GB Q8_0 3K context · ~45 tok/s tight fit
RTX 2080 Ti 11 GB Q8_0 3K context · ~57 tok/s tight fit
RTX 2060 12GB 12 GB Q8_0 6K context · ~31 tok/s tight fit
RTX 3060 12 GB Q8_0 6K context · ~33 tok/s tight fit
RTX 3080 12GB 12 GB Q8_0 6K context · ~84 tok/s tight fit
RTX 3080 Ti 12 GB Q8_0 6K context · ~84 tok/s tight fit
RTX 4070 12 GB Q8_0 6K context · ~47 tok/s tight fit
RTX 4070 SUPER 12 GB Q8_0 6K context · ~47 tok/s tight fit
RTX 4070 Ti 12 GB Q8_0 6K context · ~47 tok/s tight fit
RTX 5070 12 GB Q8_0 6K context · ~62 tok/s tight fit
RTX A2000 12GB 12 GB Q8_0 6K context · ~27 tok/s tight fit
RTX 4060 Ti 16GB 16 GB Q8_0 8K context · ~27 tok/s tight fit
RTX 4070 Ti SUPER 16 GB Q8_0 8K context · ~62 tok/s tight fit
RTX 4080 16 GB Q8_0 8K context · ~66 tok/s tight fit
RTX 4080 SUPER 16 GB Q8_0 8K context · ~68 tok/s tight fit
RTX 5060 Ti 16GB 16 GB Q8_0 8K context · ~41 tok/s tight fit
RTX 5070 Ti 16 GB Q8_0 8K context · ~83 tok/s tight fit
RTX 5080 16 GB Q8_0 8K context · ~89 tok/s tight fit
RTX 2000 Ada 16 GB Q8_0 8K context · ~21 tok/s tight fit
RTX A4000 16 GB Q8_0 8K context · ~41 tok/s tight fit
RTX PRO 2000 Blackwell 16 GB Q8_0 8K context · ~27 tok/s tight fit
Tesla T4 16 GB Q8_0 8K context · ~30 tok/s tight fit
Tesla V100 16GB 16 GB Q8_0 8K context · ~83 tok/s tight fit
RTX 4000 Ada 20 GB Q8_0 8K context · ~33 tok/s tight fit
RTX 3090 24 GB Q8_0 8K context · ~86 tok/s tight fit
RTX 3090 Ti 24 GB Q8_0 8K context · ~93 tok/s tight fit
RTX 4090 24 GB Q8_0 8K context · ~93 tok/s tight fit
RTX 4500 Ada 24 GB Q8_0 8K context · ~40 tok/s tight fit
RTX A5000 24 GB Q8_0 8K context · ~71 tok/s tight fit
RTX PRO 4000 Blackwell 24 GB Q8_0 8K context · ~62 tok/s tight fit
RTX Spark 24 GB Q8_0 8K context · ~28 tok/s tight fit
Tesla P40 24 GB Q8_0 8K context · ~32 tok/s tight fit
RTX 5090 32 GB Q8_0 8K context · ~165 tok/s tight fit
RTX 5000 Ada 32 GB Q8_0 8K context · ~53 tok/s tight fit
RTX PRO 4500 Blackwell 32 GB Q8_0 8K context · ~83 tok/s tight fit
Tesla V100 32GB 32 GB Q8_0 8K context · ~83 tok/s tight fit
L40S 48 GB Q8_0 8K context · ~80 tok/s tight fit
RTX 6000 Ada 48 GB Q8_0 8K context · ~89 tok/s tight fit
RTX A6000 48 GB Q8_0 8K context · ~71 tok/s tight fit
RTX PRO 5000 Blackwell 48 GB Q8_0 8K context · ~124 tok/s tight fit
RTX PRO 5000 Blackwell 72GB 72 GB Q8_0 8K context · ~124 tok/s tight fit
RTX PRO 5500 Blackwell 84 GB Q8_0 8K context · ~129 tok/s tight fit
RTX PRO 6000 Blackwell 96 GB Q8_0 8K context · ~165 tok/s tight fit
DGX Spark (GB 10) 128 GB Q8_0 8K context · ~25 tok/s tight fit

AMD

Radeon RX 5700 8 GB Q4_K_M 6K context · ~66 tok/s tight fit
Radeon RX 5700 XT 8 GB Q4_K_M 6K context · ~66 tok/s tight fit
Radeon RX 6600 8 GB Q4_K_M 6K context · ~33 tok/s tight fit
Radeon RX 6600 XT 8 GB Q4_K_M 6K context · ~38 tok/s tight fit
Radeon RX 6650 XT 8 GB Q4_K_M 6K context · ~41 tok/s tight fit
Radeon RX 7600 8 GB Q4_K_M 6K context · ~43 tok/s tight fit
Radeon RX 9060 8 GB Q4_K_M 6K context · ~43 tok/s tight fit
Radeon RX 9060 XT 8 GB Q4_K_M 6K context · ~47 tok/s tight fit
Radeon RX 6700 XT 12 GB Q8_0 6K context · ~35 tok/s tight fit
Radeon RX 6750 XT 12 GB Q8_0 6K context · ~40 tok/s tight fit
Radeon RX 7700 XT 12 GB Q8_0 6K context · ~40 tok/s tight fit
Radeon RX 9070 GRE 12 GB Q8_0 6K context · ~40 tok/s tight fit
AMD BC-250 16 GB Q8_0 8K context · ~41 tok/s tight fit
Radeon RX 6800 16 GB Q8_0 8K context · ~47 tok/s tight fit
Radeon RX 6800 XT 16 GB Q8_0 8K context · ~47 tok/s tight fit
Radeon RX 6900 XT 16 GB Q8_0 8K context · ~47 tok/s tight fit
Radeon RX 6950 XT 16 GB Q8_0 8K context · ~53 tok/s tight fit
Radeon RX 7600 XT 16 GB Q8_0 8K context · ~27 tok/s tight fit
Radeon RX 7800 XT 16 GB Q8_0 8K context · ~58 tok/s tight fit
Radeon RX 7900 GRE 16 GB Q8_0 8K context · ~53 tok/s tight fit
Radeon RX 9060 XT 16GB 16 GB Q8_0 8K context · ~30 tok/s tight fit
Radeon RX 9070 16 GB Q8_0 8K context · ~59 tok/s tight fit
Radeon RX 9070 XT 16 GB Q8_0 8K context · ~59 tok/s tight fit
Radeon RX 7900 XT 20 GB Q8_0 8K context · ~74 tok/s tight fit
Radeon RX 7900 XTX 24 GB Q8_0 8K context · ~89 tok/s tight fit
Radeon AI PRO R9700 32 GB Q8_0 8K context · ~59 tok/s tight fit
Ryzen AI Max (Strix Halo) 32 GB Q8_0 8K context · ~24 tok/s tight fit
Ryzen AI Max 400 (Gorgon Halo) 64 GB Q8_0 8K context · ~25 tok/s tight fit

Intel

Arc A580 8 GB Q4_K_M 6K context · ~76 tok/s tight fit
Arc A750 8 GB Q4_K_M 6K context · ~76 tok/s tight fit
Arc A770 8GB 8 GB Q4_K_M 6K context · ~76 tok/s tight fit
Arc B570 10 GB Q4_K_M 8K context · ~56 tok/s tight fit
Arc B580 12 GB Q8_0 6K context · ~42 tok/s tight fit
Arc A770 16GB 16 GB Q8_0 8K context · ~52 tok/s tight fit
Arc Pro B65 32 GB Q8_0 8K context · ~56 tok/s tight fit
Arc Pro B70 32 GB Q8_0 8K context · ~56 tok/s tight fit

Apple

M1 8 GB Q4_K_M 6K context · ~10 tok/s tight fit
M2 8 GB Q4_K_M 6K context · ~15 tok/s tight fit
M3 8 GB Q4_K_M 6K context · ~15 tok/s tight fit
M1 Pro 16 GB Q8_0 8K context · ~18 tok/s tight fit
M2 Pro 16 GB Q8_0 8K context · ~18 tok/s tight fit
M4 16 GB Q8_0 8K context · ~11 tok/s tight fit
M5 16 GB Q8_0 8K context · ~14 tok/s tight fit
M6 16 GB Q8_0 8K context · ~14 tok/s tight fit
M3 Pro 18 GB Q8_0 8K context · ~14 tok/s tight fit
M4 Pro 24 GB Q8_0 8K context · ~25 tok/s tight fit
M5 Pro 24 GB Q8_0 8K context · ~28 tok/s tight fit
M1 Max 32 GB Q8_0 8K context · ~37 tok/s tight fit
M2 Max 32 GB Q8_0 8K context · ~37 tok/s tight fit
M3 Max 36 GB Q8_0 8K context · ~28 tok/s tight fit
M4 Max 36 GB Q8_0 8K context · ~38 tok/s tight fit
M5 Max 36 GB Q8_0 8K context · ~42 tok/s tight fit
M1 Ultra 64 GB Q8_0 8K context · ~74 tok/s tight fit
M2 Ultra 64 GB Q8_0 8K context · ~74 tok/s tight fit
M3 Ultra 96 GB Q8_0 8K context · ~76 tok/s tight fit
M5 Ultra 96 GB Q8_0 8K context · ~111 tok/s tight fit

Check your exact setup in the full tool →