Mistral Small 4 VRAM requirements: which GPUs can run it locally?
The smallest GPU memory that runs Mistral Small 4 well is 84 GB (Q4_K_M, e.g. RTX PRO 5500 Blackwell). 14 of 132 GPUs run it well and 1 more are a tight fit. It's a mixture-of-experts model, so all weights must fit in memory but speed depends only on the active experts.
MMLU 88.5
VRAM needed per quantization
| Quantization | Weights | 16K context | 32K context | Full 1M |
|---|---|---|---|---|
| Q4_K_M | 67.7 GB | 76.4 GB | 85.2 GB | 629.8 GB |
| Q8_0 | 117.8 GB | 126.6 GB | 135.4 GB | 680.0 GB |
Weights plus fp16 KV cache. Leave ~1 GB headroom for the inference engine.
Run it locally
Each command downloads the GGUF from Hugging Face, then Ollama starts a chat and llama.cpp starts an OpenAI-compatible server with a web UI on port 8080. Both fetch the vision projector (mmproj) automatically.
llama-server -hf bartowski/mistralai_Mistral-Small-4-119B-2603-GGUF:Q4_K_M Split into 2 files, which Ollama can't pull from Hugging Face yet; llama.cpp downloads the whole set.
llama-server -hf unsloth/Mistral-Small-4-119B-2603-GGUF:Q8_0 Split into 4 files, which Ollama can't pull from Hugging Face yet; llama.cpp downloads the whole set.
Cheapest GPUs that can run Mistral Small 4
Lowest launch MSRP among GPUs that run its smallest quantization (Q4_K_M) well, with at least 16K context.
As an Amazon Associate, I earn from qualifying purchases.
GPUs that can run Mistral Small 4
Best quantization that fits on each GPU, with the context and speed it gets. For chips with several memory sizes, the smallest that works is shown.
NVIDIA
- RTX PRO 5000 Blackwell 72GB 72 GB Q4_K_M 7K context · ~161 tok/s tight fit
- RTX PRO 5500 Blackwell 84 GB Q4_K_M 29K context · ~165 tok/s
- RTX PRO 6000 Blackwell 96 GB Q4_K_M 51K context · ~190 tok/s
- DGX Spark (GB 10) 128 GB Q8_0 18K context · ~32 tok/s
- RTX Spark 128 GB Q8_0 18K context · ~35 tok/s
AMD
- Ryzen AI Max (Strix Halo) 96 GB Q4_K_M 51K context · ~46 tok/s
- Ryzen AI Max 400 (Gorgon Halo) 128 GB Q8_0 18K context · ~32 tok/s
Apple
- M2 Max 96 GB Q4_K_M 51K context · ~67 tok/s
- M3 Max 96 GB Q4_K_M 51K context · ~52 tok/s
- M3 Ultra 96 GB Q4_K_M 51K context · ~89 tok/s
- M5 Ultra 96 GB Q4_K_M 51K context · ~119 tok/s
- M1 Ultra 128 GB Q8_0 18K context · ~60 tok/s
- M2 Ultra 128 GB Q8_0 18K context · ~60 tok/s
- M4 Max 128 GB Q8_0 18K context · ~59 tok/s
- M5 Max 128 GB Q8_0 18K context · ~65 tok/s