What Models?

Pick the right model for your GPU — in seconds.

Mistral Small 4 VRAM requirements: which GPUs can run it locally?

119B params 1M max context VisionTool use

The smallest GPU memory that runs Mistral Small 4 well is 84 GB (Q4_K_M, e.g. RTX PRO 5500 Blackwell). 14 of 132 GPUs run it well and 1 more are a tight fit. It's a mixture-of-experts model, so all weights must fit in memory but speed depends only on the active experts.

MMLU 88.5

VRAM needed per quantization

QuantizationWeights16K context32K contextFull 1M
Q4_K_M67.7 GB76.4 GB85.2 GB629.8 GB
Q8_0117.8 GB126.6 GB135.4 GB680.0 GB

Weights plus fp16 KV cache. Leave ~1 GB headroom for the inference engine.

Run it locally

Each command downloads the GGUF from Hugging Face, then Ollama starts a chat and llama.cpp starts an OpenAI-compatible server with a web UI on port 8080. Both fetch the vision projector (mmproj) automatically.

llama.cpp llama-server -hf bartowski/mistralai_Mistral-Small-4-119B-2603-GGUF:Q4_K_M

Split into 2 files, which Ollama can't pull from Hugging Face yet; llama.cpp downloads the whole set.

llama.cpp llama-server -hf unsloth/Mistral-Small-4-119B-2603-GGUF:Q8_0

Split into 4 files, which Ollama can't pull from Hugging Face yet; llama.cpp downloads the whole set.

Cheapest GPUs that can run Mistral Small 4

Lowest launch MSRP among GPUs that run its smallest quantization (Q4_K_M) well, with at least 16K context.

As an Amazon Associate, I earn from qualifying purchases.

GPUs that can run Mistral Small 4

Best quantization that fits on each GPU, with the context and speed it gets. For chips with several memory sizes, the smallest that works is shown.

NVIDIA

AMD

Apple

  • M2 Max 96 GB Q4_K_M 51K context · ~67 tok/s
  • M3 Max 96 GB Q4_K_M 51K context · ~52 tok/s
  • M3 Ultra 96 GB Q4_K_M 51K context · ~89 tok/s
  • M5 Ultra 96 GB Q4_K_M 51K context · ~119 tok/s
  • M1 Ultra 128 GB Q8_0 18K context · ~60 tok/s
  • M2 Ultra 128 GB Q8_0 18K context · ~60 tok/s
  • M4 Max 128 GB Q8_0 18K context · ~59 tok/s
  • M5 Max 128 GB Q8_0 18K context · ~65 tok/s
Check your exact setup in the full tool →