Which local LLMs can the GeForce RTX 4090 run?
Assumes 32 GB of DDR5-5600 system RAM, Windows with this GPU driving the display, 8k context and an f16 KV cache.
Specs
- VRAM
- 24 GB
- Memory bandwidth
- 1,008.0 GB/s
- FP16 compute
- 82.6 TFLOPS
- Launch year
- 2022
- Price
- $2,755 street (as of 2026-08-09)
Verdicts at a glance
At Q4_K_M (or the closest available quant) with 8k context.
- Runs great24 modelsRuns well0 modelsRuns slowly0 modelsWon't run5 models2 more models run at a lower quant.
How to read the verdicts
- Runs great
- Fully on the GPU at 20 tok/s or more
- Runs well
- Fully on the GPU at 8–20 tok/s; MoE experts in system RAM at 20 tok/s or more; or at least 90% on the GPU at 8 tok/s or more
- Runs slowly
- 2–8 tok/s; CPU-only; less than 90% on the GPU; or MoE experts in system RAM below 20 tok/s
- Won't run
- Does not fit, or under 2 tok/s
Every model on the GeForce RTX 4090
Dense models split between GPU and CPU slow down sharply — the CPU side sets the pace. MoE models that keep only their experts in system RAM degrade far more gently.
Scroll sideways to see every column.
| Model | Verdict | Speed | Quant | Memory | Runs as | Context | Notes |
|---|---|---|---|---|---|---|---|
| EXAONE 4.0 1.2B | Runs great | est. 537.3 tok/s472.8–601.7calibrated estimate ±12% | Q4_K_M | 2.5 / 24.0 GB | Full GPU | up to 64k | — |
| HyperCLOVA X SEED 1.5B | Runs great | est. 388.7 tok/s342.1–435.3calibrated estimate ±12% | Q4_K_M | 3.0 / 24.0 GB | Full GPU | up to 16k | — |
| gpt-oss-20b | Runs greatDetails | 225.2 tok/s8k estimate 198.3 tok/s (158.7–238.0, calibrated estimate ±20%)measured (1 run, 2k context) | MXFP4 | 13.5 / 24.0 GB | Full GPU | up to 128k |
|
| Qwen3.5 35B-A3B | Runs great | est. 220.6 tok/s176.5–264.7calibrated estimate ±20% | Q4_K_M | 24.0 / 24.0 GB | Full GPU | up to 8k | — |
| Qwen3 30B-A3B (2507) | Runs greatDetails | est. 161.2 tok/s129.0–193.5calibrated estimate ±20% | Q4_K_M | 20.5 / 24.0 GB | Full GPU | up to 32k | — |
| Gemma 4 26B-A4B | Runs great | est. 156.9 tok/s125.6–188.3calibrated estimate ±20% | Q4_K_M | 18.5 / 24.0 GB | Full GPU | up to 64k | — |
| Kanana 1.5 15.7B-A3B | Runs great | est. 147.8 tok/s118.3–177.4calibrated estimate ±20% | Q4_K_M | 12.7 / 24.0 GB | Full GPU | up to 32k | — |
| DeepSeek R1 Distill Llama 8B | Runs great | est. 117.7 tok/s103.6–131.8calibrated estimate ±12% | Q4_K_M | 7.2 / 24.0 GB | Full GPU | up to 64k |
|
| Kanana 1.5 8B | Runs great | est. 117.7 tok/s103.6–131.8calibrated estimate ±12% | Q4_K_M | 7.2 / 24.0 GB | Full GPU | up to 32k | — |
| Qwen3.5 9B | Runs great | est. 114.9 tok/s101.2–128.7calibrated estimate ±12% | Q4_K_M | 7.3 / 24.0 GB | Full GPU | up to 256k | — |
| Qwen3 8B | Runs great | est. 113.1 tok/s99.5–126.7calibrated estimate ±12% | Q4_K_M | 7.4 / 24.0 GB | Full GPU | up to 32k | — |
| Gemma 4 12B | Runs great | est. 92.2 tok/s81.1–103.2calibrated estimate ±12% | Q4_K_M | 8.8 / 24.0 GB | Full GPU | up to 128k | — |
| Llama 3.1 8B | Runs greatDetails | 91.0 tok/s8k estimate 117.7 tok/s (103.6–131.8, calibrated estimate ±12%)measured (1 run, 4k context) | Q4_K_M | 7.2 / 24.0 GB | Full GPU | up to 64k | — |
| Gemma 3 12B | Runs greatDetails | est. 90.0 tok/s79.2–100.8calibrated estimate ±12% | Q4_K_M | 9.0 / 24.0 GB | Full GPU | up to 128k | — |
| HyperCLOVA X SEED Think 14B | Runs great | est. 69.2 tok/s48.4–90.0theoretical estimate ±30% | Q4_K_M | 11.4 / 24.0 GB | Full GPU | up to 64k |
|
| Qwen3 14B | Runs greatDetails | est. 68.2 tok/s60.0–76.4calibrated estimate ±12% | Q4_K_M | 11.5 / 24.0 GB | Full GPU | up to 32k | — |
| Phi-4 | Runs great | est. 65.8 tok/s57.9–73.7calibrated estimate ±12% | Q4_K_M | 11.9 / 24.0 GB | Full GPU | up to 16k |
|
| Mistral Small 3.2 24B | Runs great | est. 45.0 tok/s39.6–50.4calibrated estimate ±12% | Q4_K_M | 16.8 / 24.0 GB | Full GPU | up to 32k | — |
| Gemma 3 27B | Runs great | est. 41.0 tok/s36.1–45.9calibrated estimate ±12% | Q4_K_M | 18.4 / 24.0 GB | Full GPU | up to 64k | — |
| Qwen3.5 27B | Runs great | est. 40.0 tok/s35.2–44.8calibrated estimate ±12% | Q4_K_M | 18.8 / 24.0 GB | Full GPU | up to 64k | — |
| EXAONE 4.0 32B | Runs great | est. 35.5 tok/s31.2–39.8calibrated estimate ±12% | Q4_K_M | 21.1 / 24.0 GB | Full GPU | up to 32k |
|
| EXAONE 4.5 33B | Runs great | est. 34.3 tok/s30.2–38.4calibrated estimate ±12% | Q4_K_M | 21.8 / 24.0 GB | Full GPU | up to 32k |
|
| Qwen3 32B | Runs great | est. 32.2 tok/s28.3–36.1calibrated estimate ±12% | Q4_K_M | 23.1 / 24.0 GB | Full GPU | up to 8k | — |
| DeepSeek R1 Distill Qwen 32B | Runs great | est. 32.1 tok/s28.2–35.9calibrated estimate ±12% | Q4_K_M | 23.2 / 24.0 GB | Full GPU | up to 8k |
|
| Qwen3.5 122B-A10B | Won't runTry UD-Q2_K_XL (heavy quality loss): Runs slowly | est. 20.6 tok/s14.4–26.7theoretical estimate ±30% | UD-Q2_K_XL | 24.0 / 24.0 GB + 20.2 GB RAM | Partial offload | up to 128k |
|
| Llama 3.3 70B | Won't runTry IQ4_XS: Runs slowly | est. 2.2 tok/s1.8–2.6calibrated estimate ±20% | IQ4_XS | 24.0 / 24.0 GB + 17.8 GB RAM | Partial offload | up to 8k | — |
| gpt-oss-120b | Won't runDetails | — | MXFP4 | needs 64.3 GB | — | — |
|
| Solar Open 100B | Won't run | — | Q4_K_M | needs 64.4 GB | — | — |
|
| Solar Open 2 250B | Won't run | — | IQ4_XS | needs 137.2 GB | — | — |
|
Reasoning models spend extra tokens thinking, so their speed thresholds are 1.5× stricter (30 / 12 / 3 tok/s).
Measured results on the GeForce RTX 4090
Public benchmarks we calibrate against. Their conditions (context, backend, flags) can differ from the estimates above.
| Model | Quant | Backend | Context | Prompt (tok/s) | Generation (tok/s) | Flags | Source | Measured |
|---|---|---|---|---|---|---|---|---|
| llama-2-7b | Q4_0 | llama.cpp | 512 | 11,993.0 | 186.2 | — | github.com | 2025-08-01 |
| gpt-oss-20b | MXFP4 | llama.cpp | 2k | 8,078.0 | 225.2 | — | github.com | 2025-08-15 |
| Qwen3 8B | Q4_K_XL | llama.cpp | 16k | — | 104.3 | — | hardware-corner.net | 2026-08-09 |
| Qwen3 30B-A3B (2507) | Q4_K_XL | llama.cpp | 16k | — | 139.7 | — | hardware-corner.net | 2026-08-09 |
| Llama 3.1 8B | Q4_K_M | llama.cpp | 4k | 6,697.0 | 91.0 | ctx-weighted | localscore.ai | 2026-08-09 |
Frequently asked questions
What is the largest model that runs entirely on the GeForce RTX 4090?
Qwen3.5 35B-A3B at Q4_K_M (a 22.63 GB file) fits entirely in 24 GB with 8k context, at est. 220.6 tok/s (176.5–264.7, calibrated estimate ±20%).
How many local LLMs run well on the GeForce RTX 4090?
At Q4_K_M with 8k context, out of 29 tracked models: 24 run great, 0 run well, 0 run slowly and 5 won't run.
Can the GeForce RTX 4090 run a 70B model like Llama 3.3 70B?
Runs slowly — Partial offload, IQ4_XS: est. 2.2 tok/s (1.8–2.6, calibrated estimate ±20%).
Speeds are estimates from memory bandwidth, calibrated against public benchmarks, and each one comes with an error band and a confidence label. Real results vary with drivers, backend, context length and thermals.