GPU Tiers by VRAM, Speed and Observed Price
The table lists each GPU with its VRAM, its developer-reported llama.cpp generation speed on a 7B Q4_0 model, and the US price range seen in the sources named below. Speeds are not PromptQuorum measurements, and larger models or longer context run slower.
Speed is token generation (tg128) for Llama 2 7B Q4_0 with `llama-bench -ngl 99` and no flash attention, taken from the user-submitted llama.cpp CUDA scoreboard; the RX 7900 XTX figure comes from the llama.cpp Vulkan scoreboard. Prices are listed or observed US prices from the InsiderLLM tracker (source updated 2026-10-15) and Wccftech (2026-01-27). They are not offers or guarantees; check live listings.
GPU | VRAM | Speed (7B Q4_0) | Observed US price |
|---|---|---|---|
| RTX 3060 | 12 GB | 75.6 tok/s | Used only; not tracked here |
| RTX 5070 | 12 GB | 127.5 tok/s | Not tracked here |
| RTX 5060 Ti | 16 GB | 90.9 tok/s | From $679 (listing, 2026-01-27) |
| RTX 5070 Ti | 16 GB | 176.8 tok/s | ~$750–900, sporadic stock |
| RTX 5080 | 16 GB | 182.0 tok/s | $1,200–1,600 (MSRP $999) |
| RTX 3090 (used) | 24 GB | 158.2 tok/s | Used ~$1,400–1,550 |
| RTX 4090 | 24 GB | 186.2 tok/s | New $2,000–3,000 / used ~$2,575–3,010 |
| RX 7900 XTX | 24 GB | 182.6 tok/s | Not tracked here |
| RTX 5090 | 32 GB | 290.0 tok/s | ~$4,500–4,700 (MSRP $1,999) |
12 GB Tier: Entry-Level Local LLMs
RTX 5070 (12 GB): 127.5 tok/s on the 7B test (developer-reported). NVIDIA lists a 250 W total graphics power for the RTX 5070. Fits dense models up to about 14B at Q4 (estimate).
RTX 3060 (12 GB), used: 75.6 tok/s on the same test. Slower, but the same 12 GB capacity; used prices are not tracked here.
8 GB cards: an RTX 4060 Ti (8 GB) reaches 63.9 tok/s, but 8 GB limits you to roughly 7–8B models at Q4 with short context (estimate). Prefer 12 GB or more.
16 GB Tier: Mid-Size Models
RTX 5060 Ti (16 GB): 90.9 tok/s on the 7B test, slower generation than the 12 GB RTX 5070 but 4 GB more capacity. Retail listings from $679 were reported in early 2026 (Wccftech, 2026-01-27); check current listings.
RTX 5070 Ti (16 GB): 176.9 tok/s. heise reported on 2026-01-15 that card makers expect Nvidia to discontinue it, so new stock may be sporadic; the InsiderLLM tracker shows roughly $750–900 with sporadic availability. NVIDIA lists a 300 W total graphics power.
RTX 5080 (16 GB): 182.0 tok/s, retail $1,200–1,600 versus a $999 MSRP per InsiderLLM. It has the same 16 GB as the RTX 5070 Ti and about 3% higher 7B generation speed in these results; the larger difference is prompt processing (8,297 versus 6,952 tok/s).
gpt-oss-20b fits in 16 GB: OpenAI states it runs within 16 GB of memory (21B total, 3.6B active parameters).
24 GB and Larger: 30B-Class and 70B-Class Models
RTX 4090 (24 GB): 186.2 tok/s and 11,993 tok/s prompt processing, the fastest 24 GB card in these results. New listings were $2,000–3,000 against a $1,599 MSRP, and used sold listings $2,575–3,010 (median $2,860), per the InsiderLLM tracker.
RTX 3090 (24 GB), used: 158.2 tok/s, about 85% of the RTX 4090, but prompt processing is less than half (5,175 versus 11,993 tok/s). Used sold listings were about $1,400–1,550 per the same tracker.
RTX 5090 (32 GB): 290.0 tok/s, the fastest card in the table. Retail was around $4,500–4,700 versus a $1,999 MSRP, and used sold listings $5,900–6,350. NVIDIA lists a 575 W total graphics power and 1,000 W required system power for the reference configuration.
48 GB and above: 70B-class models at Q4 need about 43 GB for weights (Ollama lists the llama3.3:70b download at 43 GB), so they need two 24 GB cards or one workstation card with 48 GB or more. See Multi-GPU Local LLMs.
Power safety: high-wattage GPUs draw large, sustained current. Follow the card maker's power-supply and cable instructions, seat power connectors fully, and ask a qualified electrician about circuit capacity. This guide is not electrical-safety advice.
Which Local LLM Fits Your GPU's VRAM
VRAM sets the largest model that fits. The table gives estimates for weights at Q4-class quantization (about 0.6 GB per billion parameters) plus room for context. Mixture-of-experts (MoE) models need memory for all parameters but generate at the speed of their active parameters. Use the VRAM Calculator for exact fits.
GPU (VRAM) | Model class that fits | Quant | Notes |
|---|---|---|---|
| 12 GB (RTX 5070, RTX 3060) | Dense models up to ~14B | Q4_K_M | About 8–9 GB of weights for 14B (estimate); the rest holds context. |
| 16 GB (RTX 5060 Ti, 5070 Ti, 5080) | gpt-oss-20b; dense up to ~20B | MXFP4 / Q4_K_M | OpenAI states gpt-oss-20b runs within 16 GB of memory. |
| 24 GB (RTX 4090, RTX 3090, RX 7900 XTX) | Dense ~32B | Q4_K_M | About 19–20 GB of weights (estimate); limited context headroom. |
| 32 GB (RTX 5090) | Dense ~32B at Q5–Q6 | Q5_K_M / Q6_K | The extra 8 GB buys higher-fidelity quantization or longer context (estimate). |
| 48 GB (2× 24 GB) | 70B-class dense | Q4_K_M | llama3.3:70b is a 43 GB download per Ollama; little room is left for context. |
AMD and Intel GPUs for Local LLMs
AMD RX 7900 XTX (24 GB): on the Vulkan scoreboard it generates 182.6 tok/s versus 188.0 for an RTX 4090 and processes prompts at 3,727 versus 9,452 tok/s, about 2.5 times slower. Vulkan is one backend; ROCm results differ. Check the current llama.cpp and ROCm compatibility notes for your card before buying. See Best AMD GPUs for Local LLMs.
AMD RX 9070 XT (16 GB): 136.6 tok/s on the same Vulkan test.
Intel Arc: the Arc B580 reaches 70.1 tok/s and the Arc A770 52.6 tok/s on the Vulkan test, with prompt processing of 621 and 1,074 tok/s, well below the NVIDIA and AMD cards listed here.
Most local-LLM guides and community benchmarks target NVIDIA CUDA first, so troubleshooting help is easiest to find there. AMD is a reasonable option if you check backend support for your exact card and model first.
Speed Benchmarks: llama.cpp 7B Results by GPU
These are developer-reported community results for Llama 2 7B Q4_0 (tg128 for generation, pp512 for prompt processing), from the CUDA scoreboard and, for rows marked Vulkan, the Vulkan scoreboard. They are not PromptQuorum measurements; each submitter used their own build and system.
Generation speed largely follows memory bandwidth, so cards with very different prices land close together; prompt processing follows compute and separates them more clearly. Longer prompts, larger models and bigger context windows run slower than these figures.
GPU | VRAM | Generation (tok/s) | Prompt (tok/s) |
|---|---|---|---|
| RTX 5090 | 32 GB | 290.0 | 14,073 |
| RTX 4090 | 24 GB | 186.2 | 11,993 |
| RTX 5080 | 16 GB | 182.0 | 8,297 |
| RTX 5070 Ti | 16 GB | 176.8 | 6,952 |
| RTX 3090 | 24 GB | 158.2 | 5,175 |
| RTX 4080 | 16 GB | 142.5 | 8,032 |
| RTX 4070 Ti SUPER | 16 GB | 132.3 | 6,925 |
| RTX 5070 | 12 GB | 127.5 | 5,185 |
| RTX 5060 Ti | 16 GB | 90.9 | 3,737 |
| RTX 3060 | 12 GB | 75.6 | 2,138 |
| RTX 4060 Ti | 8 GB | 63.9 | 3,395 |
| RX 7900 XTX (Vulkan) | 24 GB | 182.6 | 3,727 |
| RX 9070 XT (Vulkan) | 16 GB | 136.6 | 5,202 |
| Arc B580 (Vulkan) | 12 GB | 70.1 | 621 |
| Arc A770 (Vulkan) | 8/16 GB | 52.6 | 1,074 |
Common GPU Selection Mistakes
- Paying flagship prices for a small-model workload. On the 7B test an RTX 5070 Ti (176.9 tok/s) is within about 5% of an RTX 4090 (186.2 tok/s). Buy the flagship for VRAM or prompt speed, not for 7B generation speed.
- Assuming an older 24 GB card is far slower. The RTX 3090 reaches 158.2 tok/s versus 186.2 for the RTX 4090 on the 7B test; the larger gap is prompt processing (5,175 versus 11,993 tok/s), which matters for long prompts and RAG.
- Buying 8 GB. An 8 GB card (RTX 4060 Ti, 63.9 tok/s) limits you to small models and short context; 12 GB is the practical floor for 14B-class models (estimate).
- Paying inflated prices without checking alternatives. Observed RTX 5090 and RTX 5080 prices are well above MSRP. Compare a used 24 GB card, a 16 GB card, or cloud GPU rental before buying.
- Skipping power and cooling planning. NVIDIA lists 575 W total graphics power for the RTX 5090. Check the card maker's PSU, cable and airflow guidance, and see Local LLM Power Consumption.
Frequently Asked Questions
How much VRAM do I need for local LLMs?
12 GB handles dense models up to about 14B at Q4 quantization, 16 GB fits mid-size models such as gpt-oss-20b (OpenAI states it runs within 16 GB), and 24 GB fits dense ~32B models at Q4 (estimates). 70B-class models need about 43 GB for weights at Q4, so two 24 GB GPUs or a 48 GB-class card. Use the VRAM calculator for exact fits.
What is the best GPU for local LLMs in 2026?
There is no single best GPU: choose by VRAM tier, then compare live prices. For 7B–14B models a 12–16 GB card is enough, and the RTX 5070 Ti (176.9 tok/s) is within about 5% of the RTX 4090 (186.2 tok/s) in developer-reported 7B tests. For 30B-class models you need 24 GB, and the RTX 5090 offers 32 GB. Prices are far above MSRP in the tracked data, so check current listings.
Is the RTX 4090 still worth buying for local LLMs?
It depends on the price you can get. The RTX 4090 offers 24 GB and 186.2 tok/s in the 7B test, but the InsiderLLM tracker lists new cards at $2,000–3,000 (MSRP $1,599) and used sold listings at $2,575–3,010. A used RTX 3090 (about $1,400–1,550, 158.2 tok/s) gives the same 24 GB for less, with slower prompt processing. The RTX 4090 makes sense mainly if you need that faster prompt processing.
Is a used RTX 3090 still a good choice for local LLMs?
It is the lowest-priced 24 GB NVIDIA option in the tracked data (about $1,400–1,550 in sold listings) and reaches 158.2 tok/s on the 7B test, about 85% of an RTX 4090. Its prompt processing is less than half (5,175 versus 11,993 tok/s), and used cards may have no warranty and carry wear risk; read Used GPUs for Local LLMs first.
Should I buy an AMD GPU for local LLMs?
It can work for token generation, with trade-offs. On the Vulkan scoreboard the RX 7900 XTX (24 GB) generates 182.6 tok/s versus 188.0 for an RTX 4090 but processes prompts about 2.5 times slower. Software setup differs by backend (ROCm, Vulkan), so check the current llama.cpp and Ollama notes for your exact card before buying.
What is the best GPU for running 70B models locally?
Two 24 GB GPUs (48 GB combined) are the common consumer route: llama3.3:70b is a 43 GB download per Ollama, which leaves little room for context. A single 24 GB or 32 GB card cannot hold it without much heavier quantization and quality loss. Dual-GPU speed depends on software and interconnect, so no figure is given here; see Multi-GPU Local LLMs.
Does more VRAM make a GPU faster for local LLMs?
No. VRAM sets which models fit; once a model fits, speed follows memory bandwidth and compute rather than capacity. In the 7B test the 24 GB RTX 3090 (158.2 tok/s) is slower than the 16 GB RTX 5080 (182.0 tok/s).
What GPU has 32 GB VRAM for local LLMs?
The RTX 5090 (32 GB GDDR7) is the GeForce card with the largest VRAM in these sources; it generates 290.0 tok/s on the 7B test. NVIDIA lists 575 W total graphics power and 1,000 W required system power for it, and retail prices around $4,500–4,700 were tracked against a $1,999 MSRP. The extra 8 GB over 24 GB mainly buys higher-fidelity quantization or longer context on ~32B models (estimate); 70B-class models still need about 43 GB.
Sources
- llama.cpp discussion #15013: Performance on NVIDIA CUDA -- user-submitted Llama 2 7B Q4_0 results.
- llama.cpp discussion #10879: Performance on Vulkan -- user-submitted results for AMD, Intel and NVIDIA.
- InsiderLLM: NVIDIA GPU Prices Are Rising -- MSRP, retail and eBay sold-listing prices (updated 2026-10-15).
- heise: Graphics card prices skyrocket, GeForce RTX 5070 Ti disappears -- availability report (2026-01-15).
- Wccftech: RTX 5060 Ti 16 GB retail prices -- retail listing check (2026-01-27).
- NVIDIA GeForce RTX 5090 and RTX 5070 family -- memory, power and system requirements.
- OpenAI gpt-oss -- memory requirements for gpt-oss-20b and gpt-oss-120b.
- Ollama llama3.3 -- download size of the 70b tag.
