Skip to main content
PromptQuorumBuilt for humans. Structured for AI.
Home/Local LLMs
Hardware & Performance

Best GPUs for Local LLMs in 2026: VRAM Tiers, Speed and Prices

·12 min readBy · Founder of PromptQuorum · Discovery engine for open-weight & open-source AI

Pick the GPU by VRAM tier first: 12 GB for small models, 16 GB for mid-size models, 24 GB or more for 30B-class models. In developer-reported llama.cpp results a 7B Q4_0 model generates 176.9 tok/s on an RTX 5070 Ti and 186.2 tok/s on an RTX 4090, so tier and price matter more than flagship status.

Key Takeaways

  • Choose VRAM first: 12 GB suits models up to about 14B, 16 GB suits mid-size models such as gpt-oss-20b, 24 GB suits dense ~32B models, and 48 GB (two 24 GB cards) suits 70B-class models (estimates, Q4-class quantization).
  • The speed gap is small for 7B models: developer-reported llama.cpp results give 176.9 tok/s (RTX 5070 Ti, 16 GB) versus 186.2 tok/s (RTX 4090, 24 GB); prompt processing differs more (6,952 versus 11,993 tok/s).
  • Prices are volatile: per the InsiderLLM tracker (source updated 2026-10-15), the RTX 5090 retails around $4,500–4,700 versus a $1,999 MSRP, the RTX 5080 $1,200–1,600 versus $999, and new RTX 4090 listings $2,000–3,000.
  • Lowest-priced 24 GB route in the tracked data: a used RTX 3090 at about $1,400–1,550 (eBay sold listings, same tracker) reaches 158.2 tok/s, about 85% of an RTX 4090 on the 7B test.
  • NVIDIA with CUDA is the default path. An AMD RX 7900 XTX matches RTX 4090 generation speed on the Vulkan 7B test (182.6 versus 188.0 tok/s, same scoreboard) but processes prompts about 2.5 times slower.

📍 In One Sentence

Choose a local-LLM GPU by VRAM tier first (12, 16, 24 or 32 GB), then by memory speed and current price, because a 16 GB RTX 5070 Ti generates 176.9 tok/s versus 186.2 tok/s for a 24 GB RTX 4090 in developer-reported llama.cpp 7B tests.

💬 In Plain Terms

Think of VRAM as the size of the table your model has to fit on. If the model does not fit, it slows to a crawl or will not load. Once it fits, a cheaper card is often almost as fast on small models, so match the card to the model size you want and then compare live prices, which are currently far above launch prices.

This page contains links to third-party products for reference. PromptQuorum is not enrolled in any affiliate program — these are plain links that earn no commission. Clicking links and your next steps are entirely your own responsibility. Any recommendation on this page reflects our own reading of the vendors' public pages, not a vendor endorsement or a test.

RTX 5080 16GB on Amazonproduct link · disclosedRTX 5090 32GB on Amazonproduct link · disclosedRTX 5070 Ti on Neweggproduct link · disclosed

Choose a GPU for local LLMs by VRAM first, then by memory speed and current price. In developer-reported llama.cpp results, a 7B Q4_0 model generates 176.9 tokens/s on a 16 GB RTX 5070 Ti and 186.2 tokens/s on a 24 GB RTX 4090, so the VRAM tier matters more than the flagship label. Tracked GPU prices are far above launch MSRP, so each card below is paired with its benchmark source and observed price range.

Best GPUs for Local LLMs in 2026: VRAM Tiers, Speed and Prices

GPU Tiers by VRAM, Speed and Observed Price

The table lists each GPU with its VRAM, its developer-reported llama.cpp generation speed on a 7B Q4_0 model, and the US price range seen in the sources named below. Speeds are not PromptQuorum measurements, and larger models or longer context run slower.

Speed is token generation (tg128) for Llama 2 7B Q4_0 with `llama-bench -ngl 99` and no flash attention, taken from the user-submitted llama.cpp CUDA scoreboard; the RX 7900 XTX figure comes from the llama.cpp Vulkan scoreboard. Prices are listed or observed US prices from the InsiderLLM tracker (source updated 2026-10-15) and Wccftech (2026-01-27). They are not offers or guarantees; check live listings.

GPU
VRAM
Speed (7B Q4_0)
Observed US price
RTX 306012 GB75.6 tok/sUsed only; not tracked here
RTX 507012 GB127.5 tok/sNot tracked here
RTX 5060 Ti16 GB90.9 tok/sFrom $679 (listing, 2026-01-27)
RTX 5070 Ti16 GB176.8 tok/s~$750–900, sporadic stock
RTX 508016 GB182.0 tok/s$1,200–1,600 (MSRP $999)
RTX 3090 (used)24 GB158.2 tok/sUsed ~$1,400–1,550
RTX 409024 GB186.2 tok/sNew $2,000–3,000 / used ~$2,575–3,010
RX 7900 XTX24 GB182.6 tok/sNot tracked here
RTX 509032 GB290.0 tok/s~$4,500–4,700 (MSRP $1,999)

12 GB Tier: Entry-Level Local LLMs

RTX 5070 (12 GB): 127.5 tok/s on the 7B test (developer-reported). NVIDIA lists a 250 W total graphics power for the RTX 5070. Fits dense models up to about 14B at Q4 (estimate).

RTX 3060 (12 GB), used: 75.6 tok/s on the same test. Slower, but the same 12 GB capacity; used prices are not tracked here.

8 GB cards: an RTX 4060 Ti (8 GB) reaches 63.9 tok/s, but 8 GB limits you to roughly 7–8B models at Q4 with short context (estimate). Prefer 12 GB or more.

16 GB Tier: Mid-Size Models

RTX 5060 Ti (16 GB): 90.9 tok/s on the 7B test, slower generation than the 12 GB RTX 5070 but 4 GB more capacity. Retail listings from $679 were reported in early 2026 (Wccftech, 2026-01-27); check current listings.

RTX 5070 Ti (16 GB): 176.9 tok/s. heise reported on 2026-01-15 that card makers expect Nvidia to discontinue it, so new stock may be sporadic; the InsiderLLM tracker shows roughly $750–900 with sporadic availability. NVIDIA lists a 300 W total graphics power.

RTX 5080 (16 GB): 182.0 tok/s, retail $1,200–1,600 versus a $999 MSRP per InsiderLLM. It has the same 16 GB as the RTX 5070 Ti and about 3% higher 7B generation speed in these results; the larger difference is prompt processing (8,297 versus 6,952 tok/s).

gpt-oss-20b fits in 16 GB: OpenAI states it runs within 16 GB of memory (21B total, 3.6B active parameters).

24 GB and Larger: 30B-Class and 70B-Class Models

RTX 4090 (24 GB): 186.2 tok/s and 11,993 tok/s prompt processing, the fastest 24 GB card in these results. New listings were $2,000–3,000 against a $1,599 MSRP, and used sold listings $2,575–3,010 (median $2,860), per the InsiderLLM tracker.

RTX 3090 (24 GB), used: 158.2 tok/s, about 85% of the RTX 4090, but prompt processing is less than half (5,175 versus 11,993 tok/s). Used sold listings were about $1,400–1,550 per the same tracker.

RTX 5090 (32 GB): 290.0 tok/s, the fastest card in the table. Retail was around $4,500–4,700 versus a $1,999 MSRP, and used sold listings $5,900–6,350. NVIDIA lists a 575 W total graphics power and 1,000 W required system power for the reference configuration.

48 GB and above: 70B-class models at Q4 need about 43 GB for weights (Ollama lists the llama3.3:70b download at 43 GB), so they need two 24 GB cards or one workstation card with 48 GB or more. See Multi-GPU Local LLMs.

Power safety: high-wattage GPUs draw large, sustained current. Follow the card maker's power-supply and cable instructions, seat power connectors fully, and ask a qualified electrician about circuit capacity. This guide is not electrical-safety advice.

Which Local LLM Fits Your GPU's VRAM

VRAM sets the largest model that fits. The table gives estimates for weights at Q4-class quantization (about 0.6 GB per billion parameters) plus room for context. Mixture-of-experts (MoE) models need memory for all parameters but generate at the speed of their active parameters. Use the VRAM Calculator for exact fits.

GPU (VRAM)
Model class that fits
Quant
Notes
12 GB (RTX 5070, RTX 3060)Dense models up to ~14BQ4_K_MAbout 8–9 GB of weights for 14B (estimate); the rest holds context.
16 GB (RTX 5060 Ti, 5070 Ti, 5080)gpt-oss-20b; dense up to ~20BMXFP4 / Q4_K_MOpenAI states gpt-oss-20b runs within 16 GB of memory.
24 GB (RTX 4090, RTX 3090, RX 7900 XTX)Dense ~32BQ4_K_MAbout 19–20 GB of weights (estimate); limited context headroom.
32 GB (RTX 5090)Dense ~32B at Q5–Q6Q5_K_M / Q6_KThe extra 8 GB buys higher-fidelity quantization or longer context (estimate).
48 GB (2× 24 GB)70B-class denseQ4_K_Mllama3.3:70b is a 43 GB download per Ollama; little room is left for context.

AMD and Intel GPUs for Local LLMs

AMD RX 7900 XTX (24 GB): on the Vulkan scoreboard it generates 182.6 tok/s versus 188.0 for an RTX 4090 and processes prompts at 3,727 versus 9,452 tok/s, about 2.5 times slower. Vulkan is one backend; ROCm results differ. Check the current llama.cpp and ROCm compatibility notes for your card before buying. See Best AMD GPUs for Local LLMs.

AMD RX 9070 XT (16 GB): 136.6 tok/s on the same Vulkan test.

Intel Arc: the Arc B580 reaches 70.1 tok/s and the Arc A770 52.6 tok/s on the Vulkan test, with prompt processing of 621 and 1,074 tok/s, well below the NVIDIA and AMD cards listed here.

Most local-LLM guides and community benchmarks target NVIDIA CUDA first, so troubleshooting help is easiest to find there. AMD is a reasonable option if you check backend support for your exact card and model first.

Speed Benchmarks: llama.cpp 7B Results by GPU

These are developer-reported community results for Llama 2 7B Q4_0 (tg128 for generation, pp512 for prompt processing), from the CUDA scoreboard and, for rows marked Vulkan, the Vulkan scoreboard. They are not PromptQuorum measurements; each submitter used their own build and system.

Generation speed largely follows memory bandwidth, so cards with very different prices land close together; prompt processing follows compute and separates them more clearly. Longer prompts, larger models and bigger context windows run slower than these figures.

GPU
VRAM
Generation (tok/s)
Prompt (tok/s)
RTX 509032 GB290.014,073
RTX 409024 GB186.211,993
RTX 508016 GB182.08,297
RTX 5070 Ti16 GB176.86,952
RTX 309024 GB158.25,175
RTX 408016 GB142.58,032
RTX 4070 Ti SUPER16 GB132.36,925
RTX 507012 GB127.55,185
RTX 5060 Ti16 GB90.93,737
RTX 306012 GB75.62,138
RTX 4060 Ti8 GB63.93,395
RX 7900 XTX (Vulkan)24 GB182.63,727
RX 9070 XT (Vulkan)16 GB136.65,202
Arc B580 (Vulkan)12 GB70.1621
Arc A770 (Vulkan)8/16 GB52.61,074

Common GPU Selection Mistakes

  • Paying flagship prices for a small-model workload. On the 7B test an RTX 5070 Ti (176.9 tok/s) is within about 5% of an RTX 4090 (186.2 tok/s). Buy the flagship for VRAM or prompt speed, not for 7B generation speed.
  • Assuming an older 24 GB card is far slower. The RTX 3090 reaches 158.2 tok/s versus 186.2 for the RTX 4090 on the 7B test; the larger gap is prompt processing (5,175 versus 11,993 tok/s), which matters for long prompts and RAG.
  • Buying 8 GB. An 8 GB card (RTX 4060 Ti, 63.9 tok/s) limits you to small models and short context; 12 GB is the practical floor for 14B-class models (estimate).
  • Paying inflated prices without checking alternatives. Observed RTX 5090 and RTX 5080 prices are well above MSRP. Compare a used 24 GB card, a 16 GB card, or cloud GPU rental before buying.
  • Skipping power and cooling planning. NVIDIA lists 575 W total graphics power for the RTX 5090. Check the card maker's PSU, cable and airflow guidance, and see Local LLM Power Consumption.

Frequently Asked Questions

How much VRAM do I need for local LLMs?

12 GB handles dense models up to about 14B at Q4 quantization, 16 GB fits mid-size models such as gpt-oss-20b (OpenAI states it runs within 16 GB), and 24 GB fits dense ~32B models at Q4 (estimates). 70B-class models need about 43 GB for weights at Q4, so two 24 GB GPUs or a 48 GB-class card. Use the VRAM calculator for exact fits.

What is the best GPU for local LLMs in 2026?

There is no single best GPU: choose by VRAM tier, then compare live prices. For 7B–14B models a 12–16 GB card is enough, and the RTX 5070 Ti (176.9 tok/s) is within about 5% of the RTX 4090 (186.2 tok/s) in developer-reported 7B tests. For 30B-class models you need 24 GB, and the RTX 5090 offers 32 GB. Prices are far above MSRP in the tracked data, so check current listings.

Is the RTX 4090 still worth buying for local LLMs?

It depends on the price you can get. The RTX 4090 offers 24 GB and 186.2 tok/s in the 7B test, but the InsiderLLM tracker lists new cards at $2,000–3,000 (MSRP $1,599) and used sold listings at $2,575–3,010. A used RTX 3090 (about $1,400–1,550, 158.2 tok/s) gives the same 24 GB for less, with slower prompt processing. The RTX 4090 makes sense mainly if you need that faster prompt processing.

Is a used RTX 3090 still a good choice for local LLMs?

It is the lowest-priced 24 GB NVIDIA option in the tracked data (about $1,400–1,550 in sold listings) and reaches 158.2 tok/s on the 7B test, about 85% of an RTX 4090. Its prompt processing is less than half (5,175 versus 11,993 tok/s), and used cards may have no warranty and carry wear risk; read Used GPUs for Local LLMs first.

Should I buy an AMD GPU for local LLMs?

It can work for token generation, with trade-offs. On the Vulkan scoreboard the RX 7900 XTX (24 GB) generates 182.6 tok/s versus 188.0 for an RTX 4090 but processes prompts about 2.5 times slower. Software setup differs by backend (ROCm, Vulkan), so check the current llama.cpp and Ollama notes for your exact card before buying.

What is the best GPU for running 70B models locally?

Two 24 GB GPUs (48 GB combined) are the common consumer route: llama3.3:70b is a 43 GB download per Ollama, which leaves little room for context. A single 24 GB or 32 GB card cannot hold it without much heavier quantization and quality loss. Dual-GPU speed depends on software and interconnect, so no figure is given here; see Multi-GPU Local LLMs.

Does more VRAM make a GPU faster for local LLMs?

No. VRAM sets which models fit; once a model fits, speed follows memory bandwidth and compute rather than capacity. In the 7B test the 24 GB RTX 3090 (158.2 tok/s) is slower than the 16 GB RTX 5080 (182.0 tok/s).

What GPU has 32 GB VRAM for local LLMs?

The RTX 5090 (32 GB GDDR7) is the GeForce card with the largest VRAM in these sources; it generates 290.0 tok/s on the 7B test. NVIDIA lists 575 W total graphics power and 1,000 W required system power for it, and retail prices around $4,500–4,700 were tracked against a $1,999 MSRP. The extra 8 GB over 24 GB mainly buys higher-fidelity quantization or longer context on ~32B models (estimate); 70B-class models still need about 43 GB.

Sources

A Note on Third-Party Facts

This article references third-party AI models, benchmarks, prices, and licenses. The AI landscape changes rapidly. Benchmark scores, license terms, model names, and API prices can shift between the time of writing and the time you read this. Before making deployment or compliance decisions based on this article, verify current figures on each provider’s official source: Hugging Face model cards for licenses and benchmarks, provider websites for API pricing, and EUR-Lex for current GDPR and EU AI Act text.

Run PromptQuorum with a local LLM, your own API keys, or both — you pick the backend.

Download the PromptQuorum Beta →

← Back to Local LLMs

MCP