Wichtigste Erkenntnisse
- AMD RX 6800 XT (16GB, $300–350 used) and RX 7900 XTX (24GB, $400–500 used) are the only viable options for local LLMs.
- Performance-per-dollar: AMD is 20–30% cheaper than NVIDIA, but software friction costs 5–10 hours of setup time.
- Ollama: Limited AMD support (ROCm path is buggy, CPU fallback is slow). Not recommended.
- vLLM: Full AMD ROCm support as of v0.6.0, but setup requires manual drivers. Works well if you get past setup.
- Text Generation WebUI: Excellent AMD support via ROCm. Best user experience on AMD.
- Llama.cpp: Native AMD support (HIP backend). Solid performance. Recommended AMD path.
- Setup cost: Plan 5–10 hours debugging ROCm drivers, HIPCC compilation, kernel compatibility.
- Verdict (April 2026): Use AMD only if you have AMD already, or if you find a killer used deal ($300 for 16GB card). Otherwise, NVIDIA CUDA is still simpler.
Which AMD GPUs Are Actually Worth Using?
- RX 6800 XT (16GB GDDR6): The value king for AMD. 2020 release. Still solid for 7B–22B inference. Used: $300–350.
- RX 6900 XT (16GB GDDR6): Marginally faster 6800 XT. Rare. Used: $350–400. Not worth the price bump.
- RX 7900 XT (20GB GDDR6): Newer RDNA 3 arch. 20% faster than 6800 XT. Used: $400–480. Good for 70B Q4.
- RX 7900 XTX (24GB GDDR6): Top AMD consumer GPU. 24GB VRAM is game-changer for 70B. Used: $450–550. Comparable to RTX 4090 speed.
- Radeon Pro W6800 (32GB): Enterprise card, cheaper used (~$200–300). Slower, but 32GB is excellent for 70B Q8. Niche play.
How Do AMD GPUs Compare to RTX on Price and Speed?
| GPU | VRAM | TFLOPS | Price Used | Perf/$ vs. RTX | Equivalent RTX |
|---|---|---|---|---|---|
| — | — | — | — | — | — |
| — | — | — | — | — | — |
| — | — | — | — | — | — |
| — | — | — | — | — | — |
| — | — | — | — | — | — |
What's the ROCm Setup Friction for AMD?
1. Install AMD ROCm drivers: `apt-get install rocm-dkms` (Ubuntu). On Windows, manual .exe installer. Takes 30 min.
2. Verify HIP compiler: `hipcc --version`. Often fails on first try. Debug kernel compatibility for your GPU.
3. Install HIPCC (AMD's HIP-to-C++ compiler): `apt-get install hip-runtime-amd`. Another dependency chain.
4. Test with small LLM: Run inference to verify GPU acceleration is working. Often defaults to CPU fallback.
5. Troubleshoot driver version mismatches: ROCm v5.7 works with kernel 5.15 but not 6.x. Consume 2–4 hours debugging.
NVIDIA CUDA by comparison: `nvidia-cuda-toolkit` → one apt-get, instant GPU access. AMD requires 5–10× more friction.
Can You Run Ollama and vLLM on AMD?
Ollama on AMD: Experimental/buggy as of April 2026. ROCm path works sometimes, CPU fallback is slow. Not recommended.
vLLM on AMD: Full ROCm support since v0.6.0. Works well, but requires manual ROCm/HIP driver setup. Good if you're past the setup gauntlet.
Text Generation WebUI: Excellent AMD ROCm support. Best user experience on AMD. Recommended.
Llama.cpp: Native HIP backend. Solid performance. Easiest AMD path. Recommended.
LM Studio: NVIDIA only. No AMD support.
As of April 2026: vLLM + llama.cpp are your AMD paths. Ollama is not reliable.
When Should You Actually Buy AMD Over NVIDIA?
Buy AMD if:
- You find a used RX 7900 XTX for <$450 (under-priced vs. RTX 4090 value).
- You already own AMD hardware and want ecosystem consistency.
- You're building a cluster and value compute-per-dollar over ease-of-setup.
Don't buy AMD if:
- You want a plug-and-play experience. NVIDIA CUDA is faster to get working.
- You need Ollama. AMD support is unreliable.
- You're time-constrained. ROCm debugging can eat 10+ hours.
Common AMD Adoption Mistakes
- Buying RX 6700 (12GB) thinking it's a 3060 12GB equivalent—it's 20% slower and often harder to find used.
- Assuming ROCm "just works" like CUDA—plan 5–10 hours of troubleshooting driver and kernel compatibility.
- Using Ollama with AMD expecting seamless integration—ROCm path is buggy; llama.cpp or vLLM are better bets.
FAQ
Should I buy AMD RX 6800 XT or NVIDIA RTX 3080 for local LLMs?
RTX 3080 if you value simplicity (CUDA "just works"). RX 6800 XT if you want 25% better value and don't mind 5–10 hours ROCm setup.
Is AMD RX 7900 XTX better than RTX 4090?
Similar speed, same VRAM (24GB). RX 7900 XTX is $200–300 cheaper used ($450–550 vs. $1,000–1,300). ROCm setup is the trade-off.
Can I use AMD GPUs with Ollama?
Technically yes, but expect buggy behavior. CPU fallback is common. Use vLLM or llama.cpp instead for AMD.
What's the best AMD path for local LLMs in 2026?
Llama.cpp (HIP backend) + Text Generation WebUI. Both have solid AMD support. Avoid Ollama.
Do I need Ubuntu for AMD ROCm, or does Windows work?
Windows support exists (HIP on Windows), but it's newer and buggier. Ubuntu is the recommended path.
Is RX 6700 or 6750 good for 7B models?
RX 6700 (12GB) works but is 20% slower than RX 6800 XT. Only buy if <$250. Otherwise, stretch to 6800 XT.
Can I mix AMD and NVIDIA GPUs in one system?
Theoretically yes, but management is a nightmare. Each GPU needs its own CUDA/HIP runtime. Not recommended.
Sources
- AMD ROCm documentation and GitHub: HIP compiler, driver compatibility matrix, LLM inference examples
- vLLM GitHub: AMD/ROCm backend implementation and support status (v0.6.0+)
- Llama.cpp GitHub: HIP backend for AMD GPU support