Skip to main content
PromptQuorumBuilt for humans. Structured for AI.
Home/Local LLMs
Best Models

Run a 70B LLM Locally 2026: RAM, VRAM & Speed by Hardware

·11 min readBy · Founder of PromptQuorum · Discovery engine for open-weight & open-source AI

A 70B model needs about 43 GB of memory at Q4_K_M (Llama 3.3 70B), plus 3-10 GB for context, and its speed is set by memory bandwidth, not capacity. A Mac with 64 GB or more, two 24 GB GPUs, or a 128 GB unified-memory box holds it entirely in fast memory; a single 24 GB GPU or a CPU-only PC runs it at about 1-4 tokens/sec.

Key Takeaways

  • Memory at Q4_K_M: Llama 3.3 70B is 43 GB, Q5_K_M 50 GB, Q8_0 75 GB, FP16 141 GB. Context adds about 2.7 GB per 8K tokens.
  • Speed rule: tokens/sec is capped at memory bandwidth divided by model size, and real speed is typically 60-80% of that cap. 43 GB on 614 GB/s gives a cap of about 14 tok/sec.
  • Best consumer speed: a Mac Studio M5 Max configured with 64 GB or more (about 9-11 tok/sec) or two 24 GB GPUs (about 12-17 tok/sec); the Mac Studio M5 Ultra reaches an estimated 17-21 tok/sec.
  • Mac mini: the $1,699 M5 Pro model has 24 GB; 64 GB is a build-to-order upgrade that gives about 4-5 tok/sec. The M6 Mac mini (32 GB max) cannot run 70B.
  • DGX Spark: 128 GB but only 273 GB/s; LMSYS measured 2.7 tok/sec for 70B at FP8. Buy it for CUDA and large MoE models, not for fast dense-70B chat.
  • Single 24 GB GPU or CPU only: about 2-4 tok/sec with an RTX 4090 and fast DDR5, 1-2 tok/sec on a desktop CPU. 256 GB of RAM adds capacity (Q8_0, FP16, long context), not speed.
  • There is no Qwen3 72B: the 72B model is Qwen2.5 72B. Newer large open models are mostly mixture-of-experts.

📍 In One Sentence

A 70B model needs about 43 GB of memory at Q4_K_M plus 3-10 GB for context, and generates roughly bandwidth divided by model size tokens per second, so a 64 GB+ Mac, two 24 GB GPUs or a 128 GB unified-memory box run it in fast memory while a single 24 GB GPU manages 2-4 tokens/sec.

💬 In Plain Terms

A 70B model is a very large AI model with about 70 billion adjustable numbers. Loading it takes roughly 43 GB of memory, and how fast it answers depends on how quickly that memory can be read. A Mac with lots of unified memory or two big graphics cards is the comfortable route; a normal PC with one graphics card works but answers slowly.

Running a 70B model locally takes about 43 GB of fast memory at Q4_K_M, plus 3-10 GB for context. A Mac with 64 GB or more of unified memory, two 24 GB GPUs, or a 128 GB unified-memory box can hold it entirely in fast memory. Speed depends on memory bandwidth: from 1-2 tokens/sec on a desktop CPU to roughly 10 tokens/sec on a Mac Studio M5 Max. Llama 3.3 70B and Qwen2.5 72B are the two main dense 70B-class models.

Run a 70B LLM Locally 2026: RAM, VRAM & Speed by Hardware

What Is a 70B Model?

A 70B model is a large language model with about 70 billion parameters -- the learned weights that make up the network. "B" stands for billion. Llama 3.3 70B has 70.6 billion parameters and Qwen2.5 72B has 72.7 billion.

Memory is parameters multiplied by bytes per weight: 16-bit weights take 2 bytes each (about 141 GB), 8-bit (Q8_0) about 1 byte (75 GB), and 4-bit Q4_K_M about 0.6 bytes (43 GB). That is why 70B is the size where consumer hardware first needs 40+ GB of fast memory, while 7B-14B models fit on an ordinary laptop.

What Hardware Do You Need to Run a 70B LLM Locally?

You need about 43 GB of fast memory for the model plus 3-10 GB for context, held in GPU VRAM, Apple unified memory or system RAM -- and the faster that memory, the faster the model generates.

Speeds below are single-user chat estimates at short context, derived from memory bandwidth (see "Why Memory Bandwidth Sets Speed"). They are ceilings reduced to a typical 60-80% efficiency, not benchmarks run by PromptQuorum; the DGX Spark figure is the exception and is measured by third parties.

Hardware
Fits 70B Q4?
Memory bandwidth
Est. speed (Q4_K_M)
Notes
Mac Studio M5 Max (64-128 GB)Yes -- full GPUup to 614 GB/s~9-11 tok/sBase config is 36 GB; 64 GB+ is an upgrade
Mac Studio M5 Ultra (96 GB+)Yes -- Q8_0 fitsup to 1.2 TB/s~17-21 tok/sFrom $5,499 with 96 GB
Mac mini M5 Pro (64 GB)Yes -- tight307 GB/s~4-5 tok/s$1,699 base is 24 GB; 64 GB is build-to-order
Mac mini M6 (32 GB max)No170 GB/sN/A32 GB max is below the ~43 GB Q4 file
Mac Studio M2 Ultra (64 GB+, used)Yes -- full GPU800 GB/s~12-15 tok/sUsed market only; check the memory size
NVIDIA DGX Spark (128 GB)Yes -- Q8_0 fits273 GB/s~4-5 tok/s2.7 tok/s measured at FP8; reported $6,950
AMD Ryzen AI Max+ 395 (128 GB)Yes256 GB/s~4-5 tok/sMini PCs and laptops; GPU memory share is configurable
2x RTX 3090 / 4090 (48 GB)Yes -- tight936-1,008 GB/s per card~12-17 tok/sQ4_K_M plus short context only
RTX 5090 (32 GB) + 64 GB RAMPartial offload1,792 GB/s VRAM~4-6 tok/s~70% of layers on GPU, rest in system RAM
RTX 4090 (24 GB) + 64 GB RAMPartial offload1,008 GB/s VRAM~2-4 tok/s~55% of layers on GPU, rest in system RAM
Olares One (RTX 5090 Mobile 24 GB + 96 GB RAM)Partial offloadVRAM not published / RAM ~90 GB/s~2-4 tok/s24 GB GDDR7 VRAM; $4,999 hardware cost
CPU only, dual-channel DDR5Yes -- CPU only~90 GB/s~1-2 tok/sBatch jobs only
CPU only, 8-12 channel server RAMYes -- CPU only350-460 GB/s~5-8 tok/sThreadripper PRO / EPYC; 256 GB allows Q8_0

🔍Insight: Olares One (olares.com/one) is a compact desktop AI computer with a laptop-class RTX 5090 (24 GB GDDR7 VRAM), an Intel Core Ultra 9 275HX and 96 GB of DDR5-5600 RAM (upgradable to 128 GB, according to Olares). For 70B models it behaves like the RTX 4090 row above: about 55% of a Q4_K_M model fits in VRAM and the rest is read from system RAM, so expect roughly 2-4 tokens/sec (an estimate, not a benchmark). The 96 GB of RAM adds capacity, not speed: Q8_0 (75 GB) fits with room left for context. Olares lists a $4,999 hardware cost and publishes no 70B tokens/sec figures.

How Much RAM or VRAM Does a 70B Model Need at Each Quantization Level?

Sizes below are the Llama 3.3 70B files in the Ollama library; other 70B models differ by a few percent. These are weights only -- add the context memory from the next section.

Quantization
Memory (weights)
Quality
Practical?
FP16 (full precision)~141 GBReference quality192 GB+ Mac Ultra or 256 GB RAM server
Q8_0~75 GBNear-lossless96-128 GB unified memory or 128 GB RAM
Q6_K~58 GBVery close to Q8_072-96 GB machines
Q5_K_M~50 GBMinimal loss64 GB Mac after raising the GPU limit -- tight
Q4_K_M~43 GBLow loss -- defaultYes -- 64 GB Mac or 48 GB dual GPU
Q3_K_S~31 GBModerate lossYes -- 40-48 GB machines
Q2_K~26 GBHigh lossNot recommended

How Much Extra Memory Does Context Add to a 70B Model?

Context adds memory on top of the weights: Llama 3.3 70B stores about 0.33 MB of KV cache per token at FP16, which is roughly 2.7 GB at 8K tokens, 10.7 GB at 32K and 43 GB at 128K. The figure comes from the model architecture: 80 layers x 8 key-value heads x 128 dimensions x 2 (keys and values) x 2 bytes = 327,680 bytes per token.

Each concurrent session keeps its own KV cache while the weights are shared. A multi-agent system with four agents at 32K tokens each needs about 43 GB of weights plus 43 GB of cache. Enabling flash attention and an 8-bit KV cache in Ollama (OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0) roughly halves the cache.

Context length
KV cache (FP16)
Total with Q4_K_M
8K tokens~2.7 GB~46 GB
32K tokens~10.7 GB~54 GB
128K tokens~43 GB~86 GB

Why Does Memory Bandwidth Set the Speed of a 70B Model?

Every generated token reads all the model weights once, so decode speed is capped at memory bandwidth divided by model size. A 43 GB Q4_K_M model on 614 GB/s of bandwidth is capped at about 14 tokens/sec; real speed is typically 60-80% of the cap.

The same rule explains why large memory is not the same as fast: the 128 GB DGX Spark has 273 GB/s, a cap of about 6 tokens/sec at Q4_K_M, and LMSYS measured 2.7 tokens/sec for Llama 3.1 70B at FP8 (about 70 GB, cap 3.9). Capacity decides whether a model runs; bandwidth decides how fast. Prompt processing (prefill) is compute-bound and behaves differently.

Memory
Bandwidth
Cap at 43 GB
Dual-channel DDR5-5600~90 GB/s~2 tok/s
AMD Ryzen AI Max+ 395256 GB/s~6 tok/s
NVIDIA DGX Spark273 GB/s~6 tok/s
Apple M5 Pro307 GB/s~7 tok/s
Apple M5 Maxup to 614 GB/s~14 tok/s
Apple M2 Ultra800 GB/s~19 tok/s
Apple M5 Ultraup to 1.2 TB/s~28 tok/s

Is Apple Silicon a Good Choice for 70B Models?

Apple Silicon is the simplest way to hold a 70B model entirely in GPU-accessible memory, because the CPU and GPU share one unified memory pool. A Mac with 64 GB or more runs Q4_K_M with no layer offloading, at a speed set by the chip bandwidth: roughly 9-11 tokens/sec on an M5 Max and 12-15 on a used M2 Ultra (estimates). We found no published 70B benchmarks for the M5 chips yet.

Apple announced the Mac Studio with M5 Max and M5 Ultra on August 25, 2026; both shipped September 22, 2026. The M5 Max starts at $2,499 with 36 GB and goes up to 128 GB at up to 614 GB/s -- configure 64 GB or more for 70B. The M5 Ultra starts at $5,499 with 96 GB, goes up to 512 GB at up to 1.2 TB/s, and runs Q8_0 comfortably; the 512 GB configuration was listed as arriving in late October.

The Mac mini M5 Pro starts at $1,699 with 24 GB, not 64 GB. The 64 GB build-to-order configuration (307 GB/s, Thunderbolt 5) holds a Q4_K_M 70B model but generates only about 4-5 tokens/sec, and reviews put it at roughly $2,700-$3,200 depending on CPU and storage. The base Mac mini M6 ($899) maxes out at 32 GB and 170 GB/s and cannot run a 70B model at any quantization.

macOS limits GPU-addressable memory to roughly 75% of RAM by default (about 48 GB on a 64 GB Mac). To run Q5_K_M (50 GB) raise the limit, for example sudo sysctl iogpu.wired_limit_mb=57344 on a 64 GB Mac; the setting resets on reboot and leaves the OS only about 8 GB.

Is the NVIDIA DGX Spark (or a 128 GB Strix Halo PC) Good for 70B Models?

The NVIDIA DGX Spark is a compact desktop AI computer with 128 GB of unified LPDDR5x memory at 273 GB/s, built on the GB10 Grace Blackwell Superchip with CUDA support. It loads a 70B model even at Q8_0 (75 GB), but bandwidth limits generation: LMSYS measured 2.7 tokens/sec decode for Llama 3.1 70B at FP8, and community llama.cpp runs at Q4_K_M are reported in the 4-5 tokens/sec range.

Pricing has moved sharply. NVIDIA raised the Founders Edition from $3,999 to $4,699 in February 2026 because of memory supply constraints, and a price of $6,950 was reported in October 2026, together with a 64 GB version at $4,999 sold by partners from October 23, 2026. Check live prices before buying.

The AMD Ryzen AI Max+ 395 ("Strix Halo") mini PCs and laptops offer 128 GB of LPDDR5X at 256 GB/s, the same class of bandwidth, and a similar estimate of about 4-5 tokens/sec for a dense 70B model. Both platforms are better matched to mixture-of-experts models (see below) and CUDA or fine-tuning work than to dense-70B chat.

Spec
Value
Memory128 GB unified LPDDR5x
Memory bandwidth273 GB/s
70B at Q8_0 (75 GB)Fits
70B decode, measured2.7 tok/s at FP8 (LMSYS, SGLang)
Max model size~200B parameters at FP4
Price$6,950 reported; $4,699 Feb 2026 list
Ollama commandollama run llama3.3:70b

How Does NVIDIA GPU + Layer Offloading Work for 70B Models?

Ollama and llama.cpp split a model between GPU VRAM and system RAM: layers in VRAM run at GPU speed, layers in RAM run at CPU memory speed, and the slow part dominates. With an RTX 4090 (24 GB) about 55% of a Q4_K_M 70B model fits on the GPU and the rest streams from DDR5 at roughly 90 GB/s, which limits generation to about 2-4 tokens/sec. An RTX 5090 (32 GB) holds about 70% and reaches an estimated 4-6 tokens/sec.

Ollama offloads automatically; no flag is needed. Run ollama ps to see the split (the PROCESSOR column, for example 45%/55% CPU/GPU). In llama.cpp, set the layer count with -ngl.

bash
# Ollama offloads as many layers as fit in VRAM automatically
ollama run llama3.3:70b

# Show the CPU/GPU split of the loaded model:
ollama ps

# llama.cpp: choose the number of GPU layers explicitly
./llama-cli -m llama-3.3-70b-instruct-q4_k_m.gguf \
  -ngl 44 \
  --ctx-size 4096

Which Inference Runtime Is Best for a 70B Model on Multiple GPUs?

For one user, Ollama or llama.cpp with layer splitting is the simplest multi-GPU path; for several concurrent users or agents, vLLM or SGLang with tensor parallelism delivers far more total throughput. Two 24 GB cards (RTX 3090 or 4090) give 48 GB, enough for Q4_K_M plus a short context.

Layer splitting passes activations between cards once per token and works over plain PCIe, so it suits mixed or consumer cards. Tensor parallelism splits each layer across GPUs and benefits from fast links: the RTX 3090 supports an NVLink bridge, while the RTX 4090 and 5090 do not.

Runtime
Multi-GPU method
Best for
OllamaAutomatic layer splitOne user, easiest setup
llama.cpp--split-mode, --tensor-splitMixed GPUs, GGUF quants
vLLM--tensor-parallel-size NMany users or agents; AWQ/GPTQ/FP8
SGLangTensor parallelismAgents with shared prompt prefixes
TensorRT-LLMTensor / pipeline parallelMax NVIDIA throughput; engine build

Is CPU-Only 70B Inference Practical, and Does 256 GB of RAM Help?

CPU-only 70B inference is limited by memory bandwidth: dual-channel desktop DDR5 (about 90 GB/s) gives roughly 1-2 tokens/sec at Q4_K_M, and adding RAM capacity does not make it faster. At 1.5 tokens/sec a 200-word reply (about 270 tokens) takes around three minutes.

256 GB of RAM changes what fits, not how fast it runs: it holds Q8_0 (75 GB) or FP16 (141 GB) with long context. Speed depends on memory channels -- an 8-channel Threadripper PRO or 12-channel EPYC server has three to five times the bandwidth of a desktop. Desktop boards with four large DIMMs often run their memory slower than with two. Prompt processing on CPU is also slow for long inputs.

The estimates below apply the bandwidth rule at 70% efficiency; they are not measured benchmarks.

CPU / RAM setup
Bandwidth
Q4_K_M (43 GB)
Q8_0 (75 GB)
FP16 (141 GB)
Desktop, 2-ch DDR5-5600~90 GB/s~1-1.5 tok/s~0.6-0.9 tok/s~0.3-0.5 tok/s
Threadripper PRO, 8-ch DDR5-5600~358 GB/s~5-6 tok/s~3-3.5 tok/s~1.5-1.8 tok/s
EPYC, 12-ch DDR5-4800~461 GB/s~7-8 tok/s~4-4.5 tok/s~2-2.5 tok/s

Do 70B-Class MoE Models Change the Hardware Math?

Mixture-of-experts (MoE) models need memory for all their parameters but read only the active experts for each token, so they generate much faster than a dense 70B model with the same memory footprint. OpenAI gpt-oss-120b has about 117 billion parameters with about 5.1 billion active; Meta Llama 4 Scout has 109 billion with 17 billion active.

The bandwidth rule applies to the active weights, so MoE models make the best use of high-capacity, moderate-bandwidth machines such as the DGX Spark, Ryzen AI Max+ 395 PCs and a 64 GB Mac mini M5 Pro. For a dense 70B model on that hardware, expect 4-5 tokens/sec; check each MoE model file size against your memory before downloading.

Which 70B Model Should You Run Locally?

Llama 3.3 70B is the default dense 70B model: Meta reports 86.0 on MMLU (CoT) and 88.4 on HumanEval, with a 128K context window. Qwen2.5 72B is the main alternative for multilingual and coding use. There is no Qwen3 72B -- Qwen3 dense models stop at 32B -- so a "Qwen3 72B" reference means Qwen2.5 72B.

Commands: ollama run llama3.3:70b, ollama run qwen2.5:72b, ollama run deepseek-r1:70b.

Model
Parameters
License
Best for
Llama 3.3 70B70.6B denseLlama 3.3 Community LicenseGeneral English, instruction following
Qwen2.5 72B Instruct72.7B denseQwen LicenseMultilingual, coding; 131K with YaRN
DeepSeek-R1-Distill-Llama-70B70B denseSee model cardReasoning; long outputs slow replies
Mistral Large 2123B denseMistral Research LicenseResearch only; ~75 GB at Q4

What Hardware Do You Need to Fine-Tune a 70B Model?

QLoRA fine-tuning of a 70B model needs roughly 35-48 GB of GPU memory per run: two 24 GB GPUs with FSDP+QLoRA, or a single 48-80 GB GPU. QLoRA keeps the base model in 4-bit (about 35 GB) and trains small adapters; Answer.AI demonstrated this on two 24 GB gaming GPUs.

Full 16-bit fine-tuning needs about 16 bytes per parameter for weights, gradients and optimizer states -- on the order of 1 TB -- which is a multi-GPU server job. Renting an 80 GB cloud GPU is usually more practical than building hardware for occasional runs.

Running 70B Models Locally: Regional Context

EU / GDPR: A 70B local model is the practical ceiling of privately runnable AI quality for most organizations. Processing legal, medical or financial documents on-premises keeps prompts and outputs inside your own infrastructure, which supports data-minimization and transfer-risk controls; it does not by itself make a deployment GDPR-compliant, so assess the whole processing activity.

For organizations that require EU-supply-chain hardware, Apple (USA) and NVIDIA (USA) are non-EU vendors; Dell, HP and Lenovo sell GB10-based systems with local support. On the model side, Mistral Large 2 comes from an EU developer but is released under a research license that does not cover commercial use.

Japan: Qwen2.5 72B is a common choice for Japanese text: `ollama run qwen2.5:72b`. Documenting hardware and model versions (the `ollama ps` output identifies the loaded model) supports internal governance under the Japanese AI guidelines for business.

China: Locally hosted Qwen2.5 72B keeps data on your own infrastructure, which is relevant to China's Data Security Law (数据安全法). Whether an internally served model falls under the CAC generative-AI rules depends on whether it is offered to the public; confirm with qualified counsel.

What Are the Common Mistakes When Running 70B Models on Consumer Hardware?

Buying a GPU with less than 24 GB VRAM and expecting full 70B performance

A 12 GB card holds about a quarter of a Q4_K_M 70B model; the rest runs from system RAM, so speed is close to CPU-only. Even a 24 GB RTX 4090 manages only about 2-4 tok/sec. For interactive speed use 48 GB of VRAM, a 64 GB+ Apple Silicon Mac, or a smaller 27B-34B model.

Assuming a large-memory box is also a fast one

The DGX Spark and Ryzen AI Max+ 395 systems have 128 GB but only 256-273 GB/s of bandwidth, so a dense 70B model runs at about 3-5 tok/sec. A Mac Studio M5 Max with 64 GB has more than twice the bandwidth. Check bandwidth, not only capacity.

Not checking how much of the model runs on the GPU

Ollama offloads automatically, but a tight fit can leave layers on the CPU. Run `ollama ps` after loading and read the PROCESSOR column; "100% GPU" is the goal, and a CPU share explains slow output. Reduce context length or drop to Q3_K_S if the split is poor.

Forgetting that context needs memory too

A Q4_K_M 70B model is 43 GB, but a 32K context adds about 10.7 GB and 128K adds about 43 GB at FP16. A machine that fits the weights can still run out of memory at long context. Use an 8-bit KV cache or shorter context.

Running Q5_K_M or Q8_0 when speed matters

Every token reads the whole model, so Q8_0 (75 GB) runs at roughly half the speed of Q4_K_M (43 GB) on the same hardware. On bandwidth-limited machines (DGX Spark, Strix Halo, Mac mini) choose Q4_K_M for chat and use Q8_0 only when quality is the priority.

Ordering the $1,699 Mac mini M5 Pro and expecting 70B to run

The $1,699 M5 Pro Mac mini has 24 GB of unified memory. A 70B model needs the 64 GB build-to-order configuration, and the base Mac mini M6 ($899, 32 GB maximum) cannot run it at all. Confirm "M5 Pro" and 64 GB before ordering.

Hitting the default macOS GPU memory limit

By default macOS lets the GPU use roughly 75% of RAM, about 48 GB on a 64 GB Mac. Q4_K_M fits; Q5_K_M (50 GB) does not until you raise `iogpu.wired_limit_mb`.

Common Questions About Running 70B Models on Consumer Hardware

What hardware do I need to run a 70B LLM locally?

You need about 43 GB of fast memory for Llama 3.3 70B at Q4_K_M, plus 3-10 GB for context. Three setups hold the whole model in fast memory: an Apple Silicon Mac with 64 GB or more unified memory, two 24 GB NVIDIA GPUs (48 GB total VRAM), or a 128 GB unified-memory box such as the NVIDIA DGX Spark or an AMD Ryzen AI Max+ 395 mini PC. Capacity decides whether the model runs; memory bandwidth decides how fast. A single 24 GB GPU or a CPU-only PC can run it, but only at about 1-4 tokens/sec.

What is a 70B model?

A 70B model is a large language model with about 70 billion parameters (weights). Llama 3.3 70B has 70.6 billion and Qwen2.5 72B has 72.7 billion. The file size follows from the parameter count: about 141 GB at 16-bit, 75 GB at 8-bit (Q8_0) and 43 GB at 4-bit (Q4_K_M).

How much VRAM or RAM do I need for a 70B model?

Llama 3.3 70B needs 43 GB at Q4_K_M, 50 GB at Q5_K_M, 75 GB at Q8_0 and 141 GB at FP16 (Ollama library sizes), plus the KV cache for context: about 2.7 GB at 8K tokens, 10.7 GB at 32K and 43 GB at 128K. With 24 GB of VRAM only about half of a Q4_K_M model fits on the GPU, and the rest runs from system RAM.

What is the cheapest hardware that can run a 70B model usably?

For about 10 tokens/sec or more, the practical options are a Mac Studio M5 Max configured with 64 GB or more (the $2,499 base model has 36 GB), a used Mac Studio M2 Ultra with 64 GB or more, or two used 24 GB GPUs. The cheapest new Apple option is the Mac mini M5 Pro with 64 GB: the $1,699 base model has only 24 GB, and reviews put 64 GB configurations at roughly $2,700-$3,200. Its 307 GB/s bandwidth limits it to roughly 4-5 tokens/sec. GPU street prices are volatile: RTX 5090 listings were reported far above the $1,999 launch price in October 2026, so check live prices.

Can I run a 70B model on a single 24 GB GPU?

Yes, but slowly. A 24 GB card such as the RTX 4090 holds about 55% of a Q4_K_M 70B model; the remaining layers run from system RAM at CPU memory speed, which gives roughly 2-4 tokens/sec with fast DDR5. For interactive speed, use a smaller model (27B-34B fits in 24 GB, for example Qwen3.8-27B), two GPUs, or a 64 GB+ unified-memory machine.

Can I run a 70B model on two GPUs?

Yes. Two 24 GB cards (RTX 3090 or 4090) provide 48 GB of VRAM, which fits a Q4_K_M 70B model (43 GB) with a short context of a few thousand tokens. Ollama and llama.cpp split the layers across both cards automatically or with --tensor-split. Expect roughly 12-17 tokens/sec as an estimate, because every token passes through both cards in sequence.

What is the best inference runtime for running 70B LLMs on multi-GPU hardware?

For one user, Ollama or llama.cpp with layer splitting is the simplest and works with mixed GPUs. For several concurrent users or agents, vLLM with --tensor-parallel-size set to the GPU count, or SGLang, delivers much higher total throughput because it batches requests; both prefer identical GPUs and AWQ, GPTQ or FP8 weights. TensorRT-LLM gives the highest NVIDIA throughput but needs an engine build step.

How much GPU memory does a multi-agent system with 70B backends need?

Budget the weights once plus one KV cache per concurrent session. Llama 3.3 70B at Q4_K_M needs about 43 GB for weights; each active session adds about 0.33 MB per token of context at FP16. Four agents at 32K tokens each therefore need about 43 GB + 43 GB = 86 GB. Serve all agents from one shared server (vLLM, SGLang or llama.cpp with parallel slots) rather than loading one copy of the model per agent.

Does 256 GB of RAM help CPU inference of a 70B model?

It helps capacity, not speed. 256 GB lets you load Q8_0 (75 GB) or FP16 (141 GB) with long context, but decode speed is limited by memory channels: roughly 0.5-1.5 tokens/sec on a dual-channel desktop, about 3-6 on an 8-channel Threadripper PRO and about 4-8 on a 12-channel EPYC at Q4_K_M to Q8_0 (estimates from memory bandwidth, not benchmarks we ran).

Can the Mac mini M6 run a 70B model?

No. The Mac mini M6 ($899 starting) maxes out at 32 GB of unified memory and 170 GB/s bandwidth, below the 43 GB a Q4_K_M 70B model needs. The Mac mini M5 Pro (from $1,699, 307 GB/s, Thunderbolt 5) can be configured with up to 64 GB; the base $1,699 model has 24 GB, so order the 64 GB build-to-order configuration. Both shipped on September 22, 2026.

What is the NVIDIA DGX Spark and is it worth it for 70B inference?

The DGX Spark is NVIDIA's compact desktop AI computer with 128 GB of unified LPDDR5x memory at 273 GB/s. It loads 70B models at Q8_0 but is bandwidth-limited: LMSYS measured 2.7 tokens/sec decode for Llama 3.1 70B at FP8. It is better suited to CUDA work, fine-tuning and large MoE models than to fast dense-70B chat. NVIDIA raised the Founders Edition from $3,999 to $4,699 in February 2026 and a price of $6,950 was reported in October 2026, alongside a 64 GB version at $4,999 from partners.

How does 70B local quality compare to cloud models?

Meta reports 86.0 on MMLU (CoT) and 88.4 on HumanEval for Llama 3.3 70B Instruct. Frontier cloud models released since then score higher on reasoning-heavy tasks, while 70B local models remain competitive for summarization, extraction, drafting and routine code generation. Test both on your own prompts, for example with PromptQuorum, before investing in 70B hardware.

How much electricity does running a 70B model use?

A GPU draws up to its board power while generating: 350 W for an RTX 3090, 450 W for an RTX 4090 and 575 W for an RTX 5090. At $0.15 per kWh, an RTX 4090 at full load costs about $0.07 per hour for the GPU alone. Apple Silicon and unified-memory boxes draw much less power for the same model, but at lower speed.

Are 70B models worth it compared to smaller models for everyday tasks?

For complex reasoning, long-document analysis and nuanced writing, usually yes. For simple summarization, Q&A and classification, a 7B-14B model often produces similar output at a fraction of the hardware cost. Run both on your own prompts before investing in 70B hardware.

Can I fine-tune a 70B model on consumer hardware?

Only with QLoRA. QLoRA keeps the base model in 4-bit (about 35 GB) and trains small adapters; Answer.AI showed FSDP+QLoRA training a 70B model on two 24 GB gaming GPUs. Full 16-bit fine-tuning needs on the order of 1 TB of memory and is a multi-GPU server job. For anything beyond occasional QLoRA runs, renting an 80 GB cloud GPU is usually more practical.

What is the best quantization for 70B on Apple Silicon?

On a 64 GB Mac, use Q4_K_M (43 GB). macOS limits GPU-addressable memory to roughly 75% of RAM by default (about 48 GB on a 64 GB Mac), so Q5_K_M (50 GB) only works after raising the limit with sysctl iogpu.wired_limit_mb, and leaves little headroom. On 96-128 GB, Q8_0 (75 GB) is the near-lossless choice, at roughly half the speed of Q4_K_M because every token reads twice as many bytes.

Does Ollama choose the best quantization automatically?

No. ollama run llama3.3:70b downloads the default Q4_K_M (43 GB). Name the tag for another level, for example ollama run llama3.3:70b-instruct-q5_K_M or ollama run llama3.3:70b-instruct-q8_0. Run ollama ps after loading: the PROCESSOR column shows how much of the model runs on GPU versus CPU.

Can an iPhone or laptop run a 70B model?

Not an iPhone: current iPhones have 8-12 GB of RAM, while a Q4_K_M 70B model needs about 43 GB, so phones run 1B-8B models. Laptops can only with 64 GB+ of unified memory (high-end MacBook Pro configurations) or an AMD Ryzen AI Max+ 395 system with 128 GB. Gaming laptops with 16-24 GB of VRAM, such as an ASUS ROG Strix, fall into the slow partial-offload case.

Sources

A Note on Third-Party Facts

This article references third-party AI models, benchmarks, prices, and licenses. The AI landscape changes rapidly. Benchmark scores, license terms, model names, and API prices can shift between the time of writing and the time you read this. Before making deployment or compliance decisions based on this article, verify current figures on each provider’s official source: Hugging Face model cards for licenses and benchmarks, provider websites for API pricing, and EUR-Lex for current GDPR and EU AI Act text.

Run PromptQuorum with a local LLM, your own API keys, or both — you pick the backend.

Download the PromptQuorum Beta →

← Back to Local LLMs

MCP