On-Device AI & Memory: Why HBM Memory Drives Local AI Speed (2026)

During the decode phase of LLM inference, the GPU/NPU must load the entire model into memory, perform one forward pass per token, and write the output. The bottleneck: how fast can you feed parameters to the compute units? This is memory bandwidth, not compute TOPS.

Simplified formula: tokens/sec = memory_bandwidth / (model_size_in_bytes × bytes_per_precision). For FP16 (2 bytes per parameter), a 7B model = 14 GB. At LPDDR5X 85.6 GB/s: 85.6 GB/s ÷ 14 GB = ~6 tokens/sec theoretical maximum. In practice, 3–5 tokens/sec due to compute and cache overhead.

Quantization changes the equation dramatically. Q4 (4-bit, 0.5 bytes per parameter) shrinks a 7B model to 3.5 GB. 85.6 GB/s ÷ 3.5 GB = ~24 tokens/sec theoretical. Real-world ~8–15 tokens/sec, a 3–4x improvement.

Data-center H100 with HBM3E (1.229 TB/s) can sustain 100+ tokens/sec for the same model, because HBM is 14x faster. This is why frontier models (70B, 405B) run only in data centers—you need HBM bandwidth.

Inference is different from training. Training is compute-bound (you do 1000+ passes to update weights). Inference (especially once a model is prompt-cached) is a single forward pass, pure memory-bandwidth bound. This is why inference on phones is so much slower than on servers—you cannot engineer your way out of the bandwidth gap.

SK Hynix — HBM Volume Leader: SK leads HBM shipments, with reported share ranging roughly 50–62% across different 2026 quarters and measurement methodologies (revenue vs. bit-share vs. quarter). SK supplies HBM3E to Nvidia for H100, H200, and B200 GPUs, and is targeting mass production of its 16-layer HBM4 (48 GB, >2 TB/s) for Q3 2026. HBM revenue remains critical for SK's data-center division.

Samsung — HBM4E Speed Lead, LPDDR5X-PIM Validated: Samsung shipped the industry's first 12-layer HBM4E samples on May 29, 2026, at 3.6 TB/s per stack — ahead of SK Hynix's HBM4 spec on that metric, though SK still ships more total HBM volume. On the phone side, Samsung presented LPDDR5X-PIM at Hot Chips 2026 (August 25, 2026) as validated, tested silicon: an edge AI accelerator SoC ran Llama-3.1-8B at 81.3 tokens/sec versus 27.0 tokens/sec on standard LPDDR5X (3.01x), with up to 8x internal bandwidth in PIM mode. The chip uses the same 561-ball package as standard LPDDR5X, so it is a drop-in replacement, not a new form factor.

Competitive Dynamics: Samsung continues to sample HBM (HBM3E, now HBM4E) but has historically trailed SK on yield and total HBM volume. Samsung's positioning for LPDDR5X-PIM is explicit: a complement to HBM, not a replacement — targeting smartphones, AI PCs, and edge accelerators where HBM's cost, power draw, and packaging complexity are prohibitive, rather than trying to match HBM bandwidth directly.

Both companies supply Nvidia: SK Hynix and Samsung both provide HBM for Nvidia GPU VRAM; standard DRAM (LPDDR/DDR) covers host/CPU memory. Neither supplies the compute (Nvidia designs the GPU cores). The ecosystem stays specialized: design, memory, compute are separate.

Timeline: HBM4 is in mass production now (SK Hynix targeting Q3 2026 for its 16-layer part; Samsung already sampling 12-layer HBM4E). LPDDR5X-PIM is validated silicon, ahead of the 2027-2028 production estimate given at last refresh — expect it in edge AI accelerators before phones. LPDDR6/LPDDR6X-PIM is Samsung's next target, with an initial JEDEC spec hoped for later in 2026.

The Galaxy S26 Exynos 2600 with LPDDR5X 85.6 GB/s defines the practical ceiling for on-device LLM inference. A quantized 7B model at Q4 reaches ~8–15 tokens/sec realistic performance. This is suitable for latency-sensitive tasks (autocomplete, real-time transcription, simple tasks) but impractical for long conversations.

Model size limits: A 7B model is practical (3–4 hour latency per 100-token response). A 13B model at Q4 (~6.5 GB) hits 85.6 GB/s ÷ 6.5 GB = ~13 tokens/sec, barely an improvement. A 70B model at Q4 (~35 GB) hits 85.6 GB/s ÷ 35 GB = ~2 tokens/sec—unusable.

Quantization is essential: FP16 (2 bytes/param) is impractical. Q4 (0.5 bytes/param) is the sweet spot—4x smaller models with acceptable quality loss. Q3 (3-bit) saves more space but loses quality; Q5 loses less quality but gains little bandwidth improvement.

Speed vs quality tradeoff: 7B Q4 is ~8–15 tokens/sec (acceptable for some use cases). 3B Q4 is ~24–36 tokens/sec (excellent for simple tasks). 1B Q4 is ~60+ tokens/sec (real-time, Pixel 3 era performance on modern hardware).

Practical use cases: autocomplete, real-time code suggestion, on-device transcription, local summarization. Not practical: long conversations, complex reasoning, multi-turn dialogue without caching.

The bottleneck is bandwidth, not compute or weight size. Even if you reduce model parameters to 0, the memory still needs to transfer them, and the bandwidth is fixed. This is why on-device AI is architecturally limited—you cannot engineer yourself out of 85.6 GB/s on a phone form factor.

A Nvidia H100 GPU with HBM3E (1.229 TB/s) is 14x faster at inference throughput than a Galaxy S26 (LPDDR5X 85.6 GB/s). This gap is not due to compute FLOPS (both are fast), but pure memory bandwidth. The H100 can do 100+ tokens/sec; the S26 does 8–15 tokens/sec for the same 7B Q4 model.

Why the gap exists: HBM is physically different. LPDDR5X is a thin film sitting next to the CPU (power-efficient for phones). HBM is a stack of memory chips directly bonded to the GPU using through-silicon vias (TSVs), creating massive bandwidth. HBM stacks are 2+ inches tall; impossible to fit in a phone.

Why it can't be closed: Phones are thermally and power-constrained. HBM consumes significant power (~100+ W for a full stack). LPDDR5X is ~5–10 W. Phones run on batteries; data centers have unlimited power/cooling. You cannot physically fit HBM bandwidth into a phone without destroying battery life.

Consequence: On-device AI will always be slower than cloud AI for large models. This is not a technology gap that will close—it's a physical constraint (power, thermal, form factor). Smaller models, aggressive quantization, and clever caching are the solutions, not hoping for better memory.

The flip side: on-device is private, offline-capable, and zero-latency for privacy-sensitive tasks. The 14x speed penalty is the price of privacy. Data-center AI trades speed for privacy loss.

Update: Samsung validated LPDDR5X-PIM at Hot Chips 2026 (August 25, 2026) — 3.01x measured throughput (27.0 → 81.3 tokens/sec on Llama-3.1-8B) on a pin-compatible chip, well ahead of the 2027-2028 timeline this page previously cited. LPDDR6 (still 2027-2028) targets ~200 GB/s, ~6x slower than HBM3E. Together, PIM now and LPDDR6 later meaningfully narrow the phone-vs-data-center gap without closing it — the 14x raw bandwidth gap becomes closer to 4-6x in practice once PIM ships in phone SoCs.

HBM4 / HBM4E (SK Hynix + Samsung, now in mass production): >2 TB/s per stack for SK Hynix's 16-layer HBM4 (targeting Q3 2026 mass production); 3.6 TB/s per stack for Samsung's 12-layer HBM4E, already sampling since May 2026. Arriving first in Nvidia next-gen GPUs (post-B200). Still irrelevant for phones — it will push data-center inference even faster while phones stay bandwidth-starved.

LPDDR5X-PIM (Samsung, validated August 2026): Processing-In-Memory embeds compute inside the DRAM die. Instead of only loading every model weight from memory, the chip computes some operations (matrix multiplies) inside memory itself, eliminating round-trips. This moved from "developing" to "demonstrated, tested silicon" at Hot Chips 2026: 3.01x measured inference speedup (27.0 → 81.3 tokens/sec on Llama-3.1-8B) with up to 8x internal bandwidth in PIM mode, on a pin-compatible chip. It will not match HBM bandwidth, but it is real and ahead of the prior 2027-2028 production estimate.

LPDDR6 / LPDDR6X-PIM (2027-2028): ~200+ GB/s estimated for plain LPDDR6 (x64 bus, 12.8 Gbps) — roughly 2.3x LPDDR5X bandwidth. For a 7B Q4 model: 200 GB/s ÷ 3.5 GB ≈ 57 tokens/sec theoretical (up from 24), ~20–35 tokens/sec realistic. Samsung is already looking past LPDDR5X-PIM to LPDDR6X-PIM, and hopes for an initial JEDEC specification later in 2026. LPDDR6 itself is expected in the Galaxy S27/S28 era.

Reality: Even with LPDDR5X-PIM shipping and LPDDR6 on the roadmap, phones will still be several times slower than data centers for inference — Samsung itself positions PIM as a complement to HBM for cost/power-constrained devices, not a bandwidth-parity solution. This is a physical-design gap (form factor, power, heat), not one that a single generation of memory closes.

For on-device AI, late 2026: Exynos 2600 + standard LPDDR5X is still the shipping baseline for Galaxy S26. LPDDR5X-PIM is validated but has not yet appeared in a shipping phone SoC — expect edge AI accelerators to get it first. Track the next Exynos/Snapdragon generation for whether PIM reaches phones before LPDDR6 does.