On-Device AI & Memory: Why HBM Memory Drives Local AI Speed (2026)

During the decode phase of LLM inference, the GPU/NPU must load the entire model into memory, perform one forward pass per token, and write the output. The bottleneck: how fast can you feed parameters to the compute units? This is memory bandwidth, not compute TOPS.

Simplified formula: tokens/sec = memory_bandwidth / (model_size_in_bytes × bytes_per_precision). For FP16 (2 bytes per parameter), a 7B model = 14 GB. At LPDDR5X 85.6 GB/s: 85.6 GB/s ÷ 14 GB = ~6 tokens/sec theoretical maximum. In practice, 3–5 tokens/sec due to compute and cache overhead.

Quantization changes the equation dramatically. Q4 (4-bit, 0.5 bytes per parameter) shrinks a 7B model to 3.5 GB. 85.6 GB/s ÷ 3.5 GB = ~24 tokens/sec theoretical. Real-world ~8–15 tokens/sec, a 3–4x improvement.

Data-center H100 with HBM3E (1.229 TB/s) can sustain 100+ tokens/sec for the same model, because HBM is 14x faster. This is why frontier models (70B, 405B) run only in data centers—you need HBM bandwidth.

Inference is different from training. Training is compute-bound (you do 1000+ passes to update weights). Inference (especially once a model is prompt-cached) is a single forward pass, pure memory-bandwidth bound. This is why inference on phones is so much slower than on servers—you cannot engineer your way out of the bandwidth gap.

SK Hynix — HBM Leader: SK holds ~62% of the HBM market (peak Q2 2025, projected >50% through 2026). SK supplies HBM3E to Nvidia for H100, H200, and B200 GPUs. SK is sampling HBM4 (>2 TB/s) to Nvidia for next-gen GPUs launching 2026-2027. HBM revenue is critical for SK's data-center division; Samsung is chasing.

Samsung — LPDDR5X & PIM Push: Samsung manufactures LPDDR5X for Galaxy S26, Snapdragon phones, and Apple (for A18 Pro). Samsung is developing LPDDR5X-PIM (Processing-In-Memory), which embeds compute operations inside the memory die itself. This reduces data round-trips and could eventually narrow the bandwidth gap. LPDDR5X-PIM is in lab/early sampling phase; production volume unlikely before 2027-2028.

Competitive Dynamics: Samsung has pursued HBM (HBM3, HBM3E samples), but loses to SK due to yield and cost. Samsung pivoted to LPDDR5X-PIM as a differentiated strategy: make the phone memory smarter rather than trying to match HBM bandwidth. This is a "can't compete, so innovate differently" move.

Both companies supply Nvidia: SK Hynix provides HBM for Nvidia GPU VRAM. Samsung may supply standard DRAM for Nvidia's CPU/host memory. Neither supplies the compute (Nvidia designs the GPU cores). The ecosystem is specialized: design, memory, compute are separate.

Timeline: HBM4 enters production 2026-2027 (SK Hynix). LPDDR5X-PIM enters limited production 2027-2028 (Samsung). LPDDR6 enters phones 2027-2028, with ~2x bandwidth vs LPDDR5X (~200+ GB/s vs 85.6 GB/s)—still 6x slower than HBM3E, but a meaningful improvement.

The Galaxy S26 Exynos 2600 with LPDDR5X 85.6 GB/s defines the practical ceiling for on-device LLM inference. A quantized 7B model at Q4 reaches ~8–15 tokens/sec realistic performance. This is suitable for latency-sensitive tasks (autocomplete, real-time transcription, simple tasks) but impractical for long conversations.

Model size limits: A 7B model is practical (3–4 hour latency per 100-token response). A 13B model at Q4 (~6.5 GB) hits 85.6 GB/s ÷ 6.5 GB = ~13 tokens/sec, barely an improvement. A 70B model at Q4 (~35 GB) hits 85.6 GB/s ÷ 35 GB = ~2 tokens/sec—unusable.

Quantization is essential: FP16 (2 bytes/param) is impractical. Q4 (0.5 bytes/param) is the sweet spot—4x smaller models with acceptable quality loss. Q3 (3-bit) saves more space but loses quality; Q5 loses less quality but gains little bandwidth improvement.

Speed vs quality tradeoff: 7B Q4 is ~8–15 tokens/sec (acceptable for some use cases). 3B Q4 is ~24–36 tokens/sec (excellent for simple tasks). 1B Q4 is ~60+ tokens/sec (real-time, Pixel 3 era performance on modern hardware).

Practical use cases: autocomplete, real-time code suggestion, on-device transcription, local summarization. Not practical: long conversations, complex reasoning, multi-turn dialogue without caching.

The bottleneck is bandwidth, not compute or weight size. Even if you reduce model parameters to 0, the memory still needs to transfer them, and the bandwidth is fixed. This is why on-device AI is architecturally limited—you cannot engineer yourself out of 85.6 GB/s on a phone form factor.

A Nvidia H100 GPU with HBM3E (1.229 TB/s) is 14x faster at inference throughput than a Galaxy S26 (LPDDR5X 85.6 GB/s). This gap is not due to compute FLOPS (both are fast), but pure memory bandwidth. The H100 can do 100+ tokens/sec; the S26 does 8–15 tokens/sec for the same 7B Q4 model.

Why the gap exists: HBM is physically different. LPDDR5X is a thin film sitting next to the CPU (power-efficient for phones). HBM is a stack of memory chips directly bonded to the GPU using through-silicon vias (TSVs), creating massive bandwidth. HBM stacks are 2+ inches tall; impossible to fit in a phone.

Why it can't be closed: Phones are thermally and power-constrained. HBM consumes significant power (~100+ W for a full stack). LPDDR5X is ~5–10 W. Phones run on batteries; data centers have unlimited power/cooling. You cannot physically fit HBM bandwidth into a phone without destroying battery life.

Consequence: On-device AI will always be slower than cloud AI for large models. This is not a technology gap that will close—it's a physical constraint (power, thermal, form factor). Smaller models, aggressive quantization, and clever caching are the solutions, not hoping for better memory.

The flip side: on-device is private, offline-capable, and zero-latency for privacy-sensitive tasks. The 14x speed penalty is the price of privacy. Data-center AI trades speed for privacy loss.

Future: LPDDR5X-PIM (2027-2028) and LPDDR6 (2027-2028) will improve phone bandwidth to ~200 GB/s (still 6x slower than HBM3E). This is meaningful (double the tokens/sec) but won't make phones match data-center speed. The gap will remain 6x, not 14x.

HBM4 (SK Hynix, 2026-2027): >2 TB/s per stack. Arriving first in Nvidia next-gen GPUs (post-B200). HBM4 is irrelevant for phones but will push data-center inference even faster. SK Hynix is the primary supplier.

LPDDR6 (2027-2028): ~200+ GB/s (estimated; x64 bus, 12.8 Gbps). That's ~2.3x LPDDR5X bandwidth. For a 7B Q4 model: 200 GB/s ÷ 3.5 GB ≈ 57 tokens/sec theoretical (up from 24). In practice, ~20–35 tokens/sec realistic. A meaningful improvement, but still 3x slower than data-center HBM3E. LPDDR6 will ship in Galaxy S27/S28 era (2027-2028).

LPDDR5X-PIM (Samsung, 2027-2028): Processing-In-Memory embeds compute inside the DRAM die. Instead of loading every model weight from memory, you compute operations (matrix multiplies) inside the memory itself, eliminating some round-trips. Samsung is actively developing this. Estimated 50%+ throughput improvement vs standard LPDDR5X, if successful. Still won't match HBM bandwidth, but a clever engineering solution.

Reality: Even with LPDDR6 + PIM, phones will still be 3–6x slower than data centers for inference. This is the fundamental gap that cannot be closed without changing the phone's physical design (larger, hotter, more power).

For on-device AI 2026-2027: Exynos 2600 + LPDDR5X is the current baseline. Exynos 2700 (S27) may improve compute, but bandwidth will be the bottleneck. Expect LPDDR6 and PIM as incremental improvements, not transformative.