Key Takeaways
- Edge deployment optimizes joules per token against a thermal design point β not tokens per second against VRAM, the desktop metric.
- A 1.5B model at 9.45 tok/s in a 2.1W envelope (Hailo-10H, per Hailo's published data) is a strong embedded result even though it would be a poor desktop one.
- Four hardware classes cover most products: SBC + NPU hat, dedicated accelerator module (M.2), integrated edge SoC, and embedded GPU module β matched to power budget, not to raw model size.
- NVIDIA Jetson AGX Thor (T5000) delivers 2,070 TFLOPS sparse FP4 at 40β130W with 128GB LPDDR5X at 273GB/s, per NVIDIA's datasheet β the highest end of the embedded GPU class.
- Memory bandwidth, not memory capacity, bounds token generation speed β a board with more DRAM at the same GB/s does not generate tokens faster.
- Industrial temperature range and part lifecycle must be confirmed before the model is chosen, not after β a 3-year-lifecycle part cannot go in a 10-year-service-life product.
- Many NPU toolchains accept only a fixed set of quantization schemes, so the target hardware's compiler constraints should shape the quantization plan, not the other way around.
What Are the Four Classes of Edge AI Hardware?
Four hardware classes cover nearly every edge LLM deployment, and picking the wrong class wastes more engineering time than picking the wrong model. Each class sits at a different point on the power-versus-flexibility curve. Matching the class to the product's power budget and service-life requirement comes before comparing individual chips inside that class.
π In One Sentence
Edge AI hardware splits into four classes β SBC + NPU hat, dedicated M.2 accelerator, integrated edge SoC, and embedded GPU module β matched to a product's power budget rather than to raw model size.
π¬ In Plain Terms
An SBC + NPU hat is cheap and flexible but runs hot in a sealed box. A dedicated M.2 accelerator does one job at very low power. An integrated SoC is a full computer on one chip sized for robotics and cameras. An embedded GPU module is desktop-class power shrunk to fit a product.
- SBC + NPU hat β a general-purpose single-board computer (Raspberry Pi 5-class) paired with a bolt-on NPU accelerator card. Lowest bill-of-materials cost, most flexible software stack, weakest sustained thermal performance once sealed into an enclosure without airflow.
- Dedicated accelerator module (M.2) β a fixed-function inference chip such as the Hailo-10H, added to an existing carrier board over an M.2/PCIe slot. Runs one job β quantized inference β at very low power; it is not a general compute platform and cannot run an OS or application logic.
- Integrated edge SoC β a single chip combining CPU, GPU or NPU, and I/O on one die, sized for a specific power envelope. The NVIDIA Jetson Orin family (Orin Nano Super, Orin NX, AGX Orin) and the Rockchip RK3588 are the default choice for volume robotics and camera products.
- Embedded GPU module β a discrete-class GPU shrunk into a module form factor for the highest end of edge compute, such as the NVIDIA Jetson AGX Thor (T5000). Used when the product genuinely needs desktop-class throughput β multi-camera perception, humanoid robotics β inside an embedded power and thermal budget.
Which Three Constraints Do Desktop Hardware Guides Ignore?
Desktop local-LLM guides optimize for VRAM capacity and peak tokens per second; embedded products are bound by three different constraints that a desktop benchmark never has to account for.
- Thermal design point, not peak spec. A datasheet's rated TOPS figure typically assumes airflow that most sealed, fan-less enclosures do not have. What matters is sustained throughput at 70Β°C ambient inside the actual housing β under thermal throttling, real throughput can run well below the datasheet peak.
- Memory bandwidth, not memory capacity, sets generation speed. Token generation (the decode phase) is bandwidth-bound: decode throughput is roughly bounded by memory bandwidth in GB/s divided by bytes read per token. A board with more DRAM but the same GB/s does not generate tokens faster β this is why the Hailo-10H's 1.2GB footprint and 2,048-token KV-cache at 2.1W average power is a meaningfully different design point than a desktop GPU's VRAM headroom.
- Industrial temperature range and part lifecycle, not benchmark charts. Consumer-grade SoCs are commonly sourced on 2β3 year design cycles. A product committed to a 10-year service life needs a part with a matching supply commitment and an extended operating range β a part that cannot be re-sourced in year four forces a mid-life hardware redesign regardless of how well it ran the target model on day one.
How Do You Specify Edge Compute for a Product?
Specify the power budget and the service-life requirement before evaluating any specific chip. The steps below are the order that avoids the most common rework: picking a model first, then discovering it does not fit the power or lifecycle constraint.
- 1Define the power budget first, not the model
Why it matters: Edge products have a fixed thermal or power envelope β battery capacity, a shared PoE rail, a passive-cooling limit β and the model has to fit what is left over after sensors, radios, and actuators take their share. - 2State a joules-per-token target, not a tokens-per-second target
Why it matters: A tokens/sec number without a power context cannot be compared across accelerator classes. Joules per token is the number that actually determines battery life or thermal headroom in the finished product. - 3Separate memory bandwidth from memory capacity in the spec sheet
Why it matters: Generation speed is bounded by GB/s to the weights and KV-cache, not by how much DRAM sits on the board. A larger buffer that cannot be read fast enough does not raise decode throughput. - 4Confirm industrial temperature range and sourcing lifecycle before the model, not after
Why it matters: A part with a 3-year sourcing commitment cannot go into a product with a 10-year service commitment, no matter how well it runs the target model in a lab. - 5Confirm which quantization schemes the NPU toolchain actually accepts
Why it matters: Many embedded NPU compilers support only a fixed set of weight, activation, and KV-cache quantization schemes. Validate this against the compiler before committing to a model, not after a compile failure. - 6Narrow candidates on the interactive selector below, then validate on real silicon
Why it matters: A power / model-size / video / temperature-range filter narrows the field to a shortlist in minutes. Only bench data measured on the actual accelerator, in the actual enclosure, confirms joules-per-token for your product.
Find the Edge Hardware That Fits Your Product
This is the fastest way to turn a power budget and a target model size into a shortlist. Set the power envelope, the model size in billions of parameters, and whether the product needs video ingest or an industrial temperature range β the table below updates to show which platform classes are worth benchmarking next.
Platforms that fit
| Platform | Power | Fits up to | Note |
|---|---|---|---|
| NVIDIA Jetson Orin Nano Super | 7β25 W | 8B | Entry dev kit; natural upgrade from a Raspberry Pi |
| NVIDIA Jetson Orin NX | 10β40 W | 13B | Mid-tier module, up to 157 TOPS |
| NVIDIA Jetson AGX Orin 64GB | 15β60 W | 34B | 275 TOPS; current volume robotics workhorse |
How Do Edge AI Chips Compare on Power and Compute?
Vendor-documented specifications for the platforms named in this guide, confirmed against primary vendor documentation on 2026-09-02. Figures marked "reported" come from secondary sources rather than a primary vendor datasheet and are hedged accordingly β treat them as approximate until you confirm them against your own vendor contact.
Platform | Class | Peak AI Compute | Power (TDP) | Memory / BW |
|---|---|---|---|---|
| Jetson AGX Thor (T5000) | Embedded GPU module | 2,070 TFLOPS FP4 sparse | 40β130W | 128GB LPDDR5X, 273GB/s |
| Jetson AGX Orin 64GB | Integrated edge SoC | 275 TOPS | 15β60W | 64GB LPDDR5 |
| Jetson Orin NX | Integrated edge SoC | ~157 TOPS (reported) | 10β40W | 8/16GB LPDDR5 |
| Jetson Orin Nano Super | Integrated edge SoC | ~67 TOPS (reported) | 7β25W | 8GB LPDDR5 |
| Hailo-10H | Dedicated accelerator (M.2) | Fixed-function inference | < 5W (M.2 form factor) | 1.2GB footprint (measured) |
| Rockchip RK3588 | SBC + NPU | 6 TOPS (3-core NPU) | Board-level, low single-digit W | Shared with host SBC |
| Ambarella N-series | Integrated vision SoC | Not disclosed | Not disclosed | CES 2026 launch, multi-sensor + edge GenAI |
| Qualcomm QRB-series | Integrated edge SoC | Platform-dependent | Platform-dependent | Dominant in drones β see sibling article |
Jetson AGX Thor also carries a QSFP 4Γ25GbE port and supports up to 32 MIPI CSI-2 cameras; NVIDIA states up to 7.5x the AI compute and 3.5x better energy efficiency versus Jetson AGX Orin. Both figures are NVIDIA's own comparison, not an independent benchmark run for this article.
What Does a 1.5B Model Actually Look Like on Dedicated Silicon?
Hailo published a measured result for Qwen2-1.5B-Instruct running on the Hailo-10H that illustrates the whole thesis of this guide: 9.45 tokens per second at 2.1W average power. On a desktop rubric, 9.45 tok/s from a 1.5B model is unremarkable. On an embedded rubric β a sub-5W M.2 module drawing barely more than a phone charger's idle draw β it is the difference between a product that ships on a small battery and one that needs a bigger enclosure and a fan.
The figures below are Hailo's own published benchmark data, not an independent measurement run for this article β cite them as vendor-documented when you use them in a spec.
Metric | Value |
|---|---|
| Model | Qwen2-1.5B-Instruct |
| Throughput | 9.45 tok/s |
| TTFT (96 input tokens) | 289 ms |
| Average power | 2.1 W |
| Memory footprint | 1.2 GB |
| KV-cache | 2,048 tokens (~1,536 words) |
| Weight quantization | 4-bit symmetric, group-wise |
| Activation quantization | 8-bit asymmetric, per-tensor |
| KV-cache quantization | 8-bit asymmetric, per-tensor |
| Runtime / OS support | HailoRT β Linux, Windows, Android |
HailoRT ships for Linux, Windows, and Android, and the module itself is a sub-5W M.2 form factor β the relevant comparison point for a battery- or PoE-constrained product, not a desktop tokens/sec leaderboard.
How Should You Quantize a Model for an Embedded NPU Target?
Embedded NPU toolchains typically accept a fixed, smaller set of quantization schemes than a desktop inference stack, so the target hardware's compiler should decide the quantization plan, not the other way around. Confirm the accepted scheme before selecting a model, not after a failed compile.
Static quantization fixes the scale/zero-point for weights and activations ahead of time, using a calibration dataset β this is what most fixed-function NPU compilers require, because it lets the chip skip the runtime cost of computing quantization parameters on the fly. Dynamic quantization computes those parameters at inference time and is more flexible, but few embedded NPU compilers support it, since it adds latency the fixed-function pipeline was built to avoid.
Per-tensor activation quantization (one scale factor for an entire activation tensor, as Hailo's published Qwen2-1.5B configuration uses at 8-bit asymmetric) is the common embedded default: it is cheaper for the NPU to apply than a per-channel scheme, at a modest accuracy cost that is usually acceptable for the short, task-specific prompts typical of an embedded product.
A quantized KV-cache β 8-bit asymmetric per-tensor in the Hailo-10H example above β shrinks the memory the accelerator has to hold and read back per generated token, directly raising the token/sec ceiling described in the memory-bandwidth constraint above. See how LLM quantization works and what each format actually changes for the underlying mechanics of weight, activation, and KV-cache quantization shared across desktop and embedded targets.
What Should You Buy to Prototype an Edge AI Product?
Prototype on a development kit in the same hardware class you plan to ship, not on a desktop GPU box β the thermal and power behavior of the shipping class is exactly what a desktop rig cannot tell you.
- NVIDIA Jetson developer kits β the standard way to prototype against the integrated-edge-SoC and embedded-GPU-module classes before committing to a production carrier board. Confirm current model availability and pricing directly with NVIDIA or an authorized distributor; specific SKUs and prices change too often to state reliably here.
- Hailo M.2 accelerator modules β the standard way to prototype the dedicated-accelerator class on an existing carrier board that already has a free M.2/PCIe slot. Confirm current SKU availability directly with Hailo or an authorized distributor.
- Raspberry Pi AI HAT+ β the standard way to prototype the SBC + NPU hat class cheaply before deciding whether the product needs a higher compute class at all. Confirm current pricing and availability directly with Raspberry Pi or an authorized reseller.
- Industrial carrier boards and enclosures β needed once a prototype moves from a lab bench toward the extended temperature range and vibration tolerance a shipping product requires; specify these from an industrial-embedded vendor, not a consumer carrier board, once the product timeline calls for it.
When Is Edge Inference the Wrong Call?
Edge inference is the wrong default for three common product situations β forcing it anyway usually costs more engineering time than it saves in latency or privacy.
- Bursty workloads with long idle periods. If the product only needs inference occasionally and has reliable connectivity the rest of the time, the fixed power and BOM cost of dedicated edge silicon is spent maintaining capacity the product uses rarely β a cloud call during the active window is usually cheaper over the product's life.
- Tasks that genuinely depend on frontier-scale reasoning. A model small enough to fit an edge power budget will not match a large hosted model on open-ended reasoning, multi-step planning, or broad world knowledge β edge silicon is the right call for a narrow, well-specified task, not a general assistant.
- Products with reliable connectivity and no privacy or latency constraint. If nothing about the product requires data to stay on-device and nothing requires a sub-100ms response with no network round trip, the case for paying the edge silicon's power and cost budget weakens considerably β verify the constraint is real before designing around it.
What Regulation Applies to AI-Driven Machinery in 2026?
The EU Machinery Regulation (2023/1230), applying from January 2027, treats AI-driven safety functions in machinery as high-risk β this intersects directly with the EU AI Act for any edge product where an LLM or VLM output can affect a physical safety function. This is general background information, not legal advice; confirm applicability to a specific product with qualified regulatory counsel before shipping.
The practical trigger is narrow but important: if an edge inference result (an object classification, a planning decision, a video analytics alert) feeds into a safety-relevant control function of a machine, the combination of the Machinery Regulation's essential health and safety requirements and the EU AI Act's high-risk classification can both apply. A product that only uses edge inference for a non-safety function (a voice interface, an analytics dashboard) sits outside this specific overlap, though other AI Act obligations may still apply depending on the use case.
This is a compliance dimension that a desktop local-LLM guide never has to address, because a desktop application is not, by definition, a piece of machinery with a safety function. Building it into the hardware specification early β rather than retrofitting it after a design is locked β is the practical lesson for products in scope.
Frequently Asked Questions
What is the core difference between desktop and edge LLM hardware selection?
Desktop selection optimizes tokens per second against available VRAM. Edge selection optimizes joules per token against a fixed thermal design point and power budget β the model has to fit what power the product has left over, not the other way around.
Is a small model on a dedicated NPU really better than a bigger model on a GPU?
For a fixed power budget, often yes. A 1.5B model at 9.45 tok/s in a 2.1W envelope on the Hailo-10H, per Hailo's published benchmark, is a poor desktop result and a strong embedded one β the comparison only makes sense once the power constraint is fixed.
What is a thermal design point and why does it matter for edge LLMs?
It is the sustained operating condition β typically ambient temperature inside a sealed, often fan-less enclosure β at which a chip must deliver its rated performance, as opposed to a datasheet's peak spec measured with airflow most products don't have. Real throughput under thermal throttling can run well below that peak.
Why does memory bandwidth matter more than memory capacity for token generation?
Token generation is bandwidth-bound: decode speed is roughly bounded by memory bandwidth in GB/s divided by bytes read per token. A board with more DRAM at the same GB/s does not generate tokens faster β bandwidth, not capacity, sets the ceiling.
What is an industrial temperature range and why does it matter for product lifecycle?
It is an extended operating range (well beyond typical consumer 0β40Β°C ratings) that a part is rated and sourced for over a multi-year commitment. A consumer-grade part commonly sourced on a 2β3 year cycle cannot go into a product with a 10-year service life without forcing a mid-life hardware redesign.
How much power does the Hailo-10H use running a small LLM?
Per Hailo's published data, the Hailo-10H averages 2.1W running Qwen2-1.5B-Instruct at 9.45 tok/s, with a 1.2GB memory footprint and a 2,048-token KV-cache, in a sub-5W M.2 form factor.
What compute does the NVIDIA Jetson AGX Thor deliver?
Per NVIDIA's datasheet, the Jetson AGX Thor (T5000) delivers 2,070 TFLOPS of sparse FP4 compute (1,035 TFLOPS dense FP4, 517 TFLOPS dense FP8) at 40β130W, with 128GB of LPDDR5X memory at 273GB/s over a 256-bit bus.
Should I quantize weights, activations, or the KV-cache for an edge NPU?
Usually all three, within whatever scheme the target NPU's compiler accepts. Static, per-tensor activation quantization and a quantized KV-cache are the common embedded defaults because most fixed-function NPU toolchains support only a limited, fixed set of schemes β confirm the accepted scheme before selecting a model.
When should I not use edge inference at all?
Skip edge inference for bursty, rarely-used workloads with reliable connectivity, for tasks that genuinely need frontier-scale reasoning beyond what an edge-sized model can deliver, and for products with no real privacy or latency constraint forcing data to stay on-device.
How do I start specifying edge compute for a new product?
Define the power budget and required service life before evaluating any specific chip, state a joules-per-token target instead of a tokens-per-second target, confirm memory bandwidth and industrial temperature range against the product's constraints, then use the interactive selector on this page to narrow candidates before benchmarking real silicon.