Running LLMs and VLA Models On-Robot 2026: What Fits, What Doesn't

A robot's software stack has at least five distinct jobs, and only two of them are plausible territory for an LLM or VLA today. Treating "AI in robotics" as one undifferentiated blob is the single most common framing error in coverage of this space — it leads to the wrong question ("can it run an LLM?") instead of the useful one ("which of these five jobs is the model actually doing, and at what rate?").

The five jobs, in the order data flows through them: perception (turning camera/lidar/IMU data into a structured world representation), state estimation (fusing that data over time into a stable estimate of the robot's own pose and the world's state), task planning (deciding *what* to do next — "pick up the blue cup," "navigate to the loading dock"), motion planning (translating a task into a feasible trajectory — joint angles, a path, a grasp approach), and control (converting the planned trajectory into motor commands, moment by moment, while rejecting disturbances).

Perception is where VLMs and lightweight vision models already do real work — object detection, scene description, semantic labeling — usually at the 5–30 Hz range a camera feed produces. Task planning is where a VLA or LLM adds genuine value: turning a natural-language or visual goal into a symbolic subgoal a downstream system can act on. Those are the two jobs this article treats as legitimate VLA/LLM territory.

State estimation, motion planning, and control are different in kind. State estimation typically runs a Kalman filter or particle filter — deterministic, well-understood math that a neural network doesn't improve on for the core estimation problem. Motion planning is dominated by classical trajectory optimizers (CHOMP, RRT*, MPC-based planners) precisely because they offer verifiable constraint satisfaction — a property a language model's output does not have. Control is the fastest and least model-suited of all: it runs a feedback loop against sensor error at rates a transformer forward pass cannot approach.

The binding constraint on what a robot can do with a VLA is the model's achievable decisions-per-second — not the accelerator's peak TOPS rating. A hardware datasheet quoting 100 TOPS says nothing about how many times per second a 2–7B-parameter VLA can run a full forward pass and produce a usable action; that number depends on model size, quantization, batch size, camera count, and memory bandwidth together, and it is usually far below what the TOPS figure implies for a single-stream, low-latency workload.

Different loops tolerate wildly different latency. A locomotion balance loop on a legged or wheeled-balancing robot needs corrections in the single-digit milliseconds — that is a 100–1,000 Hz control loop, and no published VLA architecture runs at that rate. A manipulation task-planning loop (deciding "reach for the cup" vs. "reach for the bowl") can tolerate 100–1,000 ms of latency without visibly degrading behavior — that is the 1–10 Hz range a VLA realistically operates in. A navigation-level replanning loop (re-routing around an obstacle) can often tolerate multiple seconds.

This is why the model output rate, not the underlying model FLOPs, is the number to budget against. Two VLAs with similar parameter counts can have very different achievable Hz depending on how much of the pipeline (vision encoder, action decoder, tokenization) sits on the critical path per decision — and depending on how many camera streams feed the same forward pass, since most VLA architectures process each additional camera as additional context, not additional parallelism.

A model in the control loop is a safety liability, not just a performance one. Neural network outputs are not formally verifiable in the way a PID or MPC controller's stability margins are — there is no proof that a transformer will never produce an unsafe action for an unseen input distribution. This is the core reason safety-rated stops, torque limits, and collision-avoidance interlocks stay in deterministic control code across every production robot stack we are aware of, regardless of how capable the reasoning-tier model becomes: a certifiable safety function needs behavior that can be exhaustively characterized, and a large neural network cannot currently offer that guarantee.

Use the calculator below to estimate what decision rate a given hardware/camera-count combination can actually hit, before committing to a model or a board. The estimates are engineering approximations for early budgeting, not vendor benchmarks — validate against the actual model and hardware before finalizing a design.

Five VLA models cover most of the open and semi-open landscape robotics teams evaluate in 2026, plus one restricted-access model worth tracking for where the field is headed. Parameter counts below marked with "~" come from secondary reporting rather than a primary vendor spec sheet — treat them as directional, not exact.

NVIDIA Isaac GR00T N1 and its N1.5 revision are humanoid-oriented VLA models, openly released, at roughly ~2.2B parameters (secondary-sourced figure). NVIDIA explicitly states that GR00T N1.5 runs on its Jetson Thor board — this is a vendor-confirmed onboard deployment target, not an inferred one.

Physical Intelligence's π0 (pi-zero) is a flow-matching VLA at roughly ~3B parameters (secondary-sourced), released with open weights, built on a PaliGemma-class vision-language backbone (secondary-sourced detail). It has drawn attention for smooth, high-frequency action generation relative to earlier VLA designs.

OpenVLA, at 7B parameters, is the common academic baseline for VLA research — open, well-documented, widely benchmarked against. Its size makes it the reference point for "how big does a capable VLA need to be," not typically a first choice for a tight onboard latency budget.

Google DeepMind's Gemini Robotics On-Device has an undisclosed parameter count and restricted/partner access — it is not an option most teams can pull today. What makes it worth tracking regardless: Google DeepMind states it was explicitly designed to run locally on the robot with no internet connection. That is the clearest signal from a major lab that on-robot inference is now a first-class deployment target, not a research afterthought bolted onto a cloud-first system.

SmolVLA is a compact, open VLA at roughly ~450M parameters (secondary-sourced) — the realistic option when the target hardware is genuinely low-power (a small accelerator rather than a full Jetson-class board), where a 2–7B model's memory footprint and forward-pass latency simply do not fit the budget.

Octo and RT-2 are earlier-generation VLA architectures, useful mainly for historical framing of how the field arrived at current flow-matching and tokenized-action designs — not a live recommendation for a 2026 deployment decision.

Beyond the VLA category, NVIDIA states that general-purpose Llama, Qwen, and DeepSeek (LLMs), plus Qwen2.5-VL and Llama 3.2 Vision (VLMs), run on Jetson Thor. These handle the language-understanding and scene-description layer in a robot stack where a full VLA is unnecessary — a service robot answering a verbal question, or a manipulation arm reading a text label, does not need an action-output model at all.

ROS 2 is the practical boundary between the slow reasoning tier and the fast control tier in most production robot stacks. A VLA node publishes a task-level intent — a target pose, a subgoal, a grasp point — as a ROS 2 message, and a separate, classical control node subscribes to that message and executes it at control-loop rates. The VLA never talks to the motors directly; it talks to a topic, and a deterministic node owns everything downstream of that topic.

On NVIDIA hardware (Jetson Orin and Thor tiers), TensorRT and TensorRT-LLM are the standard runtime path for getting a quantized VLA or LLM to its achievable inference rate — they handle kernel fusion, precision calibration (FP8/INT8/INT4), and hardware-specific optimization that a naive PyTorch inference loop leaves on the table. Skipping this step is the single most common reason a model that "should" hit a target Hz on paper does not hit it in practice.

For smaller onboard targets that are not NVIDIA-based — a Hailo-10H accelerator, or a compact ARM-based board — ExecuTorch and llama.cpp are the relevant runtime paths for the language/perception components. Neither is a drop-in VLA runtime today; teams targeting these smaller boards are more often running a compact VLM (for perception/scene description) than a full VLA, with task planning either simplified into rule-based logic or run less frequently on a cadence the smaller accelerator can sustain.

The practical integration pattern, in order: (1) the VLA/VLM node runs asynchronously at its achievable Hz, publishing intent to a ROS 2 topic; (2) a classical planner subscribes, converts intent into a trajectory; (3) a control node executes the trajectory at 100–1,000 Hz, using its own sensor feedback loop — never waiting on the next VLA output to act. If the VLA node stalls or lags, the control node continues executing the last valid trajectory or safely holds position; it does not block on the slow tier.

Autonomous mobile robots (AMRs) in warehouses and agricultural machinery represent the deployed reality of robot autonomy today — humanoids get the media coverage but are the least deployed machine class discussed in this article. This gap matters for anyone deciding where to invest engineering time: the highest-compute, most-discussed robot category is also the one with the fewest units actually running in production environments right now.

Humanoids are the highest-compute, least-deployed class: they carry the most sensors, the most degrees of freedom, and correspondingly the heaviest reasoning-tier workload (multiple camera streams, whole-body task planning), which is exactly why they are the primary target for the newest, most capable onboard silicon (Jetson Thor) and the newest VLA architectures (GR00T N1.5). Their deployment volume today is small relative to the attention they receive.

Autonomous mobile robots (AMRs) — the wheeled robots moving inventory in warehouses — have shipped commercially at meaningful scale for years, running classical navigation and obstacle-avoidance stacks. What is changing in 2026 is the addition of a language/vision interface layer on top of an already-mature autonomy stack: a VLM for scene description or a lightweight task-planning layer that lets a human give a natural-language instruction, not a wholesale replacement of the underlying navigation logic.

Industrial arms in fixed manufacturing cells sit between these extremes: motion planning and control are typically fully classical and have been for decades (a welding or pick-and-place arm does not need a VLA to repeat a taught trajectory), but perception layers — defect detection, part identification — increasingly use lightweight vision models where a fixed-position camera and a constrained task make a smaller, purpose-trained model sufficient without a full VLA.

Agricultural machinery is the deployment class most often overlooked in "AI robotics" coverage, despite having shipped commercial autonomy — GPS-guided steering, obstacle detection on tractors and harvesters — at scale for years, well before the current VLA wave. What is being added on top now is the same pattern as AMRs: a language/vision interface layer (crop or weed identification, natural-language operator commands) on top of navigation and control systems that were already autonomous and already classical.

Three hardware categories cover most of an on-robot inference build: a Jetson-class dev kit for the reasoning tier, one or more depth/stereo cameras for perception input, and a robot development platform to mount everything on. This section names categories, not specific SKUs, prices, or vendor claims — verify current specs and pricing directly with each vendor before purchasing, since both change faster than an article can track.

Jetson Orin and Jetson Thor development kits are the standard starting point for evaluating an onboard VLA or VLM — see Edge AI Hardware for Local LLMs for the tier-by-tier breakdown (Orin Nano through AGX Thor) covering power envelope and relative compute.

Depth or stereo cameras are the standard perception input for a VLA — most published VLA research and vendor reference designs assume RGB-D or stereo input rather than monocular RGB alone, since depth simplifies the grasp-point and obstacle-distance estimation the action-output layer depends on.

A robot development platform — a wheeled or arm-based reference platform rather than a from-scratch mechanical build — is the practical starting point for a team validating a VLA integration before committing to custom hardware.

Four workloads stay off the robot even in an aggressively local-first architecture: fleet learning, model updates, cross-robot context sharing, and heavy simulation. None of these are latency-sensitive in the way perception or task planning are, and all of them benefit from compute and data no single robot has access to.

Fleet learning — aggregating experience across many deployed robots to improve a shared policy or model — inherently needs data from more units than any single robot carries, and the training compute involved is far beyond an onboard accelerator's budget or purpose.

Model updates (a new VLA checkpoint, a fine-tuned perception model) are pushed to the robot rather than trained on it; the robot is an inference target, not a training node, in essentially every production deployment pattern in use today.

Cross-robot context — one robot benefiting from what another robot in the same fleet just observed — requires a shared backend, since there is no useful onboard mechanism for one robot to directly access another's sensor history.

Heavy simulation — the large-scale physics and rendering workloads used to generate training data or validate a policy before deployment — runs on data-center-class compute for the same reason fleet learning does: the compute budget has no onboard equivalent.

In the EU, an AI component that functions as a safety component of machinery is treated as high-risk under the EU AI Act, and the EU Machinery Regulation (2023/1230) — which applies from January 2027 — separately requires conformity assessment for machinery with an AI-driven safety function. A robot with a model in the path of a safety function may need to satisfy both frameworks, not just one.

Here is our own reasoned analysis, not published regulatory guidance: the two-tier architecture described throughout this article — keeping safety functions (emergency stops, torque limits, collision interlocks) in deterministic, classical control code, and confining the VLA/LLM reasoning tier to advisory or task-level roles that never directly gate a safety function — plausibly reduces the conformity assessment burden, because a certifiable safety function needs behavior that can be exhaustively characterized ahead of time, which deterministic code can offer and a large neural network currently cannot. We are not aware of a regulator having published specific guidance confirming this framing; treat it as an engineering-and-compliance argument to evaluate with your own legal counsel, not as a compliance guarantee.

This is not legal advice. Conformity assessment scope, applicable harmonized standards, and the specific classification of a given robot's safety functions depend on the actual system design and the jurisdiction of sale — consult qualified legal and regulatory counsel before making a compliance determination for a specific product.