Best Tool for Benchmarking Local LLM Tokens/Sec?
This page contains links to third-party products for reference. PromptQuorum is not enrolled in any affiliate program — these are plain links that earn no commission. Clicking links and your next steps are entirely your own responsibility. These links do not represent any endorsement or verification by PromptQuorum.
Quick Answer
llama-bench, included with llama.cpp, is the best tool for benchmarking local LLM inference speed. It reports prompt-processing and generation speed separately under a fixed context length and quantization level, making results comparable across different hardware.
- ▸llama-bench: the most rigorous option — separates prompt-processing speed from generation speed, controls for context length and quantization.
- ▸Ollama --verbose: fastest way to get a rough tokens/sec number from a normal chat session, less controlled than llama-bench.
- ▸Third-party GUIs (LM Studio's built-in stats) are convenient for a quick read but don't expose the same run-to-run controls.
Updated: July 14, 2026
Key Takeaways
- ✓llama-bench is the most rigorous benchmarking tool — it isolates prompt-processing speed from generation speed under fixed settings
- ✓Ollama's --verbose flag gives a fast, rough tokens/sec estimate from a normal chat run, useful for quick checks
- ✓Always fix context length and quantization level between runs, or the comparison is meaningless
- ✓Run multiple passes and average the result — a single run is affected by thermal throttling and cold-start loading
Best Pick: llama-bench
llama-bench is the right default for anyone who needs a repeatable, hardware-comparable tokens-per-second number. It ships as part of the llama.cpp build, runs from the command line, and reports two separate figures for every test: prompt-processing speed (how fast the model reads the input) and generation speed (how fast it produces new tokens). Those two numbers behave very differently under load, so a tool that conflates them is not giving you the full picture.
Ollama's `--verbose` flag is the practical alternative for a fast sanity check. Running `ollama run
LM Studio and other desktop frontends surface a similar live tokens/sec readout in their chat UI. Use this for a quick eyeball check while you are already using the tool, but do not rely on it for a hardware-purchase decision — it doesn't expose the run-count or context-length controls that make a benchmark result trustworthy.
What to Look For in a Benchmarking Tool
A trustworthy local LLM benchmarking tool holds context length, quantization level, and prompt content constant between runs, and reports prompt-processing speed and token-generation speed as separate numbers. Without those controls, a "tokens/sec" figure from one run tells you almost nothing about how the same setup will perform on a longer prompt or a different quantization.
Use a benchmarking tool if you are comparing GPUs before a purchase, evaluating whether a quantization level is worth the quality trade-off, or reporting performance numbers publicly. Skip the formal benchmark and just watch the live tokens/sec counter if you only want a rough feel for whether a model is usable for chat.
Avoid: single-run numbers reported without a context length or quantization level attached — they cannot be compared to anything. Avoid: tools that report only one combined tokens/sec figure, since prompt-processing and generation speed scale differently with context length and hardware.
Hardware to Pair With Your Benchmark
If you are benchmarking with the goal of deciding whether to upgrade, run the same llama-bench command on your existing GPU and on the candidate GPU using the same model file and quantization, then compare the two prompt-processing and generation numbers side by side.
Disclosure: This article contains affiliate links. We may earn a commission if you purchase through these links, at no extra cost to you.