Skip to main content
PromptQuorumBuilt for humans. Structured for AI.
Home/Power Local LLM/LMDeploy Review: High-Throughput LLM Serving and Quantization
Overview & Reference

LMDeploy Review: High-Throughput LLM Serving and Quantization

·10 min read·By Hans Kuepper · Founder of PromptQuorum · Discovery engine for open-weight & open-source AI
Read in:
🇺🇸en🇩🇪de🇫🇷fr🇯🇵ja🇨🇳zh🇪🇸es🇧🇷pt🇸🇦ar🇰🇷ko

LMDeploy is a free, open-source (Apache 2.0) command-line toolkit and Python library for compressing, quantizing, and serving large language models on NVIDIA GPUs, installed via pip install lmdeploy. Developed by the MMRazor and MMDeploy teams within the InternLM/OpenMMLab ecosystem, it ships two inference engines — TurboMind (a C++/CUDA engine optimized for throughput) and a pure-Python PyTorch engine — plus AWQ and KV-cache quantization, an OpenAI-compatible API server, and support for well over 50 open-weight LLM and VLM families.

LMDeploy (github.com/InternLM/lmdeploy) is a free, open-source (Apache 2.0) toolkit for compressing, quantizing, and serving large language models, developed by the MMRazor and MMDeploy teams inside the InternLM/OpenMMLab ecosystem. It installs via pip install lmdeploy and ships two inference engines — TurboMind and a pure-Python PyTorch engine — plus an OpenAI-compatible API server, with over 8,000 GitHub stars. This review is the companion to LMDeploy's entry in the Local LLM Software Directory and covers what it actually does, how to install it, and who it fits.

Key Takeaways

  • LMDeploy (github.com/InternLM/lmdeploy) is a free, open-source, pip-installable inference and quantization toolkit — a CLI/library, not a GUI app
  • Developed by the MMRazor and MMDeploy teams within the InternLM/OpenMMLab ecosystem (Shanghai AI Laboratory); repository created June 15, 2023
  • Apache 2.0 licensed, confirmed via the GitHub repository's LICENSE file
  • Ships two inference engines: TurboMind (C++/CUDA, optimized for throughput) and a pure-Python PyTorch engine
  • Supports AWQ 4-bit weight-only quantization and int8/int4 KV cache quantization
  • Over 8,000 GitHub stars and 750 forks as of this review; latest tagged release v0.17.0, published September 1, 2026

📍 In One Sentence

LMDeploy is a free, open-source (Apache 2.0) toolkit for compressing, quantizing, and serving LLMs on NVIDIA GPUs, developed by the MMRazor and MMDeploy teams in the InternLM/OpenMMLab ecosystem, installed via pip install lmdeploy, with over 8,000 GitHub stars.

💬 In Plain Terms

LMDeploy is a command-line tool and Python library you install with pip that takes an LLM you already have (from HuggingFace, for example) and runs it fast on your own GPU, optionally shrinking it first with 4-bit quantization to use less memory. It is not a chat app with a window you click on — it is infrastructure you run on a server, and other applications talk to it through an OpenAI-compatible API.

📌Note: This review is based on LMDeploy's own GitHub repository, README, documentation, and PyPI listing. Throughput and quantization-speed figures (such as "1.8x faster than vLLM") are LMDeploy's own self-reported benchmarks, not independent measurements by PromptQuorum — verify against current published benchmarks before relying on a specific number for a purchasing or architecture decision.

What Is LMDeploy?

LMDeploy is a toolkit for compressing, deploying, and serving large language models, built to run inference on NVIDIA GPUs at high throughput. Its own GitHub description states it was developed by the MMRazor and MMDeploy teams — both OpenMMLab projects, the open-source computer-vision and model-deployment ecosystem maintained under Shanghai AI Laboratory, the same organization behind the InternLM model family.

  • Product type: a command-line tool and Python library — no graphical interface; used from a terminal, a Python script, or a Docker container
  • Developer: the MMRazor and MMDeploy teams, part of the InternLM/OpenMMLab ecosystem under Shanghai AI Laboratory
  • Repository: InternLM/lmdeploy on GitHub, created June 15, 2023
  • License: Apache 2.0, confirmed via the repository's LICENSE file
  • Scale: over 8,000 GitHub stars, 750 forks, and 596 open issues as of this review
  • Distribution: published on PyPI as the lmdeploy package, plus official Docker images per the project's documentation

Project History and Version Milestones

LMDeploy's GitHub repository was created in June 2023, and the project has shipped continuously since, expanding from a single TurboMind inference engine into a two-engine toolkit with quantization, multi-modal (VLM) support, and distributed serving.

  1. 1
    2023/08 — 4-bit AWQ quantization and HuggingFace Hub launch
    Why it matters: Added AWQ-based 4-bit weight-only quantization and published ready-to-use 4-bit models on HuggingFace, per the official changelog.
  2. 2
    2024/01 — PyTorch inference engine introduced
    Why it matters: Added a second, pure-Python inference engine alongside TurboMind, lowering the barrier for developers to extend or debug the serving stack.
  3. 3
    2024/06–2024/07 — VLM and function-calling support
    Why it matters: Added multi-modal (vision-language model) inference pipelines and serving, plus tool/function calling for Llama 3.1 and InternLM2.5.
  4. 4
    2025/01 — DeepSeek-V3 and R1 support
    Why it matters: Added day-one-adjacent support for the DeepSeek-V3 and R1 model families, a widely used reasoning-model line.
  5. 5
    2025/06 — DeepSeek PD disaggregation
    Why it matters: Integrated prefill/decode disaggregation for DeepSeek models via DLSlime and Mooncake, a production technique for scaling large MoE model serving across machines.
  6. 6
    2025/09 — MXFP4 support on NVIDIA GPUs (V100 and newer)
    Why it matters: Added MXFP4 quantized inference, which LMDeploy's own changelog reports at 1.5x the throughput of vLLM on H800 GPUs for OpenAI gpt-oss models.
  7. 7
    2026/02 — Qwen3.5 support and llm-compressor integration
    Why it matters: Added support for the Qwen3.5 model collection and integrated vllm-project/llm-compressor for 4-bit symmetric/asymmetric quantization.
  8. 8
    v0.17.0 — September 1, 2026
    Why it matters: The latest tagged GitHub release as of this review — check the [GitHub releases page](https://github.com/InternLM/lmdeploy/releases) directly for anything shipped after this review's publish date.

What Can You Do With LMDeploy?

LMDeploy's feature set centers on three jobs: shrinking a model with quantization, running it efficiently, and exposing it as a network service. Here is what each part actually does, per LMDeploy's own README and documentation.

  • Two inference engines — TurboMind, a C++/CUDA engine that LMDeploy positions as its highest-throughput option, and a pure-Python PyTorch engine that is easier to extend and add new model architectures to; pick per model based on LMDeploy's own supported-models table
  • Continuous batching and paged attention — persistent (continuous) batching, blocked/paged KV cache, and dynamic split-and-fuse, the same class of techniques used by other production inference engines to raise GPU utilization under concurrent requests
  • Quantization — AWQ 4-bit weight-only quantization, plus int8/int4 KV cache quantization that can be combined with AWQ at the same time, and (since 2026/02) 4-bit symmetric/asymmetric quantization via an llm-compressor integration
  • Broad model support — dozens of LLM families including Llama, Llama2/3/3.1/3.2, InternLM2/3, Qwen1.5/2/2.5/3, Qwen3-MoE, Qwen3-Next, DeepSeek-MoE/V2/V3/R1, Mistral, Mixtral, ChatGLM2, GLM-4, Yi, Baichuan2, and Code Llama, per LMDeploy's supported-models documentation
  • Vision-language model (VLM) support — offline inference pipelines and API serving for multi-modal models such as InternVL, LLaVA, MiniGemini, and CogVLM2
  • OpenAI-compatible API server — lmdeploy serve api_server starts a server that mirrors the OpenAI chat-completions request/response format, documented separately for LLMs and VLMs
  • Offline batch inference pipeline — a Python lmdeploy.pipeline() API for running inference directly inside a script, without standing up a server
  • Multi-model, multi-machine proxy server — a request-distribution service for running several models across multiple machines and GPUs behind one entry point
  • Hardware support beyond NVIDIA CUDA GPUs — Huawei Ascend NPU support via the PyTorch engine, and Windows support (tensor-parallel degree 1) via TurboMind

Usage Examples: Three Ways to Use LMDeploy

These are concrete workflows built from LMDeploy's documented commands above — not hypothetical use cases.

Run offline batch inference from a Python script

After pip install lmdeploy, run a model directly inside Python with the pipeline API: `import lmdeploy; pipe = lmdeploy.pipeline("internlm/internlm3-8b-instruct"); print(pipe(["Hi, please introduce yourself"]))`. LMDeploy downloads the model from HuggingFace by default (or ModelScope/openMind Hub if you set the relevant environment variable) and runs it through whichever engine fits that model.

Start an OpenAI-compatible API server for other apps to use

Run lmdeploy serve api_server <model-path-or-hf-id> to start a local server that speaks the OpenAI chat-completions format. Point any OpenAI-client library or app at that address instead of api.openai.com, letting you reuse existing OpenAI-integration code against a self-hosted model.

Quantize a model to 4-bit AWQ before serving it

Use LMDeploy's AWQ quantization workflow (documented at docs/en/quantization/w4a16.md) to produce a 4-bit weight-only version of a supported model, then serve that quantized model through TurboMind — LMDeploy's own figures put 4-bit inference at up to 2.4x the throughput of FP16 on the same hardware, letting you fit a larger model or serve more concurrent requests on the same GPU.

Pricing and Licensing

LMDeploy is free and open source, with no paid tier. The GitHub repository's LICENSE file applies the Apache License, Version 2.0, without modification, and there is no pricing page anywhere in the project's documentation.

  • No subscription, no paid tier, no usage limits imposed by LMDeploy itself
  • No account or sign-up required to install or use the toolkit
  • Your actual cost is whatever GPU hardware or cloud GPU instance you run LMDeploy on — LMDeploy itself adds no fee on top
  • Apache 2.0 is a permissive license: no copyleft obligation to release your own code, and commercial/production use is explicitly permitted

LMDeploy vs. vLLM

LMDeploy and vLLM are two of the most-used open-source LLM inference engines, and LMDeploy's own marketing explicitly benchmarks itself against vLLM. Both are free, Apache 2.0-adjacent (vLLM is also Apache 2.0), Python-installable, and support an OpenAI-compatible API server — the differences are mostly in engine architecture, ecosystem size, and model-family focus.

Core engines

LMDeploy:
TurboMind (C++/CUDA) plus a PyTorch engine
vLLM:
A single PagedAttention-based engine

Throughput claim

LMDeploy:
Up to 1.8x vLLM (self-reported)
vLLM:
Widely cited as an industry-standard baseline

Quantization

LMDeploy:
AWQ, int8/int4 KV cache, MXFP4
vLLM:
AWQ, GPTQ, FP8, and other formats

Ecosystem size

LMDeploy:
~8,000 GitHub stars
vLLM:
A larger community and broader third-party integration surface

Model-family strength

LMDeploy:
Strong first-class support for InternLM and Qwen
vLLM:
Very broad, frequently first to support new releases across many labs

Developer

LMDeploy:
MMRazor/MMDeploy teams (Shanghai AI Laboratory)
vLLM:
UC Berkeley-originated, now a broad multi-company open-source project

LMDeploy's throughput claims are its own self-published figures, not an independent PromptQuorum benchmark — run your own comparison on your target model, GPU, and traffic pattern before choosing one over the other for production. See the vLLM explainer for more on vLLM specifically.

Who Should Use LMDeploy?

LMDeploy fits teams deploying LLMs on their own NVIDIA GPU infrastructure who want a quantization-aware, high-throughput serving stack rather than a desktop chat app.

Use LMDeploy if you

  • Are deploying LLM inference on NVIDIA GPU hardware you control, in a data center or cloud GPU instance
  • Want built-in AWQ and KV-cache quantization to fit larger models or more concurrent requests on the same GPU memory
  • Are running InternLM or Qwen model families, where LMDeploy has particularly strong, first-party support
  • Want an OpenAI-compatible API server you can drop behind existing OpenAI-client code
  • Need multi-modal (VLM) serving alongside text-only LLM serving in the same toolkit
  • Are comfortable working from a terminal, Python script, or Docker container rather than a graphical app

Look elsewhere if you

  • Want a desktop chat application with a window and a model picker — LMDeploy has no GUI; see the Local LLM Software Directory for desktop apps instead
  • Need CPU-only or Apple Silicon inference — LMDeploy's TurboMind engine targets NVIDIA CUDA GPUs; consider llama.cpp or MLX-LM for those cases
  • Want the single largest ecosystem and broadest third-party tooling around an inference engine — vLLM has a larger community and integration surface
  • Need enterprise support contracts or an SLA — LMDeploy's support channels are GitHub Issues, Discord, and WeChat, typical of a research-lab open-source project rather than a commercial vendor

Competitors and Alternatives

LMDeploy sits in the production LLM-serving-engine segment alongside vLLM, TensorRT-LLM, SGLang, and ExLlamaV2 — tools built to run models at scale on dedicated GPU hardware, distinct from consumer-facing desktop apps.

vLLM

Best known for:
The most widely adopted open-source inference engine, PagedAttention, broad model-day-one support
Link:
vLLM explained
Articles about vLLM (10)

+82 more not shown

ExLlamaV2

Best known for:
A quantization-focused engine popular for running GPTQ/EXL2 models on consumer GPUs
Link:
ExLlamaV2 explained
Articles about ExLlamaV2 (3)

Also mentioned in:

This list reflects tools commonly compared to LMDeploy in the production-serving segment, not an independent PromptQuorum ranking — verify each tool's current feature set and hardware requirements before choosing.

Common Mistakes When Evaluating LMDeploy

Most confusion about LMDeploy comes from treating it like a desktop chat app, or citing its self-reported benchmark numbers as independently verified.

Mistake 1: Expecting a graphical chat interface

LMDeploy has no GUI. It is a command-line toolkit and Python library used to run inference servers and batch pipelines — for a chat window, see desktop apps like Jan or LM Studio instead, which often use different engines underneath.

Mistake 2: Citing "1.8x faster than vLLM" as an independently verified fact

That figure, and similar throughput claims in LMDeploy's README, are the project's own self-published benchmarks, not a third-party or PromptQuorum measurement. Run a benchmark on your own model, GPU, and traffic pattern before relying on it for a capacity-planning decision.

Mistake 3: Assuming LMDeploy runs on any GPU or on CPU only

LMDeploy's primary TurboMind engine targets NVIDIA CUDA GPUs. Huawei Ascend NPU support exists via the PyTorch engine, but there is no CPU-only or Apple Silicon path — check the supported-models and supported-hardware documentation for your specific setup.

Mistake 4: Assuming every model is supported equally well

LMDeploy documents which models each engine (TurboMind vs. PyTorch) supports, and support depth varies by model family — InternLM and Qwen models tend to get first-class treatment given LMDeploy's origin. Check the supported-models table for your specific model before committing to it.

Frequently Asked Questions

What is LMDeploy?

LMDeploy (github.com/InternLM/lmdeploy) is a free, open-source (Apache 2.0) toolkit for compressing, quantizing, and serving large language models on NVIDIA GPUs, developed by the MMRazor and MMDeploy teams within the InternLM/OpenMMLab ecosystem.

Is LMDeploy free?

Yes. LMDeploy is Apache 2.0 licensed with no pricing page and no paid tier. Your actual cost is whatever GPU hardware or cloud instance you run it on.

How do I install LMDeploy?

Run pip install lmdeploy inside a Python 3.10–3.13 environment (a conda environment is recommended). Since v0.13.0, the default PyPI wheels target CUDA 12.8, so this is typically sufficient without a separate CUDA install.

Does LMDeploy have a graphical interface?

No. LMDeploy is a command-line toolkit and Python library. It has no chat window — you interact with it via the terminal, a Python script, or its OpenAI-compatible API server.

Is LMDeploy faster than vLLM?

LMDeploy's own README reports up to 1.8x higher request throughput than vLLM, based on its own self-published benchmarks. This is not an independently verified figure — run your own benchmark on your target model and hardware before relying on it.

What quantization does LMDeploy support?

AWQ 4-bit weight-only quantization, int8/int4 KV cache quantization (which can be combined with AWQ), MXFP4 on supported NVIDIA GPUs, and, since 2026/02, 4-bit symmetric/asymmetric quantization via an integration with vllm-project/llm-compressor.

What models does LMDeploy support?

Dozens of LLM families — including Llama, InternLM2/3, Qwen1.5 through Qwen3, DeepSeek-MoE/V2/V3/R1, Mistral, ChatGLM2, GLM-4, and Code Llama — plus vision-language models like InternVL, LLaVA, and CogVLM2. See LMDeploy's own supported-models documentation for the full, current list.

Does LMDeploy support GPUs other than NVIDIA?

Its primary TurboMind engine targets NVIDIA CUDA GPUs. The PyTorch engine adds support for Huawei Ascend NPUs. This review found no CPU-only or Apple Silicon support path.

Who develops LMDeploy?

The MMRazor and MMDeploy teams, part of the InternLM/OpenMMLab open-source ecosystem maintained under Shanghai AI Laboratory.

Does LMDeploy have an OpenAI-compatible API server?

Yes. Running lmdeploy serve api_server starts a local server that mirrors the OpenAI chat-completions request/response format, so existing OpenAI-client code can point at it instead of a cloud endpoint.

Sources

← Back to Power Local LLM