Skip to main content
PromptQuorum
Home/Power Local LLM/LMDeploy Review: High-Throughput LLM Serving and Quantization
Overview & Reference

LMDeploy Review: High-Throughput LLM Serving and Quantization

Β·10 min readΒ·By Hans Kuepper Β· Founder of PromptQuorum, multi-model AI dispatch tool Β· PromptQuorum
Read in:
πŸ‡ΊπŸ‡ΈenπŸ‡©πŸ‡ͺdeπŸ‡«πŸ‡·frπŸ‡―πŸ‡΅jaπŸ‡¨πŸ‡³zhπŸ‡ͺπŸ‡ΈesπŸ‡§πŸ‡·ptπŸ‡ΈπŸ‡¦arπŸ‡°πŸ‡·ko

LMDeploy is a free, open-source (Apache 2.0) command-line toolkit and Python library for compressing, quantizing, and serving large language models on NVIDIA GPUs, installed via pip install lmdeploy. Developed by the MMRazor and MMDeploy teams within the InternLM/OpenMMLab ecosystem, it ships two inference engines β€” TurboMind (a C++/CUDA engine optimized for throughput) and a pure-Python PyTorch engine β€” plus AWQ and KV-cache quantization, an OpenAI-compatible API server, and support for well over 50 open-weight LLM and VLM families.

LMDeploy (github.com/InternLM/lmdeploy) is a free, open-source (Apache 2.0) toolkit for compressing, quantizing, and serving large language models, developed by the MMRazor and MMDeploy teams inside the InternLM/OpenMMLab ecosystem. It installs via pip install lmdeploy and ships two inference engines β€” TurboMind and a pure-Python PyTorch engine β€” plus an OpenAI-compatible API server, with over 8,000 GitHub stars. This review is the companion to LMDeploy's entry in the Local LLM Software Directory and covers what it actually does, how to install it, and who it fits.

Key Takeaways

  • LMDeploy (github.com/InternLM/lmdeploy) is a free, open-source, pip-installable inference and quantization toolkit β€” a CLI/library, not a GUI app
  • Developed by the MMRazor and MMDeploy teams within the InternLM/OpenMMLab ecosystem (Shanghai AI Laboratory); repository created June 15, 2023
  • Apache 2.0 licensed, confirmed via the GitHub repository's LICENSE file
  • Ships two inference engines: TurboMind (C++/CUDA, optimized for throughput) and a pure-Python PyTorch engine
  • Supports AWQ 4-bit weight-only quantization and int8/int4 KV cache quantization
  • Over 8,000 GitHub stars and 750 forks as of this review; latest tagged release v0.17.0, published September 1, 2026

πŸ“ In One Sentence

LMDeploy is a free, open-source (Apache 2.0) toolkit for compressing, quantizing, and serving LLMs on NVIDIA GPUs, developed by the MMRazor and MMDeploy teams in the InternLM/OpenMMLab ecosystem, installed via pip install lmdeploy, with over 8,000 GitHub stars.

πŸ’¬ In Plain Terms

LMDeploy is a command-line tool and Python library you install with pip that takes an LLM you already have (from HuggingFace, for example) and runs it fast on your own GPU, optionally shrinking it first with 4-bit quantization to use less memory. It is not a chat app with a window you click on β€” it is infrastructure you run on a server, and other applications talk to it through an OpenAI-compatible API.

πŸ“ŒNote: This review is based on LMDeploy's own GitHub repository, README, documentation, and PyPI listing. Throughput and quantization-speed figures (such as "1.8x faster than vLLM") are LMDeploy's own self-reported benchmarks, not independent measurements by PromptQuorum β€” verify against current published benchmarks before relying on a specific number for a purchasing or architecture decision.

What Is LMDeploy?

LMDeploy is a toolkit for compressing, deploying, and serving large language models, built to run inference on NVIDIA GPUs at high throughput. Its own GitHub description states it was developed by the MMRazor and MMDeploy teams β€” both OpenMMLab projects, the open-source computer-vision and model-deployment ecosystem maintained under Shanghai AI Laboratory, the same organization behind the InternLM model family.

  • Product type: a command-line tool and Python library β€” no graphical interface; used from a terminal, a Python script, or a Docker container
  • Developer: the MMRazor and MMDeploy teams, part of the InternLM/OpenMMLab ecosystem under Shanghai AI Laboratory
  • Repository: InternLM/lmdeploy on GitHub, created June 15, 2023
  • License: Apache 2.0, confirmed via the repository's LICENSE file
  • Scale: over 8,000 GitHub stars, 750 forks, and 596 open issues as of this review
  • Distribution: published on PyPI as the lmdeploy package, plus official Docker images per the project's documentation

Project History and Version Milestones

LMDeploy's GitHub repository was created in June 2023, and the project has shipped continuously since, expanding from a single TurboMind inference engine into a two-engine toolkit with quantization, multi-modal (VLM) support, and distributed serving.

  1. 1
    2023/08 β€” 4-bit AWQ quantization and HuggingFace Hub launch
    Why it matters: Added AWQ-based 4-bit weight-only quantization and published ready-to-use 4-bit models on HuggingFace, per the official changelog.
  2. 2
    2024/01 β€” PyTorch inference engine introduced
    Why it matters: Added a second, pure-Python inference engine alongside TurboMind, lowering the barrier for developers to extend or debug the serving stack.
  3. 3
    2024/06–2024/07 β€” VLM and function-calling support
    Why it matters: Added multi-modal (vision-language model) inference pipelines and serving, plus tool/function calling for Llama 3.1 and InternLM2.5.
  4. 4
    2025/01 β€” DeepSeek V3 and R1 support
    Why it matters: Added day-one-adjacent support for the DeepSeek V3 and R1 model families, a widely used reasoning-model line.
  5. 5
    2025/06 β€” DeepSeek PD disaggregation
    Why it matters: Integrated prefill/decode disaggregation for DeepSeek models via DLSlime and Mooncake, a production technique for scaling large MoE model serving across machines.
  6. 6
    2025/09 β€” MXFP4 support on NVIDIA GPUs (V100 and newer)
    Why it matters: Added MXFP4 quantized inference, which LMDeploy's own changelog reports at 1.5x the throughput of vLLM on H800 GPUs for OpenAI gpt-oss models.
  7. 7
    2026/02 β€” Qwen3.5 support and llm-compressor integration
    Why it matters: Added support for the Qwen3.5 model collection and integrated vllm-project/llm-compressor for 4-bit symmetric/asymmetric quantization.
  8. 8
    v0.17.0 β€” September 1, 2026
    Why it matters: The latest tagged GitHub release as of this review β€” check the [GitHub releases page](https://github.com/InternLM/lmdeploy/releases) directly for anything shipped after this review's publish date.

What Can You Do With LMDeploy?

LMDeploy's feature set centers on three jobs: shrinking a model with quantization, running it efficiently, and exposing it as a network service. Here is what each part actually does, per LMDeploy's own README and documentation.

  • Two inference engines β€” TurboMind, a C++/CUDA engine that LMDeploy positions as its highest-throughput option, and a pure-Python PyTorch engine that is easier to extend and add new model architectures to; pick per model based on LMDeploy's own supported-models table
  • Continuous batching and paged attention β€” persistent (continuous) batching, blocked/paged KV cache, and dynamic split-and-fuse, the same class of techniques used by other production inference engines to raise GPU utilization under concurrent requests
  • Quantization β€” AWQ 4-bit weight-only quantization, plus int8/int4 KV cache quantization that can be combined with AWQ at the same time, and (since 2026/02) 4-bit symmetric/asymmetric quantization via an llm-compressor integration
  • Broad model support β€” dozens of LLM families including Llama, Llama2/3/3.1/3.2, InternLM2/3, Qwen1.5/2/2.5/3, Qwen3-MoE, Qwen3-Next, DeepSeek-MoE/V2/V3/R1, Mistral, Mixtral, ChatGLM2, GLM-4, Yi, Baichuan2, and Code Llama, per LMDeploy's supported-models documentation
  • Vision-language model (VLM) support β€” offline inference pipelines and API serving for multi-modal models such as InternVL, LLaVA, MiniGemini, and CogVLM2
  • OpenAI-compatible API server β€” lmdeploy serve api_server starts a server that mirrors the OpenAI chat-completions request/response format, documented separately for LLMs and VLMs
  • Offline batch inference pipeline β€” a Python lmdeploy.pipeline() API for running inference directly inside a script, without standing up a server
  • Multi-model, multi-machine proxy server β€” a request-distribution service for running several models across multiple machines and GPUs behind one entry point
  • Hardware support beyond NVIDIA CUDA GPUs β€” Huawei Ascend NPU support via the PyTorch engine, and Windows support (tensor-parallel degree 1) via TurboMind

Usage Examples: Three Ways to Use LMDeploy

These are concrete workflows built from LMDeploy's documented commands above β€” not hypothetical use cases.

Pricing and Licensing

LMDeploy is free and open source, with no paid tier. The GitHub repository's LICENSE file applies the Apache License, Version 2.0, without modification, and there is no pricing page anywhere in the project's documentation.

  • No subscription, no paid tier, no usage limits imposed by LMDeploy itself
  • No account or sign-up required to install or use the toolkit
  • Your actual cost is whatever GPU hardware or cloud GPU instance you run LMDeploy on β€” LMDeploy itself adds no fee on top
  • Apache 2.0 is a permissive license: no copyleft obligation to release your own code, and commercial/production use is explicitly permitted

LMDeploy vs. vLLM

LMDeploy and vLLM are two of the most-used open-source LLM inference engines, and LMDeploy's own marketing explicitly benchmarks itself against vLLM. Both are free, Apache 2.0-adjacent (vLLM is also Apache 2.0), Python-installable, and support an OpenAI-compatible API server β€” the differences are mostly in engine architecture, ecosystem size, and model-family focus.

Core engines

LMDeploy:
TurboMind (C++/CUDA) plus a PyTorch engine
vLLM:
A single PagedAttention-based engine

Throughput claim

LMDeploy:
Up to 1.8x vLLM (self-reported)
vLLM:
Widely cited as an industry-standard baseline

Quantization

LMDeploy:
AWQ, int8/int4 KV cache, MXFP4
vLLM:
AWQ, GPTQ, FP8, and other formats

Ecosystem size

LMDeploy:
~8,000 GitHub stars
vLLM:
A larger community and broader third-party integration surface

Model-family strength

LMDeploy:
Strong first-class support for InternLM and Qwen
vLLM:
Very broad, frequently first to support new releases across many labs

Developer

LMDeploy:
MMRazor/MMDeploy teams (Shanghai AI Laboratory)
vLLM:
UC Berkeley-originated, now a broad multi-company open-source project

LMDeploy's throughput claims are its own self-published figures, not an independent PromptQuorum benchmark β€” run your own comparison on your target model, GPU, and traffic pattern before choosing one over the other for production. See the vLLM explainer for more on vLLM specifically.

Who Should Use LMDeploy?

LMDeploy fits teams deploying LLMs on their own NVIDIA GPU infrastructure who want a quantization-aware, high-throughput serving stack rather than a desktop chat app.

Competitors and Alternatives

LMDeploy sits in the production LLM-serving-engine segment alongside vLLM, TensorRT-LLM, SGLang, and ExLlamaV2 β€” tools built to run models at scale on dedicated GPU hardware, distinct from consumer-facing desktop apps.

vLLM

Best known for:
The most widely adopted open-source inference engine, PagedAttention, broad model-day-one support
Link:
vLLM explained
Articles about vLLM (10)

+81 more not shown

ExLlamaV2

Best known for:
A quantization-focused engine popular for running GPTQ/EXL2 models on consumer GPUs
Link:
ExLlamaV2 explained
Articles about ExLlamaV2 (2)

Also mentioned in:

This list reflects tools commonly compared to LMDeploy in the production-serving segment, not an independent PromptQuorum ranking β€” verify each tool's current feature set and hardware requirements before choosing.

Common Mistakes When Evaluating LMDeploy

Most confusion about LMDeploy comes from treating it like a desktop chat app, or citing its self-reported benchmark numbers as independently verified.

Frequently Asked Questions

What is LMDeploy?

LMDeploy (github.com/InternLM/lmdeploy) is a free, open-source (Apache 2.0) toolkit for compressing, quantizing, and serving large language models on NVIDIA GPUs, developed by the MMRazor and MMDeploy teams within the InternLM/OpenMMLab ecosystem.

Is LMDeploy free?

Yes. LMDeploy is Apache 2.0 licensed with no pricing page and no paid tier. Your actual cost is whatever GPU hardware or cloud instance you run it on.

How do I install LMDeploy?

Run pip install lmdeploy inside a Python 3.10–3.13 environment (a conda environment is recommended). Since v0.13.0, the default PyPI wheels target CUDA 12.8, so this is typically sufficient without a separate CUDA install.

Does LMDeploy have a graphical interface?

No. LMDeploy is a command-line toolkit and Python library. It has no chat window β€” you interact with it via the terminal, a Python script, or its OpenAI-compatible API server.

Is LMDeploy faster than vLLM?

LMDeploy's own README reports up to 1.8x higher request throughput than vLLM, based on its own self-published benchmarks. This is not an independently verified figure β€” run your own benchmark on your target model and hardware before relying on it.

What quantization does LMDeploy support?

AWQ 4-bit weight-only quantization, int8/int4 KV cache quantization (which can be combined with AWQ), MXFP4 on supported NVIDIA GPUs, and, since 2026/02, 4-bit symmetric/asymmetric quantization via an integration with vllm-project/llm-compressor.

What models does LMDeploy support?

Dozens of LLM families β€” including Llama, InternLM2/3, Qwen1.5 through Qwen3, DeepSeek-MoE/V2/V3/R1, Mistral, ChatGLM2, GLM-4, and Code Llama β€” plus vision-language models like InternVL, LLaVA, and CogVLM2. See LMDeploy's own supported-models documentation for the full, current list.

Does LMDeploy support GPUs other than NVIDIA?

Its primary TurboMind engine targets NVIDIA CUDA GPUs. The PyTorch engine adds support for Huawei Ascend NPUs. This review found no CPU-only or Apple Silicon support path.

Who develops LMDeploy?

The MMRazor and MMDeploy teams, part of the InternLM/OpenMMLab open-source ecosystem maintained under Shanghai AI Laboratory.

Does LMDeploy have an OpenAI-compatible API server?

Yes. Running lmdeploy serve api_server starts a local server that mirrors the OpenAI chat-completions request/response format, so existing OpenAI-client code can point at it instead of a cloud endpoint.

Sources

← Back to Power Local LLM