Qwen3 Local Deployment: Complete Production Guide (2026)

Qwen Model Performance by Hardware

Docker API Server Setup — Ollama + Open WebUI + Nginx

1. 1
   Install Docker and Docker Compose
   Why it matters: Containers keep Qwen isolated from your OS — no Python environment conflicts, easy updates.
2. 2
   Create docker-compose.yml with Ollama + Open WebUI services
   Why it matters: The compose file manages GPU passthrough, port mapping, and restart policies in one place.
3. 3
   Set OLLAMA_HOST=0.0.0.0 in the Ollama container environment
   Why it matters: Without this, Ollama only listens on localhost and will not accept API requests from other containers or hosts.
4. 4
   Pull your Qwen model: docker exec ollama ollama pull qwen3:8b
   Why it matters: Models are stored in a Docker volume so they persist across container restarts.
5. 5
   Add Nginx as API gateway with basic auth for public-facing deployments
   Why it matters: Exposing Ollama directly to the internet without auth allows anyone to run inference on your GPU.
6. 6
   Set container restart policy to unless-stopped
   Why it matters: This ensures your Qwen server survives system reboots — critical for always-on mini PC deployments.

version: "3.8"
services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    environment:
      - OLLAMA_HOST=0.0.0.0
      - OLLAMA_KEEP_ALIVE=-1
    volumes:
      - ollama_data:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: unless-stopped
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - open_webui_data:/app/backend/data
    depends_on:
      - ollama

volumes:
  ollama_data:
  open_webui_data:

Multi-GPU Configuration for Qwen2.5-72B

• Ollama automatically splits the model across all available GPUs — set CUDA_VISIBLE_DEVICES=0,1 in the compose environment to target specific cards
• For dual RTX 4090s, both must be in the same PCIe bandwidth tier — a B650 or Z790 board with two PCIe Gen 4 x8 slots is the minimum
• NVLink between two RTX 4090s is not officially supported by NVIDIA on consumer cards but works on RTX 4090 Founders Edition pairs via third-party NVLink bridges — adds ~15% throughput
• vLLM is an alternative inference engine that uses tensor parallelism for more efficient multi-GPU utilization — use vLLM instead of Ollama for sustained 70B inference loads above 100 concurrent requests
• For occasional Qwen2.5-72B use, RunPod A40 48 GB at $0.44/hr is cheaper than a dual-RTX-4090 build ($4,600+ at current EOL used pricing)

# vLLM multi-GPU alternative (better for high-traffic 72B)
docker run --gpus all \
  -p 8000:8000 \
  -e VLLM_WORKER_MULTIPROC_METHOD=spawn \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen2.5-72B-Instruct \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --quantization awq

Production API Configuration

• OLLAMA_KEEP_ALIVE=-1 — prevents the model from unloading after inactivity (default is 5 minutes, fatal for server deployments)
• OLLAMA_NUM_PARALLEL=4 — allows up to 4 concurrent inference requests; raise this if you have multiple VRAM GB headroom
• OLLAMA_MAX_LOADED_MODELS=1 — keep only one model in VRAM at a time on small GPU builds to prevent thrashing
• OLLAMA_FLASH_ATTENTION=1 — enables flash attention for 20–30% speed improvement on NVIDIA Ampere/Ada GPUs (RTX 3060 and newer)
• OLLAMA_GPU_OVERHEAD=512 — reserve 512 MB VRAM for OS and driver overhead; reduces OOM crashes on cards with exactly 8 or 16 GB

Cost Comparison: Self-Hosted vs Alibaba Cloud vs RunPod

Always-On Qwen Server Hardware Recommendations

• Budget (Qwen3 8B CPU inference): Minisforum UM890 Pro — AMD Ryzen 9 8945HS, 32 GB DDR5, 512 GB NVMe. ~$429 new. Qwen3 8B runs via Ollama CPU backend at 3–5 tok/s. Adequate for personal assistants and document summarization. 12W idle, 45W peak. Very quiet. Ships from US/EU warehouse.
• Recommended (Qwen3 14B GPU): AOOSTAR GEM12 Pro OCuLink — supports external GPU via OCuLink port. Pair with an RTX 4060 Ti 16 GB eGPU enclosure (~$340 GPU + $100 enclosure). Total ~$800. Runs Qwen3 14B at 16–18 tok/s. Significantly better than CPU fallback for interactive use.
• Power user (Qwen3 32B): Compact ATX desktop with a used RTX 4090 — the RTX 4090 is EOL (NVIDIA discontinued it in 2026), so every unit is used, at ~$2,000–2,600. Examples: Fractal Node 804 case ($90), RTX 4090 (~$2,300 used), Ryzen 9 7950X (~$600), 64 GB DDR5 (~$180). Total ~$3,200. Runs Qwen3 32B at 10–14 tok/s indefinitely. Qwen 3.6 27B (qwen3.6:27b) fits the same card and is worth benchmarking against Qwen3 32B before committing to a model.

Verdict: Which Deployment for Which Model Size

Related Guides

• Basic Qwen Ollama setup (beginner): /power-local-llm/run-qwen-locally-guide-2026
• GPU buying guide for local LLMs: /power-local-llm/best-gpu-buying-guide-local-llm-2026
• NAS storage for model files: /power-local-llm/best-nas-storage-local-ai-models-2026
• Cloud GPU comparison (Western providers): /power-local-llm/cloud-gpu-rental-guide-2026

Frequently Asked Questions

Update Log

• 2026-05-26: Initial publication. Benchmark data from May 2026 hardware. Prices confirmed against Newegg, Amazon, and GPU market trackers.
• 2026-08-27: RTX 4090 reframed as EOL/used-market only (NVIDIA discontinued the RTX 40-series in 2026); prices updated from ~$1,900 new to ~$2,000–2,600 used across all builds and tables. Added Qwen 3.6 27B as a newer same-tier alternative to Qwen3 32B.
• Next review scheduled: 2027-02-27