How to Run Qwen 3 Locally in 2026: Ollama + LM Studio Setup Guide

Why Run Qwen Locally in 2026

Choose Your Qwen Model

Hardware Requirements

• Minimum (Qwen 3.6 27B): GPU with 16 GB VRAM — RTX 4080, RTX 4070 Ti Super, or RTX 3090
• Recommended GPU: RTX 4090 (24 GB VRAM) — runs Q4_K_M at 35 tokens/sec with 8 GB headroom
• Apple Silicon M3/M4 (previous generation): M3 Max or M4 Pro with 48 GB unified memory — still capable, silent, power-efficient, 40+ tokens/sec via MLX
• Mac mini M6 ($899, 32 GB max): entry-level current-generation mini — not enough unified memory for Qwen 3.6 27B Q4_K_M; better suited to the 14B or 7B sizes
• Mac mini M5 Pro ($1,699, 64 GB max): replaces the previous 48 GB M4 Pro mini as the current-generation compact option, with more unified-memory headroom for Qwen 3.6 27B Q4_K_M
• Mac Studio M5 Max ($2,499, 128 GB max): runs Qwen 3 72B Q4_K_M comfortably with headroom
• Mac Studio M5 Ultra ($5,499 starting, 96 GB, configurable up to 256 GB — a 512 GB configuration ships late October 2026): for the largest Qwen 3 configurations or running multiple models concurrently
• Availability: the new Mac mini and Mac Studio lineup was announced August 25, 2026 and ships September 22, 2026; PromptQuorum has not independently benchmarked these chips yet
• RAM: 32 GB system RAM minimum alongside GPU inference; 64 GB recommended alongside a full dev environment
• Storage: 20 GB free disk space for Qwen 3.6 27B Q4_K_M (GGUF file ~17 GB)

Setup with Ollama

1. 1
   Install Ollama
   Why it matters: Ollama handles model downloads, GGUF format, and provides an OpenAI-compatible local API.
2. 2
   Pull the Qwen 3.6 27B model with explicit tag
   Why it matters: Use qwen3.6:27b explicitly. The bare `qwen3` tag defaults to 8B — not the 27B model this guide targets.
3. 3
   Create a Modelfile with correct context length
   Why it matters: The runtime default (VRAM-scaled) tokens is too small for real-world coding tasks. 32768 tokens handles most files and conversations.
4. 4
   Build the custom model and run it
   Why it matters: Creates a Qwen 3.6 27B instance with the extended context window. Verify with a test prompt.
5. 5
   Test the API endpoint
   Why it matters: Ollama exposes an OpenAI-compatible API at localhost:11434/v1. Use this endpoint to connect LLM clients, IDEs, and PromptQuorum.

# Step 1 — Install Ollama
# macOS
brew install ollama

# Linux
curl -fsSL https://ollama.com/install.sh | sh

# Windows — download from https://ollama.com/download

# Step 2 — Pull Qwen 3.6 27B (explicit tag required)
ollama pull qwen3.6:27b
# Downloads Qwen 3.6 27B Q4_K_M (~17 GB)
# Note: 'ollama pull qwen3' without a tag downloads the 8B model

# Step 3 — Create Modelfile with correct num_ctx
cat > Modelfile <<'EOF'
FROM qwen3.6:27b
PARAMETER num_ctx 32768
PARAMETER temperature 0.7
EOF

# Step 4 — Build and run
ollama create qwen3-32k -f Modelfile
ollama run qwen3-32k

# Expected output (Qwen working correctly):
# >>> Write a Python function to reverse a string.
# def reverse_string(s: str) -> str:
#     return s[::-1]
#
# This function takes a string s as input and returns the reversed
# string using Python slice notation with step -1.

# Step 5 — Test API
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-32k",
    "messages": [{"role": "user", "content": "Write a Python function to reverse a string."}]
  }'

Setup with LM Studio

1. 1
   Download and install LM Studio
   Why it matters: Free, cross-platform GUI for local LLM inference. No CLI required.
2. 2
   Search for and download Qwen 3 27B
   Why it matters: LM Studio's model browser searches Hugging Face. Search "Qwen 3 27B" and select the Q4_K_M GGUF variant for 16 GB VRAM.
3. 3
   Configure context length in LM Studio settings
   Why it matters: Same num_ctx issue as Ollama — change Context Length to 32768 in the model parameters before loading.
4. 4
   Start the local server
   Why it matters: LM Studio's "Start Server" creates an OpenAI-compatible API at localhost:1234. Use this URL in clients and PromptQuorum.
5. 5
   Install Claude Code (optional)
   Why it matters: Claude Code is Anthropic's CLI for running Claude locally. Download from [claude.com/claude-code](https://claude.com/claude-code) (all platforms: macOS, Windows, Linux).
6. 6
   Install the Claude Code Proxy
   Why it matters: The free Claude Code Proxy (OpenClaw-based) bridges Claude Code to local LLMs. Run: `uv run python -m uvicorn server:app --host 0.0.0.0 --port 8082`. On Windows, launch with: `uv run python -m uvicorn server:app --host 0.0.0.0 --port 8082`.
7. 7
   Configure Claude Code to use local Qwen
   Why it matters: In Claude Code settings, set API endpoint to http://localhost:8082. Claude Code will route requests through the proxy to your LM Studio instance (localhost:1234), letting you use Qwen 3.6 27B as your coding assistant.

// LM Studio local server config (exported JSON)
{
  "model": "qwen3.6-27b-q4_k_m",
  "server": {
    "host": "localhost",
    "port": 1234,
    "cors": true
  },
  "inference": {
    "context_length": 32768,
    "temperature": 0.7,
    "gpu_layers": -1
  }
}

Connecting to PromptQuorum

# PromptQuorum dispatch config — local Qwen via Ollama
# Set in your .env or PromptQuorum settings panel

OLLAMA_BASE_URL=http://localhost:11434/v1
LOCAL_LLM_MODEL=qwen3.6:27b

# Example routing rules (PromptQuorum dispatch):
# - task_type: code       → model: qwen3.6:27b  (local Ollama, GDPR-safe)
# - task_type: analysis   → model: claude-sonnet-4-6 (Anthropic API, separate config)
# - task_type: private    → model: qwen3.6:27b  (local Ollama, no cloud egress)

Troubleshooting

• Model response is cut off mid-sentence: num_ctx is too low. Rebuild your Modelfile with `PARAMETER num_ctx 32768` and recreate the model with `ollama create`.
• CUDA out of memory error: The model does not fit your VRAM. Switch to Qwen 3 14B Q4_K_M (~9 GB VRAM) or try a Q3_K_S quantization of 27B.
• Ollama API returns 404: Confirm the model name matches exactly. Run `ollama list` to see available models. Use the exact name shown (e.g., `qwen3-32k`).
• Slow generation (< 5 tokens/sec): GPU layers not fully offloaded. Run `ollama run qwen3-32k` and check that `num_gpu_layers` is maximised. Ensure no other GPU-intensive process is running.
• LM Studio shows "failed to load model": Insufficient VRAM. Reduce Q4_K_M context length to 16384 or switch to Qwen 3 14B.
• PromptQuorum returns authentication error: Set `OLLAMA_BASE_URL=http://localhost:11434/v1` in PromptQuorum's local LLM settings. If a key is required by the form, enter any non-empty string — Ollama does not require API key authentication.
• Ollama uses CPU instead of GPU: On NVIDIA: confirm CUDA drivers are installed (`nvidia-smi` should show the GPU). On Mac: Ollama uses Metal automatically — no configuration needed. If Metal is not active, reinstall Ollama from ollama.com.
• Model download stalls or fails: Large models (Qwen 3.6 27B ~17 GB) time out on slow connections. Run `ollama pull qwen3.6:27b` again — Ollama resumes from where it left off. Alternatively, download the GGUF directly from Hugging Face and use `ollama create` with a local path in the Modelfile FROM clause.

Power Consumption and TCO

Related Reading

• Qwen 3.6 vs Claude Sonnet 5 vs DeepSeek — 2026 Comparison
• Why EU Companies Are Switching to Local Qwen
• Qwen Coder vs DeepSeek vs Mistral — Coding Benchmark 2026
• Local AI Production Deployment: Building a Privacy-First Customer Service Agent — Rob Braxman case study covering hardware trade-offs, memory management, and real-world performance for self-hosted AI agents.
• Browse All Local LLM Guides
• Join the PromptQuorum Waitlist

常见问题