Best Local Vision Models 2026: Gemma 4, Qwen3-VL, LLaVA & Ollama Multimodal Setup

Quick Facts

• What VLMs do: Accept image + text input → produce text output. Not image generators — these are image *understanders*.
• Ollama support: Gemma 4, Qwen3.8, Qwen3.5, Qwen3-VL, Llama 3.2 Vision, MiniCPM-V, LLaVA and Moondream are in the official Ollama library; InternVL, PaliGemma 2 and SmolVLM need community builds.
• Smallest usable models: Moondream 2 (1.8B, 1.7 GB download), Qwen3-VL 2B (1.9 GB) and Qwen3.5 0.8B (1.2–1.3 GB).
• Largest practical local models: Qwen3.5 122B (81 GB) on a 128 GB Mac and Llama 3.2 Vision 90B (55 GB) on a 96 GB+ Mac (Apple M-series) or multi-GPU; Qwen3-VL 235B needs 143 GB.
• Image input format: JPEG, PNG, WebP accepted. Max resolution varies by model (typically 1024×1024 to 4096×4096).
• Audio and video: Gemma 4 E2B, E4B and 12B also accept audio; all Gemma 4 sizes and Qwen3.8 process video as frames.
• OCR strength: Qwen3.8 27B ≈ Qwen3-VL 8B ≈ Qwen3.5 9B > Gemma 4 > MiniCPM-V > LLaVA 1.6 13B > LLaVA 1.6 7B > Moondream 2.
• Multimodal ≠ fast: Vision models add a vision encoder on top of the LLM — expect ~30–60% slower token generation than a text-only model of the same parameter count.

What Are Vision-Language Models (VLMs)?

• How they differ from image generators: Stable Diffusion, FLUX, and DALL-E 3 are text-to-image generators — they produce images from text prompts. VLMs are image-to-text models — they describe, analyze, and answer questions about images.
• How they differ from OCR-only tools: Traditional OCR (Tesseract, PaddleOCR) extracts text from structured documents using pattern recognition. VLMs understand context — they can describe what a table means, answer questions about a chart, or identify objects in a photograph.
• Why run them locally: Private documents (medical records, legal scans, financial statements), proprietary screenshots (internal dashboards, source code), or any workflow where sending images to cloud APIs raises compliance or confidentiality concerns.
• What they cannot do: Generate images, execute code shown in screenshots, or access the internet. VLMs only produce text output based on what's visible in the image.

Available Local Vision Models — Comparison Table

Illustrative Invoice Extraction Comparison

Multi-Image Support

• MiniCPM-V accepts up to 4 images per prompt; Qwen3-VL handles up to 8. LLaVA and Moondream accept only single images per request.
• When multi-image matters: Send all pages of a multi-page PDF for full-document extraction. Compare two product photos side by side. Analyze before/after screenshots in a single prompt.

Setup via Ollama — Step by Step

• Step 1 — Install Ollama: Download from ollama.com for macOS, Linux, or Windows. Installation takes under 2 minutes.
• Step 2 — Pull a vision model: ollama pull gemma4:e4b (6.6–9.5 GB download) for 8 GB cards, ollama pull qwen3-vl:8b (6.2 GB) for OCR, or ollama pull moondream (1.8B, 1.7 GB) for low-VRAM setups.
• Step 3 — Run from the CLI: ollama run gemma4:e4b "What is in this image? /path/to/photo.jpg" — put the image path inside the prompt; ollama run has no --image flag.
• Step 4 — Use the HTTP API: POST to http://localhost:11434/api/generate with the image as a base64-encoded string in the images array. Add a JSON schema in the format field to get structured output.
• Step 5 — Python example: Use the requests library with base64 encoding — see code block below.

import base64
import requests

def ask_vision_model(image_path: str, prompt: str, model: str = "gemma4:e4b") -> str:
    with open(image_path, "rb") as f:
        image_b64 = base64.b64encode(f.read()).decode("utf-8")

    response = requests.post(
        "http://localhost:11434/api/generate",
        json={
            "model": model,
            "prompt": prompt,
            "images": [image_b64],
            "stream": False,
        },
    )
    return response.json()["response"]

# Example usage
result = ask_vision_model("invoice.png", "Extract all line items and totals from this invoice.")
print(result)

Use Case 1: Document OCR and Extraction

• What works well: Scanned invoices, PDF screenshots, handwritten notes (printed, not cursive), tables with clear borders, business cards.
• What works less well: Handwritten cursive text, low-DPI scans (< 150 DPI), heavily compressed JPEGs, overlapping text.
• Best models for OCR: Qwen3-VL 8B and Qwen3.5 9B (6–8 GB class), Qwen3.8 27B (18 GB, 91.1 on OmniDocBench 1.5 per its model card), Gemma 4 for mixed document types.
• Prompt engineering for OCR: "Extract all text from this document exactly as written, preserving line breaks." Or: "Return the contents of this invoice as JSON with fields: vendor, date, line_items[], total."
• Vs. traditional OCR: VLMs are slower but more semantic. Tesseract is faster for pure text extraction from clean documents. Use VLMs when you need structured data extraction, not just raw text.

Use Case 2: Image Q&A and Description

• Scene description: "What is in this photo?" — objects, people, activities, setting, mood.
• Product cataloguing: Feed product photos with a prompt template like "Describe this product: color, shape, material, condition." Useful for e-commerce inventory without cloud APIs.
• Accessibility: Generate alt text for images at scale. Accurate enough for accessibility compliance on standard product photos.
• Best models: Gemma 4 12B or 26B A4B for general photo Q&A. Qwen3.5 4B or LLaVA 1.6 7B for volume processing where speed matters more than accuracy.
• Speed consideration: Speed depends on model, GPU and description length — benchmark a 20-image sample on your own hardware before committing to a batch of thousands.

Use Case 3: Screenshot and UI Analysis

• Developer workflows: Pipe screenshots of error messages to the model: "What is wrong in this screenshot and how would you fix it?"
• Bug report generation: Automatically generate bug report descriptions from screenshots with a structured prompt.
• Dashboard monitoring: Analyze screenshots of monitoring dashboards for anomalies — "Are there any warning-level metrics in this Grafana screenshot?"
• Accessibility testing: Compare screenshots before and after UI changes to verify visual accessibility properties.
• Best models: InternVL 2.5 8B (community build, strong on UI), Gemma 4 and Qwen3-VL (official Ollama support, tool calling for agent workflows).

Use Case 4: Chart and Graph Reading

• What works: Reading axis labels, identifying trends, comparing relative bar heights, reading table values in clear fonts.
• What is unreliable: Precise numeric extraction from continuous charts (e.g., "the Q3 bar is 43.7 units"), pie chart percentages without explicit labels.
• Prompt strategy: "Describe the trend shown in this line chart" works better than "What is the exact value at March 2026?"
• Best models for charts: InternVL 2.5 (strong chart understanding), Gemma 4 and Qwen3-VL (good on clearly labeled charts).
• Limitation note: No local VLM in 2026 reliably extracts precise numbers from visually complex charts. For structured chart data, extract via the underlying data source if possible.

Use Case 5: Video Frame Analysis

• Frame extraction: Use ffmpeg to extract at 1fps: ffmpeg -i video.mp4 -vf fps=1 frames/frame_%04d.jpg
• Per-frame analysis: Run each frame through the vision model with a consistent prompt (e.g., "Describe what is happening in this frame in one sentence").
• Cross-frame summarization: Collect all frame descriptions and pass them to a text LLM with a summarization prompt.
• Use cases: Security camera review (flag frames with unusual activity), lecture recording analysis (generate slide-by-slide notes), manufacturing quality inspection (flag frames where defects appear).
• Best models for video frames: Gemma 4 or Qwen3.8 for quality (both accept video as frames natively), Qwen3.5 4B or LLaVA 1.6 7B for speed (higher frame throughput).
• Speed reality: At 1 frame/second extraction and ~1–3 seconds per frame, a 10-minute video (600 frames) takes roughly 10–30 minutes to process, depending on model and GPU.

import base64
import subprocess
import os
import requests

def extract_frames(video_path: str, output_dir: str, fps: int = 1) -> list[str]:
    os.makedirs(output_dir, exist_ok=True)
    subprocess.run([
        "ffmpeg", "-i", video_path,
        "-vf", f"fps={fps}",
        f"{output_dir}/frame_%04d.jpg",
        "-y"
    ], check=True)
    return sorted([
        os.path.join(output_dir, f)
        for f in os.listdir(output_dir)
        if f.endswith(".jpg")
    ])

def analyze_frame(image_path: str, model: str = "gemma4:e4b") -> str:
    with open(image_path, "rb") as f:
        image_b64 = base64.b64encode(f.read()).decode("utf-8")
    response = requests.post(
        "http://localhost:11434/api/generate",
        json={
            "model": model,
            "prompt": "Describe what is happening in this frame in one sentence.",
            "images": [image_b64],
            "stream": False,
        },
    )
    return response.json()["response"]

frames = extract_frames("lecture.mp4", "frames/", fps=1)
descriptions = [analyze_frame(f) for f in frames]
print("\n".join(f"[{i+1}s] {d}" for i, d in enumerate(descriptions)))

VRAM and Performance Reality Check

How Local Vision Models Compare to GPT-5.5 Vision

• Structured documents (invoices, forms): Our editorial assessment, not a measured benchmark: strong local models come close to GPT-5.5 Vision on clean, well-formatted documents — good enough for production use once you validate the output.
• Complex scenes, ambiguous imagery: Local models trail GPT-5.5 Vision noticeably on images with unusual context, lighting, or ambiguous content (editorial assessment, not benchmarked).
• Handwriting recognition: Local models significantly weaker, especially on cursive. GPT-5.5 Vision handles handwriting substantially better.
• Chart data extraction: Unreliable on both local models and GPT-5.5, but GPT-5.5 is more accurate on precise numeric values.
• Cost: GPT-5.5 Vision is billed per request (check current pricing); local models cost $0 per image apart from hardware and electricity.
• Privacy: Local models process images on-device by default, so images are not sent to a cloud API unless you enable cloud features or connected tools. GPT-5.5 sends images to OpenAI servers.
• Speed: Local models generate roughly 8–35 tok/sec on an RTX 4070 depending on model size (indicative estimates, see the table above); cloud speed varies by provider and load, but local processing has no network latency for batch jobs.

LLaVA Deep Dive

• Architecture: CLIP ViT-L/14 vision encoder + Llama-2 or Mistral text decoder, connected by a simple linear projection layer.
• LLaVA 1.5 vs 1.6: Version 1.6 (released early 2024) added support for higher-resolution input via dynamic patching, significantly improving OCR and chart reading accuracy.
• Training: Instruction-tuned on LLaVA-Instruct-150K — a dataset of visual conversations generated from image captions and object detection annotations.
• Strengths: Broad general knowledge, well-documented, large community, extensive Ollama integration.
• Weaknesses: Weaker OCR than Qwen3-VL and MiniCPM-V, weaker UI analysis than InternVL 2.5, outperformed overall by Gemma 4 and the Qwen3 family.
• Why still recommended: LLaVA 1.6 has the largest community, the most tutorials, and the most example prompts of any local VLM. If you run into problems, help is easier to find.
• Full review: LLaVA Review (2026) — PromptQuorum's dedicated review, including its Apache-2.0-plus-base-model license nuance and real Ollama commands.

Qwen3-VL, Qwen3.5 and Qwen3.8 — Best Multilingual and OCR Performance

• Architecture: Dynamic resolution support up to 4096×4096 — significantly higher than LLaVA 1.6 (672×672) or Llama 3.2 Vision (1120×1120). This enables reading high-DPI scans without downsampling.
• Multilingual OCR: Best-in-class for Chinese, Japanese, Korean, and English OCR. Training data included large-scale multilingual document corpora — a significant advantage over LLaVA and Llama 3.2 Vision for non-English documents.
• Which size: Qwen3-VL 8B is a 6.2 GB download and suits 8 GB cards; Qwen3.5 9B is 6.6–7.6 GB; Qwen3.8 27B is 18 GB and needs a 24 GB card; Qwen3-VL 32B is 21 GB. Qwen3-VL has no 72B variant on Ollama.
• Qwen3-VL 4B: a 3.3 GB download (256K context), the midpoint between the 2B (1.9 GB) and 8B (6.2 GB) builds for cards with about 4–6 GB of VRAM.
• Ollama install: ollama pull qwen3-vl:8b, ollama pull qwen3.5:9b or ollama pull qwen3.8 — all in the Ollama model library.
• Multi-image support: Accepts up to 8 images per request — highest multi-image capacity of any model in this comparison.
• Model pages: Qwen3-VL on Ollama, Qwen3.8-27B on Hugging Face

How to Choose Your Vision Model

• Under 4 GB VRAM: Qwen3.5 2B (2.7–3.1 GB), Qwen3-VL 2B (1.9 GB) or Moondream 2 (1.7 GB) are the practical options. PaliGemma 2 (3B) and SmolVLM (2.2B) are community-build alternatives. None are suitable for dense text OCR.
• 6–8 GB VRAM: Gemma 4 E4B for general photo Q&A, Qwen3-VL 8B or Qwen3.5 9B for document OCR and multilingual text, LLaVA 1.6 7B where community support matters.
• 12–16 GB VRAM: Gemma 4 12B (7.7–8.0 GB download) is the clear recommendation — strong vision scores for its size, with audio input and a 256K context window. LLaVA 1.6 13B and InternVL 2.5 8B (UI and code screenshots) also fit.
• 24 GB VRAM: Gemma 4 26B A4B (16–19 GB), Gemma 4 31B (19–20 GB), Qwen3.8 27B (18 GB) or Qwen3-VL 32B (21 GB) — the closest local option to cloud-tier vision quality.
• 96+ GB unified memory (Apple M-Max/Ultra, multi-GPU): Qwen3.5 122B (81 GB, needs a 128 GB Mac) or Llama 3.2 Vision 90B (55 GB) for the largest local models; both need high-memory hardware and are slower per token than the 24 GB-class models.
• Always verify numbers: Regardless of model, cross-check any numeric values extracted from charts or tables against the source data. Local VLMs hallucinate precise numbers from visual charts.

Frequently Asked Questions

Sources

• Gemma 4 model card on Hugging Face — Google's Apache 2.0 license, supported modalities per size, and MMMU Pro results.
• Gemma 4 on Ollama — tags, download sizes and context windows.
• Qwen3.8-27B on Hugging Face — Apache 2.0 license, 262K context, OmniDocBench 1.5 result.
• Qwen3.5 on Ollama and Qwen3-VL on Ollama — sizes and download sizes.
• Llama 3.2 Vision on Ollama and Ollama issue #16547 — listing and the reported mllama load error.
• LLaVA project page — LLaVA 1.5 and 1.6 model cards, architecture details, and training dataset description.
• MiniCPM-V on Ollama and OpenBMB on Hugging Face — model cards and OCR benchmarks.
• Moondream on Ollama and Moondream on GitHub — model download and inference scripts.
• InternVL 2.5 on Hugging Face — OpenGVLab model card, benchmark scores on document and UI tasks.
• Ollama documentation — Vision model support, structured outputs, API reference, and model library.
• PaliGemma 2 on Hugging Face — Google's PaliGemma 2 3B model card.
• SmolVLM on Hugging Face — HuggingFace's SmolVLM model card and inference instructions.
• llama.cpp multimodal documentation — supported vision models and llama-server usage.
• vLLM supported models — multimodal architectures vLLM can serve.
• Gemma 4 tags on Ollama — q4_K_M, q8_0, bf16, QAT and MLX build sizes.

Related Reading

• Which Ollama Models Support Vision? — quick answer on which Ollama library models accept images, with VRAM specs.
• Ollama Vision Models (2026) — a focused, practical guide to running vision models specifically through Ollama: real CLI/API syntax and which models are actually in its library.
• VLMs in Cameras and on Drones 2026 -- Desktop VLM basics are covered here; for embedded camera and drone deployment, see this edge-focused guide.
• Local Multimodal AI Pipeline 2026 — Combine vision models with STT and TTS into a full local multimodal stack.
• Local Speech-to-Text 2026: Whisper.cpp vs faster-whisper — Add voice input to complement vision output.
• Local TTS and Voice Cloning 2026 — Add voice output to read VLM responses aloud.
• How to Install Ollama — Prerequisite: Ollama setup and configuration.
• Local LLM Hardware Guide — VRAM and RAM requirements for vision model inference.