Local Vision Models 2026: LLaVA, Llama 3.2 Vision, Qwen2-VL & Ollama Multimodal Setup

Quick Facts

• What VLMs do: Accept image + text input → produce text output. Not image generators — these are image *understanders*.
• Ollama support: All models in this comparison have official or community Ollama integration as of May 2026.
• Smallest usable model: Moondream 2 at 1.9B parameters, ~2 GB VRAM.
• Largest practical local model: Llama 3.2 Vision 90B at ~64 GB unified memory (Apple M-series or multi-GPU).
• Image input format: JPEG, PNG, WebP accepted. Max resolution varies by model (typically 1024×1024 to 4096×4096).
• OCR strength: Qwen2-VL 7B ≈ MiniCPM-V 2.6 > Llama 3.2 Vision 11B > LLaVA 1.6 13B > LLaVA 1.6 7B > Moondream 2.
• Multimodal ≠ fast: Vision models add a vision encoder on top of the LLM — expect ~30–60% slower token generation than a text-only model of the same parameter count.

What Are Vision-Language Models (VLMs)?

• How they differ from image generators: Stable Diffusion, FLUX, and DALL-E 3 are text-to-image generators — they produce images from text prompts. VLMs are image-to-text models — they describe, analyze, and answer questions about images.
• How they differ from OCR-only tools: Traditional OCR (Tesseract, PaddleOCR) extracts text from structured documents using pattern recognition. VLMs understand context — they can describe what a table means, answer questions about a chart, or identify objects in a photograph.
• Why run them locally: Private documents (medical records, legal scans, financial statements), proprietary screenshots (internal dashboards, source code), or any workflow where sending images to cloud APIs raises compliance or confidentiality concerns.
• What they cannot do: Generate images, execute code shown in screenshots, or access the internet. VLMs only produce text output based on what's visible in the image.

Available Local Vision Models — Comparison Table

Real-World Accuracy Test: Invoice Extraction

Multi-Image Support

• MiniCPM-V 2.6 accepts up to 4 images per prompt; Qwen2-VL handles up to 8. LLaVA and Moondream accept only single images per request.
• When multi-image matters: Send all pages of a multi-page PDF for full-document extraction. Compare two product photos side by side. Analyze before/after screenshots in a single prompt.

Setup via Ollama — Step by Step

• Step 1 — Install Ollama: Download from ollama.com for macOS, Linux, or Windows. Installation takes under 2 minutes.
• Step 2 — Pull a vision model: ollama pull llama3.2-vision (11B, ~8 GB download) or ollama pull moondream (1.9B, ~2 GB) for low-VRAM setups.
• Step 3 — Transcribe from CLI: ollama run llama3.2-vision "What is in this image?" --image /path/to/photo.jpg
• Step 4 — Use the HTTP API: POST to http://localhost:11434/api/generate with the image as a base64-encoded string in the images array.
• Step 5 — Python example: Use the requests library with base64 encoding — see code block below.

import base64
import requests

def ask_vision_model(image_path: str, prompt: str, model: str = "llama3.2-vision") -> str:
    with open(image_path, "rb") as f:
        image_b64 = base64.b64encode(f.read()).decode("utf-8")

    response = requests.post(
        "http://localhost:11434/api/generate",
        json={
            "model": model,
            "prompt": prompt,
            "images": [image_b64],
            "stream": False,
        },
    )
    return response.json()["response"]

# Example usage
result = ask_vision_model("invoice.png", "Extract all line items and totals from this invoice.")
print(result)

Use Case 1: Document OCR and Extraction

• What works well: Scanned invoices, PDF screenshots, handwritten notes (printed, not cursive), tables with clear borders, business cards.
• What works less well: Handwritten cursive text, low-DPI scans (< 150 DPI), heavily compressed JPEGs, overlapping text.
• Best models for OCR: MiniCPM-V 2.6 (highest OCR accuracy in 6 GB class), Llama 3.2 Vision 11B (best for mixed document types).
• Prompt engineering for OCR: "Extract all text from this document exactly as written, preserving line breaks." Or: "Return the contents of this invoice as JSON with fields: vendor, date, line_items[], total."
• Vs. traditional OCR: VLMs are slower but more semantic. Tesseract is faster for pure text extraction from clean documents. Use VLMs when you need structured data extraction, not just raw text.

Use Case 2: Image Q&A and Description

• Scene description: "What is in this photo?" — objects, people, activities, setting, mood.
• Product cataloguing: Feed product photos with a prompt template like "Describe this product: color, shape, material, condition." Useful for e-commerce inventory without cloud APIs.
• Accessibility: Generate alt text for images at scale. Accurate enough for accessibility compliance on standard product photos.
• Best models: LLaVA 1.6 13B or Llama 3.2 Vision 11B for general photo Q&A. LLaVA 1.6 7B for volume processing where speed matters more than accuracy.
• Speed consideration: On a 6 GB GPU, Llama 3.2 Vision 11B Q4 generates ~8–12 tokens/second for image responses — processing 100 images takes ~2–5 minutes depending on description length.

Use Case 3: Screenshot and UI Analysis

• Developer workflows: Pipe screenshots of error messages to the model: "What is wrong in this screenshot and how would you fix it?"
• Bug report generation: Automatically generate bug report descriptions from screenshots with a structured prompt.
• Dashboard monitoring: Analyze screenshots of monitoring dashboards for anomalies — "Are there any warning-level metrics in this Grafana screenshot?"
• Accessibility testing: Compare screenshots before and after UI changes to verify visual accessibility properties.
• Best models: InternVL 2.5 8B (best UI understanding), MiniCPM-V 2.6 (second best, with Ollama support).

Use Case 4: Chart and Graph Reading

• What works: Reading axis labels, identifying trends, comparing relative bar heights, reading table values in clear fonts.
• What is unreliable: Precise numeric extraction from continuous charts (e.g., "the Q3 bar is 43.7 units"), pie chart percentages without explicit labels.
• Prompt strategy: "Describe the trend shown in this line chart" works better than "What is the exact value at March 2026?"
• Best models for charts: InternVL 2.5 (best chart understanding), Llama 3.2 Vision 11B (good on clearly labeled charts).
• Limitation note: No local VLM in 2026 reliably extracts precise numbers from visually complex charts. For structured chart data, extract via the underlying data source if possible.

Use Case 5: Video Frame Analysis

• Frame extraction: Use ffmpeg to extract at 1fps: ffmpeg -i video.mp4 -vf fps=1 frames/frame_%04d.jpg
• Per-frame analysis: Run each frame through the vision model with a consistent prompt (e.g., "Describe what is happening in this frame in one sentence").
• Cross-frame summarization: Collect all frame descriptions and pass them to a text LLM with a summarization prompt.
• Use cases: Security camera review (flag frames with unusual activity), lecture recording analysis (generate slide-by-slide notes), manufacturing quality inspection (flag frames where defects appear).
• Best models for video frames: Llama 3.2 Vision 11B for quality, LLaVA 1.6 7B for speed (higher frame throughput).
• Speed reality: At 1 frame/second extraction and ~1 second per frame inference on an RTX 4070, a 10-minute video takes ~20–30 minutes to process fully.

import base64
import subprocess
import os
import requests

def extract_frames(video_path: str, output_dir: str, fps: int = 1) -> list[str]:
    os.makedirs(output_dir, exist_ok=True)
    subprocess.run([
        "ffmpeg", "-i", video_path,
        "-vf", f"fps={fps}",
        f"{output_dir}/frame_%04d.jpg",
        "-y"
    ], check=True)
    return sorted([
        os.path.join(output_dir, f)
        for f in os.listdir(output_dir)
        if f.endswith(".jpg")
    ])

def analyze_frame(image_path: str, model: str = "llama3.2-vision") -> str:
    with open(image_path, "rb") as f:
        image_b64 = base64.b64encode(f.read()).decode("utf-8")
    response = requests.post(
        "http://localhost:11434/api/generate",
        json={
            "model": model,
            "prompt": "Describe what is happening in this frame in one sentence.",
            "images": [image_b64],
            "stream": False,
        },
    )
    return response.json()["response"]

frames = extract_frames("lecture.mp4", "frames/", fps=1)
descriptions = [analyze_frame(f) for f in frames]
print("\n".join(f"[{i+1}s] {d}" for i, d in enumerate(descriptions)))

VRAM and Performance Reality Check

How Local Vision Models Compare to GPT-4o Vision

• Structured documents (invoices, forms): Local models at 80–90% of GPT-4o quality — good enough for production use on clean, well-formatted documents.
• Complex scenes, ambiguous imagery: Local models at 50–70% of GPT-4o — noticeable quality gap on images with unusual context, lighting, or ambiguous content.
• Handwriting recognition: Local models significantly weaker, especially on cursive. GPT-4o Vision handles handwriting substantially better.
• Chart data extraction: Unreliable on both local models and GPT-4o, but GPT-4o is more accurate on precise numeric values.
• Cost: GPT-4o Vision at $0.01–0.03 per image vs $0 for local. 10,000 images/month = $100–300 saved with local models.
• Privacy: Local models process images on-device — no data leaves the machine. GPT-4o sends images to OpenAI servers.
• Speed: Local models at 10–20 tok/sec vs GPT-4o at 30–80 tok/sec, but local has no network latency for batch processing.

LLaVA Deep Dive

• Architecture: CLIP ViT-L/14 vision encoder + Llama-2 or Mistral text decoder, connected by a simple linear projection layer.
• LLaVA 1.5 vs 1.6: Version 1.6 (released early 2024) added support for higher-resolution input via dynamic patching, significantly improving OCR and chart reading accuracy.
• Training: Instruction-tuned on LLaVA-Instruct-150K — a dataset of visual conversations generated from image captions and object detection annotations.
• Strengths: Broad general knowledge, well-documented, large community, extensive Ollama integration.
• Weaknesses: Weaker OCR than MiniCPM-V 2.6, weaker UI analysis than InternVL 2.5, outperformed overall by Llama 3.2 Vision 11B on quality benchmarks.
• Why still recommended: LLaVA 1.6 has the largest community, the most tutorials, and the most example prompts of any local VLM. If you run into problems, help is easier to find.

Qwen2-VL — Best Multilingual and OCR Performance

• Architecture: Dynamic resolution support up to 4096×4096 — significantly higher than LLaVA 1.6 (672×672) or Llama 3.2 Vision (1120×1120). This enables reading high-DPI scans without downsampling.
• Multilingual OCR: Best-in-class for Chinese, Japanese, Korean, and English OCR. Training data included large-scale multilingual document corpora — a significant advantage over LLaVA and Llama 3.2 Vision for non-English documents.
• 7B vs 72B: The 7B fits in ~6 GB VRAM (Q4) and is competitive with Llama 3.2 Vision 11B on most document tasks. The 72B uses ~48 GB and leads most open-source benchmarks.
• Ollama install: ollama pull qwen2-vl:7b — available directly from the Ollama model library.
• Multi-image support: Accepts up to 8 images per request — highest multi-image capacity of any model in this comparison.
• Model page: Qwen2-VL 7B on Hugging Face

How to Choose Your Vision Model

• Under 4 GB VRAM: Moondream 2 (1.9B) — only viable option at 2 GB. Beyond Moondream 2, PaliGemma 2 (3B) and SmolVLM (2.2B) are viable alternatives. PaliGemma 2 has stronger document understanding than Moondream at a slight VRAM cost (~3 GB). SmolVLM trades quality for extreme efficiency. None are suitable for dense text OCR.
• 6 GB VRAM: MiniCPM-V 2.6 for document OCR and invoice processing. LLaVA 1.6 7B for general photo Q&A where community support matters. Qwen2-VL 7B for multilingual OCR or when the highest OCR accuracy matters.
• 8–16 GB VRAM: Llama 3.2 Vision 11B is the clear recommendation — best overall quality at this VRAM tier, wide Ollama support.
• 16+ GB VRAM: LLaVA 1.6 13B adds more capacity for complex scene understanding vs. the 7B variant. InternVL 2.5 8B if your primary use case is UI or code screenshots.
• 64+ GB unified memory (Apple M-Max/Ultra, multi-GPU): Llama 3.2 Vision 90B for the best local VLM quality available, approaching cloud-tier performance for document tasks. Qwen2-VL 72B is an alternative with top open-source benchmark scores.
• Always verify numbers: Regardless of model, cross-check any numeric values extracted from charts or tables against the source data. Local VLMs hallucinate precise numbers from visual charts.

FAQ

Sources

• LLaVA project page — LLaVA 1.5 and 1.6 model cards, architecture details, and training dataset description.
• Llama 3.2 Vision on Hugging Face — Meta's official model release, model card, and benchmark figures.
• MiniCPM-V 2.6 on Hugging Face — OpenBMB model card, OCR benchmarks, and fine-tuning instructions.
• Moondream on GitHub — Architecture description, inference scripts, and model download.
• InternVL 2.5 on Hugging Face — OpenGVLab model card, benchmark scores on document and UI tasks.
• Ollama documentation — Vision model support, API reference, and model library.
• Qwen2-VL on Hugging Face — Alibaba's Qwen2-VL model card, architecture details, and multilingual OCR benchmarks.
• PaliGemma 2 on Hugging Face — Google's PaliGemma 2 3B model card.
• SmolVLM on Hugging Face — HuggingFace's SmolVLM model card and inference instructions.

Related Reading

• Local Multimodal AI Pipeline 2026 — Combine vision models with STT and TTS into a full local multimodal stack.
• Local Speech-to-Text 2026: Whisper.cpp vs faster-whisper — Add voice input to complement vision output.
• Local TTS and Voice Cloning 2026 — Add voice output to read VLM responses aloud.
• How to Install Ollama — Prerequisite: Ollama setup and configuration.
• Local LLM Hardware Guide 2026 — VRAM and RAM requirements for vision model inference.