Local LLM in Your CI/CD: Automated Code Review Without Cloud

Quick Facts

• Recommended model: Qwen3-Coder 30B at Q4_K_M (~17 GB VRAM, Apache 2.0).
• Recommended runtime: Ollama for setup simplicity; vLLM if you need higher concurrency on the same GPU.
• Minimum GPU for serious teams: RTX 4090 (24 GB). Smaller cards force the 7B model and noticeably worse review quality.
• Concurrency on a single 24 GB GPU: comfortably 1–3 simultaneous reviews on Qwen3-Coder 30B; queue beyond that.
• Latency target: under 30 seconds for a 200-line diff. Past that, PR-author behaviour shifts and reviews get bypassed.
• Audit posture: zero outbound egress on the GPU server is provable; entire surface is ollama serve + a single log file.
• Crossover vs GitHub Advanced Security ($19/dev/mo): 15–25 paid seats covers an RTX 4090 build in 5–10 months.
• GitLab CI parity: identical architecture, replace the GitHub Action with a CI job calling the same HTTP endpoint.

Architecture Comparison: Three Real Options for Code Review in CI

The Recommended Stack: Ollama + Qwen3-Coder + a Thin GitHub Action

• GPU server running Ollama (or vLLM for higher concurrency). Ollama exposes an OpenAI-compatible HTTP API on localhost:11434 by default; bind it to a private interface or a reverse proxy with auth before exposing to runners.
• Coding-tuned model: Qwen3-Coder 30B at Q4_K_M is the May 2026 default — strongest open-weight coding direction, 256K context, Apache 2.0 licence, fits on a 24 GB GPU. For 8–16 GB GPUs, use Qwen3-Coder 7B with the understanding that review quality drops noticeably.
• CI integration: a self-hosted GitHub Actions runner on the same network as the GPU server, or your existing GitHub-hosted runners reaching the GPU server over a private network (Tailscale, WireGuard, or a VPC peering).
• Custom GitHub Action (JavaScript or composite) that fetches the PR diff via the GitHub API, POSTs it to the Ollama endpoint with a review prompt, parses the structured response, and posts inline comments back to the PR.
• Optional: a small Redis or SQLite cache keyed on file hash + diff hash to avoid re-reviewing unchanged files in subsequent CI runs.
• GitLab parity: the same architecture, with a GitLab CI job replacing the GitHub Action. The LLM call is identical.

A Working GitHub Actions Workflow

• The runner must be able to reach OLLAMA_HOST over the network — self-hosted on the same VPC, or via Tailscale / WireGuard if the GPU server lives elsewhere.
• The system prompt enforces a structured JSON response so the action can branch on verdict cleanly. Without format: "json" and a strict schema in the prompt, you will spend operational time parsing free-form output.
• The fetch-depth: 0 checkout is necessary to compute a real diff against the base branch — shallow checkouts produce malformed diffs.
• For repositories above ~50K lines of changed code per PR, truncate or chunk the diff before sending. The 256K context on Qwen3-Coder 30B is generous, but practical working context is closer to 64K–128K (see Best Local Coding Models in 2026).
• For prompt-design depth — system vs user prompts, examples, structured outputs — see System Prompt vs User Prompt: What's the Difference.

# .github/workflows/local-llm-review.yml
name: Local LLM Code Review

on:
  pull_request:
    types: [opened, synchronize]

jobs:
  review:
    runs-on: self-hosted   # or any runner that can reach OLLAMA_HOST
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Generate diff
        id: diff
        run: |
          git diff origin/${{ github.base_ref }}...HEAD > /tmp/pr.diff
          echo "size=$(wc -c < /tmp/pr.diff)" >> "$GITHUB_OUTPUT"

      - name: Call local LLM for review
        id: review
        env:
          OLLAMA_HOST: ${{ secrets.OLLAMA_HOST }}   # e.g. http://gpu-server.internal:11434
        run: |
          DIFF=$(jq -Rs . < /tmp/pr.diff)
          curl -sS "$OLLAMA_HOST/api/chat" \
            -H 'Content-Type: application/json' \
            -d "{
              \"model\": \"qwen3-coder:30b\",
              \"stream\": false,
              \"format\": \"json\",
              \"messages\": [
                {\"role\": \"system\", \"content\": \"You are a senior code reviewer. Return JSON: {verdict: 'approve'|'comment'|'block', summary: string, comments: [{path, line, severity, message}]}\"},
                {\"role\": \"user\", \"content\": $DIFF}
              ]
            }" > /tmp/review.json
          echo "verdict=$(jq -r '.message.content | fromjson | .verdict' < /tmp/review.json)" >> "$GITHUB_OUTPUT"

      - name: Post review comment
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const review = JSON.parse(JSON.parse(fs.readFileSync('/tmp/review.json')).message.content);
            const body = `### Local LLM Review: \`${review.verdict}\`\n\n${review.summary}`;
            await github.rest.issues.createComment({
              owner: context.repo.owner,
              repo: context.repo.repo,
              issue_number: context.issue.number,
              body
            });

      - name: Block on critical verdict
        if: steps.review.outputs.verdict == 'block'
        run: exit 1

Hardware Sizing by Team Size

GPU Sharing Across Builds and Other Workloads

• Dedicated GPU for review only: simplest model. Latency is predictable; capacity planning is straightforward; failure modes are isolated. The recommendation for any team that does not already operate GPU infrastructure.
• Shared GPU with ML inference: workable if the inference workload has a steady envelope (e.g. a small embedding service that fits in 4–6 GB). The review model occupies the rest of VRAM. Scheduling collisions are rare on this pattern.
• Shared GPU with ML training: strongly discouraged. Training jobs spike VRAM usage to the limit and starve the review model, causing 30–120-second review latencies that erode developer trust in the system.
• vLLM with paged attention: purpose-built for high-concurrency LLM serving. The same RTX 4090 that handles 1–3 concurrent reviews under Ollama can handle 4–8 under vLLM, at the cost of a more complex setup. Worth it past 25 developers.
• Multi-tenant on H100: at the 100+ developer scale, partition an H100 into MIG slices or run vLLM with per-tenant quotas. This is platform-engineering territory; do not improvise it.

Cost Comparison vs GitHub Advanced Security

• GitHub Advanced Security (Code Security): $19/developer/month at list price (verify on the GitHub pricing page; volume discounts available for enterprise customers).
• Cloud LLM API (e.g. OpenAI, Anthropic): roughly $50–200/month per active developer at typical PR volume; varies wildly by codebase size and review prompt design.
• Self-hosted local LLM, RTX 4090 build: roughly $2,500 one-time hardware (GPU + a basic server chassis). Power: ~50W idle, ~350W under load — call it $20–30/month in electricity at typical usage. No per-seat fees.
• Crossover at 10 devs: GHAS $190/month vs self-hosted $25/month operating + $2,500 capex. Capex pays back in ~14 months.
• Crossover at 25 devs: GHAS $475/month vs self-hosted $25/month operating + $2,500 capex. Capex pays back in ~5–6 months.
• Crossover at 50 devs: GHAS $950/month vs self-hosted $40/month operating + $7,500 capex (48 GB GPU). Capex pays back in ~8 months.
• The capex number is what dominates the math. If you are buying a GPU specifically for this, the payback is real. If you have existing GPU capacity, the marginal cost is closer to zero and self-hosting wins immediately.

Security Model and Audit Posture

• The model only sees the diff your action sends it. No telemetry, no hidden network calls. Confirmable with tcpdump or nft monitor on the GPU server's outbound interface — under steady-state operation, you should see zero outbound packets to non-internal hosts.
• The full audit surface is one process and one log file. ollama serve is the entire LLM stack. Its logs (request bodies, latency, model loading events) are the audit record. No SaaS dashboard to query, no third-party retention policy to read.
• Network isolation is straightforward. Bind ollama serve to a private interface; put a reverse proxy with mTLS or shared-secret auth in front; deny outbound on the GPU server's network namespace except to your CI runner subnet. Standard zero-trust pattern, no LLM-specific magic.
• The model weights are static, signed-by-vendor artefacts. Pull them once via Ollama, pin the digest, and the model cannot change without an operator action. This is a stronger supply-chain story than a SaaS API where the upstream model can be silently swapped.
• Compliance posture: zero data egress is straightforward to document for SOC 2, ISO 27001, GDPR, and the EU AI Act limited-risk classification. The hardest part of self-hosting compliance is usually documenting the inference server itself; Ollama and vLLM are both well documented.
• The model still sees your code. Self-hosted does not mean private from the model — it means private from third parties. Insider-threat scenarios (engineer with GPU server access reads logs containing past PR diffs) are still in scope; rotate logs and limit access accordingly.

Prompt Design for Code Review

• Structured output is non-negotiable. Force JSON with a strict schema (verdict, summary, comments[]). Without it, the action spends 30% of its code parsing free-form output and the failure modes are subtle.
• For the full guide on structured output enforcement across models, see structured output and JSON mode.
• Severity thresholds belong in the prompt, not the action. Tell the model what counts as critical, high, medium, low; tell it to filter low-severity findings unless explicitly asked. This is far more reliable than post-hoc filtering on a free-form severity field.
• Anchor the prompt with examples. A 1–2-shot prompt with a real diff and a real ideal-review JSON dramatically outperforms zero-shot for the same model and the same diff size.
• Distinguish "review" from "comment" intent. A reviewer comment ("consider extracting this to a helper") and a blocker ("this introduces a SQL injection") need different actions in CI. Tag them in the structured output and have the action block only on blockers.
• Per-language prompt variants help past a certain size. A polyglot codebase benefits from a prompt that references the relevant language idioms (Pythonic vs idiomatic Rust). This is optional below ~25 developers; valuable above.
• For deeper prompt-engineering grounding — system vs user prompts, structured outputs, few-shot prompting — see System Prompt vs User Prompt: What's the Difference.

Handling False Positives Without Eroding Developer Trust

• Set a "block" threshold high. A block verdict that fires on every minor lint issue trains developers to bypass the check. Reserve block for security issues, broken tests, and clear correctness failures.
• Make non-blocking comments feel cheap. Inline comments that the model is uncertain about should be tagged ("tentative" / "consider") so authors can dismiss them quickly without ceremony.
• Build a feedback loop in month one. Add a reaction (👍 / 👎) to each review comment. Periodically (weekly works) review the 👎s and update the system prompt with explicit "do not flag X" instructions for the most common false-positive categories.
• Rate-limit comment volume per PR. A single PR should not receive more than 5–10 comments from the LLM; past that, signal-to-noise collapses. Cap the action at the prompt level ("return at most N comments").
• Track the verdict-to-merge correlation weekly. If 80% of block verdicts merge anyway, your threshold is too aggressive. If 0% of comment verdicts get any human action, your prompt is producing noise.

Operational Pitfalls That Bite in Month Two

• Model updates break prompts. A new Qwen3-Coder release changes output formatting subtly; structured-JSON parsing fails in CI; reviews stop posting. Pin the model digest with ollama show <model> --modelfile; upgrade in a staging branch before promoting.
• GPU memory fragmentation under long uptime. A GPU server running 24/7 can fragment VRAM and refuse new allocations after weeks of operation. Restart ollama serve weekly via a cron job; this is cheap and avoids the failure mode entirely.
• CI runner contention. A self-hosted runner that hosts both the LLM server and other CI jobs will see review latency spike under build load. Separate the runner and the GPU server when team size crosses ~25 developers.
• Diff size growth. PR sizes drift upward; eventually a PR exceeds the model's practical working context and reviews silently degrade. Add a guard in the action that splits or truncates diffs above ~30K tokens and warns the author.
• Power and cooling. A continuously loaded RTX 4090 draws ~350W under inference and produces a meaningful amount of heat. A closet-sized server room without active cooling will throttle the GPU; throttling costs latency and developers notice.
• Forgotten log rotation. Ollama logs every request body by default. After three months of PR reviews, the log file is large and contains historical PR diffs in plain text. Rotate logs weekly; archive or purge per your data-retention policy.

Common Mistakes Setting Up Local-LLM Code Review

• Mistake 1: starting with a 7B model on a 16 GB GPU. Qwen3-Coder 7B reviews are noticeably worse than 30B reviews; developers lose trust quickly and the project gets shelved. If you cannot fit the 30B, either upgrade the GPU or use a cloud API for the first six months while you secure budget.
• **Mistake 2: blocking PRs on block verdicts from day one.** The first month is calibration; treat all output as advisory until you have measured the false-positive rate. Promote to blocking only after the rate is below ~5%.
• **Mistake 3: exposing ollama serve on 0.0.0.0:11434 with no auth.** This is the LLM-era equivalent of leaving Redis bound to a public interface. Bind to a private interface and put auth in front before any cross-host exposure.
• Mistake 4: skipping the cache. Re-reviewing unchanged files on every CI run wastes ~80% of the inference budget on a typical PR. A small file-hash + diff-hash cache (Redis or SQLite) cuts review latency and GPU load dramatically.
• Mistake 5: running training jobs on the same GPU. Training spikes VRAM to the limit and starves the review model. Use separate GPUs or, if you must share, run training on a strict schedule that does not overlap with peak PR hours.
• Mistake 6: building the GitHub Action without a feedback loop. A review system without 👍/👎 reactions cannot improve. Build the loop in week one; collect data; iterate on the prompt monthly.

Sources

• Ollama Documentation — Official HTTP API reference for /api/chat, /api/generate, structured output, and model management.
• vLLM Documentation — High-throughput inference server documentation; the upgrade path past Ollama for high-concurrency teams.
• GitHub Actions Documentation — Official reference for self-hosted runners, secrets, and the Actions JavaScript SDK used in the workflow above.
• GitHub Advanced Security Pricing — List-price reference for the cost comparison; verify against your actual negotiated terms.
• Qwen3-Coder Model Card — Architecture, context window, and licence terms for the recommended review model.
• GitLab CI/CD Reference — Equivalent reference for GitLab teams; the LLM-call portion of the workflow is identical.

FAQ

Related Reading

• Replace GitHub Copilot With a Local LLM — broader cost-replacement context for teams considering self-hosted AI tooling.
• Continue.dev vs Cline vs Aider: Best Local Coding Agent in 2026 — what runs in pipelines vs interactive editor work; the harness layer beneath the model.
• Best Local Coding Models in 2026 — model side of the equation: Qwen3-Coder, DeepSeek, Codestral, and the licensing landscape.
• System Prompt vs User Prompt: What's the Difference — prompt-design grounding for the review system prompt.
• llama.cpp vs Ollama vs vLLM — inference engine comparison; vLLM is the upgrade path past Ollama for high-concurrency teams.
• Power Local LLM Hub — full guide library.