Skip to main content
PromptQuorumBuilt for humans. Structured for AI.
Home/Power Local LLM/Shimmy Review: A 5MB Rust Alternative to Ollama
Overview & Reference

Shimmy Review: A 5MB Rust Alternative to Ollama

·10 min read·By Hans Kuepper · Founder of PromptQuorum · Discovery engine for open-weight & open-source AI
Read in:
🇺🇸en🇩🇪de🇫🇷fr🇯🇵ja🇨🇳zh🇪🇸es🇧🇷pt🇸🇦ar🇰🇷ko

Shimmy is a free, open-source, single-binary inference server written entirely in Rust that serves local GGUF and SafeTensors models through an OpenAI-compatible API, with no Python runtime, no C++ toolchain, and no llama.cpp dependency. Created and maintained by Michael A. Kuykendall as an independent, solo-maintained open-source project under the Apache-2.0 license, Shimmy positions itself specifically against Ollama's much larger binary and slower cold-start time — its own README claims a binary a few megabytes in size versus Ollama's several-hundred-megabyte install.

Shimmy (github.com/Michael-A-Kuykendall/shimmy) is a free, open-source, single-binary inference server written entirely in Rust, positioned as a lightweight, dependency-free alternative to Ollama for serving local GGUF and SafeTensors models through an OpenAI-compatible API. It has no Python runtime, no C++ toolchain, and no llama.cpp dependency — the binary itself is a few megabytes rather than several hundred. This review covers what Shimmy actually does, how it compares to Ollama and llama.cpp, how to install it, and who it fits.

Key Takeaways

  • Shimmy (github.com/Michael-A-Kuykendall/shimmy) is a free, open-source, single-binary inference server, not an IDE or chat app
  • Created and maintained by Michael A. Kuykendall as an independently maintained project — its own README states "free forever," with no evidence of a company or funding round behind it
  • Apache-2.0 licensed, per the repository's own LICENSE file
  • Runs on Airframe, its own pure-Rust WebGPU (WGSL) transformer engine, avoiding a C++ toolchain or llama.cpp dependency entirely
  • Serves GGUF and SafeTensors models through an OpenAI-compatible API; the v2.6.2 release added Ollama-compatible and Anthropic-compatible routes alongside it
  • 5,890 GitHub stars as of this review (github.com/Michael-A-Kuykendall/shimmy, verified via GitHub API)

📍 In One Sentence

Shimmy is a free, open-source, single-binary local inference server written entirely in Rust — a lightweight, dependency-free alternative to Ollama — that serves GGUF and SafeTensors models through an OpenAI-compatible API, with no Python runtime, no C++ toolchain, and no llama.cpp dependency, per the project's own README.

💬 In Plain Terms

Instead of installing Ollama's several-hundred-megabyte package, Shimmy is one small executable file (a few megabytes) you download or build with Cargo, point at a GGUF model file on your disk, and run. Any tool that already speaks the OpenAI API format can talk to Shimmy instead, without code changes.

📌Note: This review is the deep-dive companion to Shimmy's entry in the Local LLM Software Directory — see that page for how Shimmy compares at a glance to dozens of other local AI tools. It is based on Shimmy's own README, changelog, and release notes, not hands-on benchmarking by PromptQuorum.

What Is Shimmy?

Shimmy is a command-line inference server: a single binary that loads a local GGUF or SafeTensors model file and exposes it through an OpenAI-compatible HTTP API. Its own GitHub description calls it a "pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary."

  • Product type: a command-line, single-binary server — not a GUI app and not an IDE extension
  • Creator: Michael A. Kuykendall, an independent developer; this review found no evidence of a company, investor, or funding round behind the project
  • Repository: github.com/Michael-A-Kuykendall/shimmy
  • License: Apache-2.0, confirmed via the repository's own LICENSE file
  • Scale: 5,890 GitHub stars as of this review, per GitHub API
  • Engine: Airframe, a pure-Rust WebGPU (WGSL) transformer engine that Shimmy's own README credits with letting it avoid a C++ toolchain entirely

Shimmy's Project History and Version Milestones

Shimmy has shipped frequent, incremental releases, with its own CHANGELOG documenting close to 200 commits and a steady cadence of point releases through 2026, centered on its self-published model certification pipeline and its Airframe inference engine.

  1. 1
    v2.5.0 — August 6, 2026: Airframe engine updated to 0.3.0
    Why it matters: Updated the underlying pure-Rust WebGPU transformer engine that Shimmy runs on.
  2. 2
    v2.6.0 — August 27, 2026: Certification pipeline consolidated
    Why it matters: Consolidated Shimmy's self-published model certification into a "3-box" MATH + INFERENCE + DETERMINISM regimen, certifying 26 model/quantization combinations across 12 model families, and removed roughly 1,600 lines of legacy engine code.
  3. 3
    v2.6.1 — August 28, 2026: Unified chat templating
    Why it matters: Moved all prompt formatting through one shared prompt_render module using real GGUF chat templates via Jinja, and retired a legacy GPU server code path (roughly 1,847 lines removed).
  4. 4
    v2.6.2 — August 28, 2026: OpenAPI docs and additional compatible routes
    Why it matters: Added a machine-readable API contract (`GET /openapi.json`) and interactive Swagger UI (`GET /docs`), and added Ollama-compatible and Anthropic-compatible routes alongside the existing OpenAI-compatible API.
  5. 5
    v2.6.3 — August 29, 2026: Integrated-GPU model loading fix
    Why it matters: Fixed model loading on integrated GPUs, including a specific fix for Intel Arc GPUs, by updating the Airframe engine dependency to 0.4.2.
  6. 6
    v2.6.4 — August 30, 2026: Latest point release as of this review
    Why it matters: The most recent tagged release found on the project's GitHub releases page at the time of this review — check [github.com/Michael-A-Kuykendall/shimmy/releases](https://github.com/Michael-A-Kuykendall/shimmy/releases) directly for anything shipped after this review's publish date.

What Does Shimmy Actually Do?

Shimmy's feature set centers on serving local models through a familiar API surface while keeping the binary itself minimal and dependency-free, per Shimmy's own README.

  • GGUF-native model loading — loads GGUF model files directly, without recompiling or hardcoding per-model constants, per the project's own documentation; SafeTensors format is also supported
  • OpenAI-compatible API — chat completions, text completions, streaming responses, and model-listing endpoints that match the OpenAI API format, so existing OpenAI-client code and tools can point at Shimmy without modification
  • Additional compatible routes — as of v2.6.2, Shimmy also exposes Ollama-compatible and Anthropic-compatible routes alongside its OpenAI-compatible API
  • Self-published model certification — a "3-box" test regimen (MATH, INFERENCE, DETERMINISM) that Shimmy runs against supported models; the project states 26 model/quantization combinations across 12 model families pass this regimen as of this review
  • TurboShimmy INT4 KV-cache compression — Shimmy's own documentation describes roughly 7x lower KV-cache memory use in tested configurations, intended to let larger context windows or smaller GPUs handle a given model
  • Extended context via YaRN RoPE scaling — configurable through environment variables, per the project's documentation
  • No external inference dependency — Shimmy runs on Airframe, its own pure-Rust WebGPU engine, rather than wrapping llama.cpp or requiring a Python/CUDA toolchain

Usage Examples: Three Ways to Use Shimmy

These are concrete workflows built from Shimmy's documented features above — not hypothetical use cases.

Serve a local GGUF model as an OpenAI-compatible endpoint

Install Shimmy via Cargo, then run shimmy serve --model-path /absolute/path/to/model.gguf --bind 127.0.0.1:11435 to expose that model at a local address. Point any tool or SDK that expects the OpenAI API format at that address instead of api.openai.com, reusing existing client code against a fully local model.

Swap Shimmy in for Ollama without changing application code

Because Shimmy exposes Ollama-compatible routes as of v2.6.2, an application already built to call Ollama's API can often point at Shimmy's address instead, letting you test whether the smaller, dependency-free binary meets your latency and memory needs before committing to a migration.

Check the OpenAPI contract before integrating

With Shimmy running, open GET /docs in a browser for the interactive Swagger UI, or fetch GET /openapi.json for the machine-readable API contract, to confirm exactly which endpoints and parameters a given Shimmy version supports before wiring up a new integration.

Platform, Pricing, and Licensing

Platform

What Shimmy states:
A command-line, single-binary server — runs on macOS, Windows, and Linux; no GUI installer.

Cost

What Shimmy states:
Free and open source. The project's own README states "Shimmy will be free forever." No paid tier exists.

Licensing

What Shimmy states:
Apache-2.0, confirmed via the repository's own LICENSE file.

Funding

What Shimmy states:
Independently maintained by Michael A. Kuykendall; the repository lists a voluntary sponsorship program (from a $5/month tier up to a $500/month infrastructure-partner tier), not a funding round or company.

Verify current licensing and funding status directly on github.com/Michael-A-Kuykendall/shimmy before relying on this table for a compliance or vendor-risk decision.

Shimmy vs. Ollama

Shimmy positions itself directly against Ollama, its own README calling itself "the 5MB alternative to Ollama." Both serve local models through similar HTTP APIs; the difference is mainly in binary size, dependency footprint, and each project's inference engine.

Binary size

Shimmy:
A few MB, per Shimmy's own README claim
Ollama:
Several hundred MB, per third-party comparisons

Inference engine

Shimmy:
Airframe — its own pure-Rust WebGPU (WGSL) engine
Ollama:
Built on llama.cpp

Runtime dependencies

Shimmy:
None — no Python, no C++ toolchain, per its README
Ollama:
Bundles its own runtime; no separate Python needed either

API compatibility

Shimmy:
OpenAI-compatible, plus Ollama- and Anthropic-compatible routes (v2.6.2+)
Ollama:
Has its own native API plus an OpenAI-compatible layer

Model format

Shimmy:
GGUF and SafeTensors
Ollama:
GGUF-based, via its own model library and Modelfile format

Maintainer

Shimmy:
Independent solo developer
Ollama:
Ollama Inc., a funded company

If binary size, cold-start time, or avoiding llama.cpp specifically are the deciding factors, Shimmy is built to win on exactly those axes, per its own positioning. If you want the largest existing model library, community tooling, and a funded company behind long-term maintenance, Ollama's track record is more established — see the Ollama Review for a full breakdown. Verify current benchmarks yourself rather than relying on either project's own marketing.

Who Should Use Shimmy?

Shimmy fits developers who specifically want a minimal-footprint, dependency-free inference server, and are comfortable running a still-young, solo-maintained project.

Use Shimmy if you

  • Want the smallest possible install footprint for a local inference server — a few-megabyte binary rather than a several-hundred-megabyte package
  • Want to avoid a Python runtime, C++ toolchain, or llama.cpp dependency entirely
  • Already have tooling built against the OpenAI API format and want a drop-in local backend
  • Are comfortable evaluating a solo-maintained, independent open-source project rather than one backed by a funded company
  • Want to try Ollama-compatible routes on a lighter-weight server without rewriting existing Ollama-targeted code

Look elsewhere if you

  • Want the largest existing model library and one-command pulls from a hosted registry — Ollama's model library is more established
  • Need a GUI for model management rather than a command-line-only tool — Shimmy has no built-in interface
  • Require guaranteed long-term maintenance backed by a company with dedicated staff, rather than a single independent maintainer
  • Need broad, battle-tested compatibility with every quantization and model architecture — Shimmy currently certifies 26 specific model/quantization combinations, not every GGUF file that exists

Competitors and Alternatives

Within lightweight local-inference servers, Shimmy is most directly comparable to Ollama and llama.cpp — the engine most local tools, including Ollama, build on — plus LMDeploy for teams evaluating higher-throughput serving options.

LMDeploy

Best known for:
High-throughput LLM serving and quantization, aimed at production workloads
Link:
LMDeploy Review
Articles about LMDeploy (2)

Also mentioned in:

This is not an exhaustive list of local inference servers — see the Local LLM Software Directory for the full, regularly updated catalog, including Shimmy's own directory entry.

Common Mistakes When Evaluating Shimmy

Most confusion about Shimmy comes from conflating it with unrelated same-named projects, or assuming it has Ollama's model-library scale.

Mistake 1: Confusing Shimmy with an unrelated project of the same name

The name "shimmy" is used by other, unrelated open-source projects (for example, testing-mock or Windows-shim utilities). Verify you are looking at github.com/Michael-A-Kuykendall/shimmy specifically before installing.

Mistake 2: Expecting Ollama's model library scale

Shimmy certifies 26 specific model/quantization combinations across 12 model families as of this review, not an open-ended catalog. It loads GGUF files directly rather than pulling from a large hosted registry the way Ollama does.

Mistake 3: Assuming Shimmy is backed by a company

This review found no evidence of a funding round or commercial company behind Shimmy — it is maintained by a single independent developer, Michael A. Kuykendall, supported by a voluntary sponsorship program.

Mistake 4: Assuming Shimmy wraps llama.cpp like most other lightweight servers

Unlike Ollama and many similar tools, Shimmy does not use llama.cpp. It runs on Airframe, its own pure-Rust WebGPU transformer engine, which is part of how it avoids a C++ toolchain dependency.

Frequently Asked Questions

What is Shimmy?

Shimmy (github.com/Michael-A-Kuykendall/shimmy) is a free, open-source, single-binary inference server written entirely in Rust that serves local GGUF and SafeTensors models through an OpenAI-compatible API.

Is Shimmy free?

Yes. Shimmy is free and open source under the Apache-2.0 license. Its own README states "Shimmy will be free forever," with no paid tier.

How do I install Shimmy?

The simplest method, per the project's own README, is cargo install shimmy. Prebuilt release binaries and a Docker configuration are also available on the GitHub repository.

How is Shimmy different from Ollama?

Shimmy's own README describes it as "the 5MB alternative to Ollama" — a much smaller binary with no Python runtime or C++ toolchain, running on its own Airframe engine rather than llama.cpp. Ollama has a larger, more established model library and is backed by a funded company; Shimmy is maintained by a single independent developer.

Does Shimmy use llama.cpp?

No. Shimmy runs on Airframe, its own pure-Rust WebGPU (WGSL) transformer engine, which the project states lets it avoid a llama.cpp or C++ toolchain dependency entirely.

What model formats does Shimmy support?

GGUF and SafeTensors, per the project's own documentation. Shimmy loads GGUF files directly without recompiling or hardcoding per-model constants.

Is Shimmy's API compatible with OpenAI tools?

Yes. Shimmy exposes an OpenAI-compatible API (chat completions, text completions, streaming, model listing). As of the v2.6.2 release, it also added Ollama-compatible and Anthropic-compatible routes.

Who created Shimmy?

Michael A. Kuykendall created and maintains Shimmy as an independent, solo open-source project. This review found no evidence of a company or funding round behind it.

How many models does Shimmy support?

As of this review, Shimmy certifies 26 specific model/quantization combinations across 12 model families through its own "MATH + INFERENCE + DETERMINISM" test regimen — check the project's GitHub repository for the current, up-to-date list.

Has PromptQuorum independently tested Shimmy's performance claims?

This review is based on Shimmy's own README, CHANGELOG, and release notes, rather than hands-on benchmarking of startup time, memory footprint, or KV-cache compression by PromptQuorum.

Sources

← Back to Power Local LLM