How Do You Convert a Model to GGUF With llama.cpp?
Convert a model to GGUF with llama.cpp's conversion script, which reads a source model's weights and metadata and writes them into the single-file GGUF format. Run the script against a Hugging Face-format model directory, choose an output type (full precision or a quantization level), and the result is a `.gguf` file ready to load with llama.cpp, Ollama, or LM Studio.
Quick Answer
Run llama.cpp's conversion script against a Hugging Face-format model directory, specifying an output type such as f16 (full precision) or a quantized format like q4_k_m. The script reads the model's weights and config, writes them into a single .gguf file, and that file can be loaded directly by llama.cpp, Ollama, or LM Studio.
- ▸Source model must be in Hugging Face format (safetensors or PyTorch checkpoint) before conversion
- ▸Choose an output type: f16/f32 for full precision, or a quantized type to convert and shrink in one step
- ▸Quantizing separately after an f16 conversion gives more control than quantizing during conversion
Technique & Concept ExplainersIntermediate
Key Takeaways
- ✓GGUF conversion reads a Hugging Face-format model's weights and metadata and writes them into a single portable .gguf file
- ✓Convert to f16/f32 first if you plan to try multiple quantization levels — quantizing from an already-quantized conversion loses more quality than quantizing from full precision
- ✓The output type you choose at conversion time determines both file size and inference quality — there is no free lunch between the two
- ✓Most conversion errors trace back to an incompatible or incomplete source model directory, not the conversion script itself
The Conversion Process
Conversion has three practical steps: prepare the source model, run the conversion script with an output type, and — if you converted to full precision — quantize afterward.
- ▸1. Prepare the source model: the conversion script expects a Hugging Face-format directory containing the model's weight files (safetensors is preferred over legacy PyTorch `.bin` checkpoints) plus its config and tokenizer files. A partially downloaded or manually assembled directory missing the tokenizer files is the single most common cause of conversion failures.
- ▸2. Run the conversion script with an output type: `python convert_hf_to_gguf.py
--outtype f16` produces a full-precision GGUF file. Passing a quantized `--outtype` (e.g. `q8_0`) quantizes during conversion in one step, but gives you less control than converting to f16 first and quantizing separately afterward with a dedicated quantization tool. - ▸3. Quantize separately for more control (optional but recommended): starting from an f16 GGUF file, run llama.cpp's quantization tool to produce multiple quantized variants (e.g. q4_k_m, q5_k_m) from the same full-precision source, letting you compare quality/size tradeoffs without re-running the conversion each time.
Common Conversion Errors
Most conversion failures fall into a small number of categories, almost always traceable to the source model rather than the script itself.
- ▸Unrecognized architecture: the conversion script maintains an internal list of supported model architectures. A very new or unusual architecture may not be supported yet — check the tool's supported-architecture list before assuming the model files are broken.
- ▸Missing tokenizer files: conversion needs the tokenizer configuration alongside the weights. A model directory with only weight files and no tokenizer files will fail partway through conversion.
- ▸Mismatched file format expectations: legacy PyTorch `.bin` checkpoints sometimes need an extra conversion step to safetensors first, depending on the tool version — prefer downloading models that already ship in safetensors format to avoid this entirely.
- ▸Insufficient disk space or RAM during conversion: conversion loads the full-precision model into memory before writing it out, so it needs roughly the same RAM as the model's full-precision size, plus the disk space for both the source files and the output GGUF file simultaneously.
Frequently Asked Questions
Should I quantize during conversion or convert to full precision first?▾
Convert to full precision (f16) first if you plan to compare multiple quantization levels — quantizing separately afterward from the same f16 source gives more accurate results than quantizing directly from a source model during conversion, and lets you produce several quantized variants without repeating the conversion step.
Why does my converted GGUF file fail to load in Ollama or LM Studio even though conversion succeeded?▾
A successful conversion doesn't guarantee the loading tool supports that specific architecture or GGUF version. GGUF format and architecture support both evolve over time, and an older tool version may not recognize a file produced by a newer conversion script — keep both tools updated together.
How much disk space does conversion need?▾
Budget for the source model's full size plus the output GGUF file's size simultaneously — for a full-precision conversion, that roughly doubles the model's footprint during the process. Quantized output files are smaller, but you still need the source files present until conversion finishes.
Can I convert a model that's already been fine-tuned?▾
Yes — the conversion process works the same way on a fine-tuned model as a base model, as long as the fine-tuned weights are saved in a standard Hugging Face-compatible format. If the fine-tune used adapter weights (like LoRA) rather than full weights, merge the adapter into the base model first, then convert the merged result.