Go to download a model and you’re faced with twenty files: Q4_K_M, Q8_0, bf16, IQ3_XS, nvfp4. They’re all the same model. They differ in how precisely the weights are stored, which decides how much memory the thing needs and how well it works.
Getting this choice right is the difference between a model that runs on your laptop and one that doesn’t run at all.
1. What quantization does
A trained model is billions of numbers. Originally each is a 16-bit float — about 65,000 possible values per weight. Quantization stores them with fewer bits: 8, 4, sometimes 2 or fewer.
The trick that makes it work is grouping. Weights are quantized in small blocks, each with its own scaling factor, so instead of one coarse scale across the whole model you get a local one per block. That preserves far more of the original behaviour than naive rounding would.
Rough arithmetic: bits per weight ÷ 8 × parameter count = gigabytes. A 7B model at 4 bits is about 3.5GB. At 8 bits, about 7GB. In 16-bit float, about 14GB.
2. Reading the names
For GGUF files (the llama.cpp format), a name like Q4_K_M decodes as:
- Q4 — roughly 4 bits per weight
- _K — “K-quant,” a smarter scheme that spends more bits on the layers that matter most
- _M — the size tier within that scheme: S (small), M (medium), L (large)
So Q4_K_M is a 4-bit K-quant, medium. Q8_0 is 8-bit, the older straightforward scheme. IQ prefixes are “importance matrix” quants, which use a calibration pass to decide what to protect, and generally beat plain quants of the same size at the cost of a more involved build process.
Other ecosystems use their own names — AWQ, GPTQ, NVFP4, int8 — but the underlying idea is the same, and NVFP4 in particular is tuned for recent NVIDIA hardware.
3. Choosing, practically
The rule most people converge on: run the largest model you can fit at Q4 or above, rather than a smaller model at higher precision. A 27B model at 4 bits is usually better than a 7B model at 8 bits, even though they take similar memory.
Work out your budget:
- VRAM on a discrete GPU is your hard ceiling, minus about 1–2GB for context and overhead.
- Apple Silicon shares memory between CPU and GPU, so your budget is roughly your unified memory minus what the OS needs. This is why Macs punch above their weight for local models.
- CPU-only works with llama.cpp but is much slower; RAM is the limit.
Then: pick the biggest model that fits at Q4_K_M, and if you have headroom left, move up to Q5 or Q6 rather than to a bigger model you can only run at Q2.
4. Where quality actually goes
The degradation isn’t uniform, and knowing its shape helps.
- Q8 to Q6 — essentially invisible for most uses.
- Q6 to Q4 — small, usually acceptable. This is the sweet spot most people use.
- Below Q4 — degradation becomes noticeable, and it shows up first in long-context coherence, precise instruction following, and anything requiring exactness (code, maths, structured output). Casual conversation degrades last, which is why low-bit models can feel fine until you ask them to do something demanding.
- Extreme quantization (2-bit, ternary) is an active research area and moving fast. Treat vendor retention claims with the scepticism you’d apply to any self-reported benchmark.
5. Image, video and audio models are different
Most quantization advice is written about language models and doesn’t transfer cleanly.
Diffusion models are more sensitive to precision in some components than others, which is why you see mixed setups — a quantized transformer with the VAE kept at higher precision, since the VAE is what turns latents into pixels and its errors are directly visible as artifacts. Audio models behave similarly: the decoder that produces the waveform is usually where you don’t economise.
Practical rule: quantize the big generative core, keep the small encoder/decoder pieces at higher precision. It’s what most published builds do; look at what the model’s own repo ships rather than converting blindly.
6. When to stop
Quantization buys you access to hardware you own. It doesn’t buy you quality. If a model at Q4 does the job, going to Q3 to free 2GB is a bad trade unless that 2GB is what makes it run at all.
And measure it yourself on your task. Benchmarks measure benchmarks. If you’re using a model to caption images for an installation, run both builds over fifty of your own images and compare. That takes twenty minutes and beats any published table.