You want to run a current image model locally. The repository offers you model_bf16.safetensors at 19 GB, model_fp8_e4m3fn.safetensors at 10 GB, model_int8_convrot.safetensors at 5.4 GB, and a folder of GGUF files named Q8_0, Q6_K, Q5_K_M, Q4_K_M and Q3_K_S. You have 12 GB of VRAM.
Nothing in those filenames tells you what you lose. This is what they mean.
The one idea
A model is a few billion numbers. Quantisation stores each number in fewer bits.
A weight in bf16 takes 2 bytes. In 8-bit formats it takes 1 byte; in 4-bit, half a byte. So a 10-billion-parameter model is 20 GB at bf16, 10 GB at 8-bit, 5 GB at 4-bit. That is the entire source of the savings, and the arithmetic is that simple.
What you lose is precision: fewer bits means fewer distinct values a weight can take, so each weight gets rounded to the nearest representable one. The question every format is trying to answer is which numbers to be imprecise about.
Why you cannot just round everything
Here is the problem that makes quantisation interesting rather than trivial.
To store a tensor in 8-bit integers you need a scale factor — you find the largest magnitude in the group, and map the range onto the 256 available integer values. Fine, until the tensor contains a few channels whose values are vastly larger than the rest.
Those are outlier channels, and they are pervasive in transformers. When one channel reaches 50 and the rest sit under 1, the scale has to cover 50, so everything under 1 collapses into the first couple of integer steps. You have preserved the outliers perfectly and destroyed the other 99% of the tensor.
Every serious quantisation format is a different answer to outliers. Once you see that, the names stop being arbitrary.
The formats
fp8 (e4m3fn, e5m2)
8-bit floating point. Unlike integers, floats devote bits to an exponent, so they represent a wide dynamic range with relative rather than absolute precision — a small value gets fine steps, a large one gets coarse steps. Outliers are handled naturally.
The names are the bit split: e4m3 is 4 exponent bits and 3 mantissa bits; e5m2 trades mantissa for more range. fn means finite-only — no infinities.
Use it when: your GPU supports it natively (Ada/Hopper/Blackwell — RTX 40-series and up). Then it is often faster than bf16, not just smaller. On older cards fp8 weights get converted on the fly, which saves memory but not time.
Quality: close to bf16 for image models. The safest 8-bit choice.
int8 with per-channel scales
8-bit integers with a separate scale per row or channel rather than one for the whole tensor. Per-channel scaling is a partial outlier fix: a channel with large values gets its own large scale, so it stops ruining its neighbours.
int8 with a rotation (ConvRot, QuaRot, SpinQuant)
The clever version, and worth understanding because it is increasingly the default. Before quantising, the weights are rotated by a Hadamard matrix.
A Hadamard rotation mixes every channel into every other channel. Energy that was concentrated in three outlier channels gets spread across all of them, so after rotation there are no outliers — just a tensor of broadly similar magnitudes, which is exactly what int8 quantises well. The rotation is undone mathematically during inference, so the result is equivalent.
The elegance is that a regular Hadamard matrix can be regenerated from its size alone, so nothing has to be stored. In the Kromoss release, for instance, 168 linear layers are stored int8 after a 256-wide Hadamard rotation, and the loader rebuilds the rotation at load time — “no rotation matrices are stored.”
Use it when: offered. It is usually the best quality-per-byte at 8-bit, and the files are smaller than fp8 because the scales are cheap.
GGUF Q-levels
GGUF is a container format (from llama.cpp) with a family of quantisation schemes. The naming decodes cleanly:
| Name | Bits | Notes |
|---|---|---|
Q8_0 | 8 | Near-lossless. Rarely worth it over fp8. |
Q6_K | ~6.6 | Very close to 8-bit quality |
Q5_K_M | ~5.7 | Good balance |
Q4_K_M | ~4.8 | The popular sweet spot |
Q4_K_S | ~4.6 | Smaller, slightly worse |
Q3_K_M | ~3.9 | Visible degradation |
Q2_K | ~3.4 | Usually not worth running |
The _K means k-quants: weights are stored in small blocks, each with its own scale, and the scales themselves are quantised. _S, _M and _L are small/medium/large variants that differ in which layers get extra bits — _M typically keeps attention and embedding layers at higher precision because those are the ones that degrade visibly.
Use it when: you need to go below 8-bit. A 4-bit GGUF will fit where nothing else does, and it will cost you some fine detail and prompt adherence.
Important: GGUF for image models needs a loader node — in ComfyUI, the ComfyUI-GGUF custom node pack, with its own Unet Loader (GGUF) rather than the standard one.
What not to quantise
This is the part people get wrong, and it causes a specific, recognisable failure.
Keep the VAE in fp32 or bf16. The VAE decodes latents into pixels, and it is small — a few hundred megabytes — so quantising it saves almost nothing. Run it at reduced precision and you get colour banding, washed-out gradients and blotchy skies. If your output has subtle posterisation in smooth areas, the VAE is the first suspect.
Model cards often specify this explicitly. Kromoss, for example, instructs that the transformer and text encoder use bfloat16 while “the VAE and initial noise use float32.”
Text encoders tolerate quantisation less well than the diffusion transformer. A quantised text encoder degrades prompt adherence rather than image quality, which is much harder to diagnose — the images look fine, they just are not quite what you asked for. If you have memory to spend, spend it here.
Choosing, by card
| VRAM | Reasonable choice |
|---|---|
| 24 GB+ | bf16, or fp8 for speed on 40-series+ |
| 16 GB | fp8, or int8 with rotation |
| 12 GB | int8 ConvRot, or Q6_K / Q5_K_M GGUF |
| 8 GB | Q4_K_M GGUF, plus CPU offload for the text encoder |
| 6 GB | Q4_K_S or Q3_K_M, offload aggressively, expect slow |
Two things that buy more than a smaller quant:
Offloading. You do not need everything resident at once. A diffusion pipeline runs the text encoder, then the transformer for N steps, then the VAE. ComfyUI will move components to system RAM between stages; --lowvram and --novram make it more aggressive. This trades speed for memory and keeps precision intact, which is usually the better deal than dropping two Q-levels.
Step-reduction LoRAs. An 8-step turbo or DMD LoRA at CFG 1 does roughly a twelfth the compute of 50 steps at CFG 3, because classifier-free guidance runs the model twice per step. That is a speed fix, not a memory fix — but if you went to a low quant purely because generation was unbearably slow, fix the speed instead and keep the quality.
How to actually evaluate a quant
Do not judge on one image. Fix the seed, the prompt and all sampler settings, generate the same image at bf16 and at your candidate quant, and compare. Look specifically at:
- Fine text in the image — the first thing to break
- Hands and faces at small scale
- Smooth gradients — banding here means the VAE, not the transformer
- Prompt adherence on a compound prompt with three or four specified attributes — degradation here means the text encoder
If you cannot tell the difference on a fixed seed, you have found your format. Most people run a quant two levels lower than they need to, on the assumption that smaller is safer for stability. It is not — it is just worse.
Sources
- Qwen Image Edit in ComfyUI: GGUF & FP8 Low-VRAM Workflow — ComfyUI Workflow Blog
- ComfyUI-GGUF custom node
- GGUF format specification — ggml
- GGUF on the Hugging Face Hub
- lodestones/Kromoss — an int8 ConvRot build with its quantisation script