Evaluating an Image Model Without Fooling Yourself

Here is a real benchmark table, from the Iris-3B model card:

Benchmark	Base	Turbo
GenEval	0.798	0.820
DPG-Bench	86.52	86.68
LongText-Bench	0.857	0.804
OneIG-Bench	0.540	0.516

The 4-step distilled version wins two and loses two. Which model is better?

The answer depends entirely on what you are making, and you cannot get there from the table alone. Understanding what each number measures is the whole skill.

The two questions benchmarks try to answer

Every text-to-image metric is attacking one of two separate problems:

Did it make the thing I asked for? — prompt fidelity, compositional correctness, counting, spatial relations, attribute binding
Does it look good? — realism, aesthetic quality, distribution match

These come apart constantly. A model can render a flawless photograph of two dogs when you asked for three.

What each metric measures

GenEval — compositional correctness

GenEval runs an object detector over the generated image and checks structured claims: is the object present, are there the right number, is it the stated colour, is it positioned as described.

Measures: object presence, counting, colour binding, spatial relation, attribute binding
Blind to: whether the image is beautiful, coherent or well-composed. A hideous image with the right three red cubes in the right places scores well.
Range: 0–1, higher is better

Because it depends on a detector, GenEval inherits that detector’s failures. Unusual objects and unusual styles get marked wrong when the detector simply cannot see them.

DPG-Bench — dense, long prompts

DPG-Bench uses long, dense prompts packed with many simultaneous requirements, and scores how many are satisfied. It targets the failure mode where a model handles three constraints and quietly discards the fourth.

Measures: adherence across many constraints at once
Blind to: aesthetics; also tends to reward verbose literalism
Range: 0–100, higher is better

FID — distribution distance

Fréchet Inception Distance compares feature statistics of a set of generated images against a set of real ones. It is a set-level metric and says nothing about any individual image.

Measures: how closely the overall distribution resembles the reference set
Blind to: prompt fidelity entirely. A model that ignores every prompt and emits beautiful plausible images can score superbly.
Range: lower is better; not comparable across different reference sets or sample counts

That last caveat is the one that invalidates most FID comparisons you will see. FID computed on 10,000 samples against COCO is a different number from FID on 2,000 samples against something else. Comparing them is meaningless.

CLIPScore — image–text agreement

Cosine similarity between a CLIP embedding of the prompt and of the image.

Measures: rough semantic agreement
Blind to: counting, spatial relations, negation. CLIP is notoriously weak at all three.
Use: a cheap sanity check, never a ranking

Text-rendering benchmarks

LongText-Bench and similar evaluate whether text inside the image is legible and correct. This is now a major differentiator, and it is also where distillation hurts most — note Iris-3B Turbo losing 0.857 → 0.804 on exactly this axis while gaining on GenEval.

If your work involves posters, UI mockups, signage or any typography, this metric outranks the others for you, and the “better” model from the composite table is the wrong choice.

Running a fair comparison

Most informal comparisons are confounded before the first image is generated. Fix these:

1. Fix the seeds. Use the same seed list for every model. Different seeds means you are partly measuring luck.

2. Fix the sampler and step count — or report that you did not. A model evaluated at 100 steps against one at 4 steps is not a comparison of models, it is a comparison of compute budgets. If you are deliberately comparing a turbo build to a base build, say so, because that is the question.

3. Do not tune CFG per model and then compare. Either hold CFG constant or sweep it for both and report the best of each. Note that distilled models often bake CFG into the weights and ignore the parameter entirely.

4. Generate enough samples. Per-prompt variance is enormous. Four images per prompt is a vibe; a few hundred prompts with several seeds each is a measurement.

5. Keep resolution identical. Scores move with resolution, and models have a native resolution they were trained for.

6. Write down the versions. Model revision, sampler, scheduler, library version. Diffusion stacks change behaviour between releases.

Build a private eval set — this is the part that matters

Public benchmarks measure general capability. You care about your work. A small private eval set will tell you more than every leaderboard combined, and it takes an afternoon.

Collect 30–50 prompts from your actual practice. Not clever prompts — real ones, the kind you have typed in anger.

Group them by what they test:

Group	Example of what it probes
Counting	”five identical ceramic bowls”
Spatial	”the lamp behind and to the left of the chair”
Text	the exact string you need rendered
Style consistency	the same subject across three described styles
Negation	”a street with no cars”
Your domain	whatever your work is actually about

Fix seeds, generate a grid, and score blind. Shuffle the outputs, strip the filenames, and rate them without knowing which model produced which. Self-deception here is the default, not the exception — you will favour the model you expected to win.

Score on a rubric, not a feeling. For each image: did it satisfy every stated requirement (yes/no), and separately, is it usable (1–5). Keep fidelity and quality apart, exactly as the public metrics do.

Thirty prompts × 4 seeds × 2 models is 240 images and about an hour of scoring. It is the highest-value hour available in this field.

How to read someone else’s numbers

Unspecified sample count on FID → ignore the number
No seeds, no sampler, no step count → treat as marketing
A single composite score → ask what it averages; composites hide exactly the trade-offs you need
Wins on every metric → check whether the benchmarks were run by the model’s authors on their own harness
Benchmarks not reported at all → often more honest than a cherry-picked subset

And the broadest caution: benchmark saturation is real. When every model scores 0.80–0.85 on GenEval, the differences are inside the noise of the detector doing the scoring, and the ranking stops meaning anything. At that point the private eval set is not a supplement to the public ones. It is the only measurement left.