Appearance
How to Run Blind Tests for AI Image and Text Models in 2026
Almost every "which model is better" conclusion inside a company is produced by someone who already knew which answer they wanted, looking at two outputs side by side with the model names visible.
That is not a test. It is a preference with a screenshot attached. The tell is that the conclusion changes when the same person repeats the comparison a week later, or when the file names are swapped.
A blind test is not complicated, but it has four moving parts that all have to be right: the outputs must be anonymised, the order must be randomised, the criteria must be fixed before the candidates are seen, and the raters must not be able to reverse-engineer which model they are looking at. Get any one of those wrong and the result is decoration.
This is the protocol that holds up when someone asks how you decided.
Table of Contents
- Why Your Own Comparison Is Not Evidence
- Blind What, Exactly
- Designing the Pair
- Set the Criteria Before the Candidates
- Rater Mechanics
- Two Protocols: Pairwise and Absolute
- Reading the Results Without Fooling Yourself
- Where Blind Tests Still Mislead
- A Template You Can Run This Week
- FAQs
- Related Articles
Why Your Own Comparison Is Not Evidence
Three biases do almost all the damage, and none of them feel like bias from the inside.
Brand prior. You know GPT-6.1 Terra is expensive and Claude Opus 5.5 is strong at long-form reasoning. That knowledge leaks into your judgement of a paragraph, and it leaks harder when the outputs are close.
Position bias. In a side-by-side, the first output wins more often than it should. People read left to right, form an impression in the first ten seconds, and then evaluate the second against it rather than independently.
Effort bias. The longer, more detailed answer feels better even when it is wrong. Verbosity reads as thoroughness, which is exactly the failure mode models have learned to exploit.
Blinding removes the first, randomising order removes the second, and scoring against fixed criteria removes most of the third.
Blind What, Exactly
There are three things you can blind, and most tests only do the easiest one.
Blind the identity. Strip model names, watermarks and metadata. For images this means re-exporting everything through the same pipeline so file size and colour profile do not betray the source. For text it means normalising formatting — the model that uses em dashes and bullet lists is identifiable at a glance, and that identification is the leak.
Blind the order. Randomise which output appears as A and which as B, per pair, not once for the whole set. If A is always the incumbent, position bias becomes incumbent bias.
Blind the rater to the hypothesis. Do not tell the rater "we think the new model is better at hands". Tell them what to score and let them score it. If they know the hypothesis, they will find evidence for it, especially when the difference is small.
The third one is the hardest to arrange inside a small team, and it is the one that most often gets skipped. The workaround is to use raters who have no stake in the decision — a designer, a support agent, anyone who will not be in the meeting where the result is used.
Designing the Pair
Same prompt, same parameters, same seed where the model supports it. That sounds obvious and gets violated constantly, usually because someone tweaked the prompt between generations to "give it a fair shot".
For text tasks, the pair must be the same task with the same constraints: same length ceiling, same format requirement, same input document. A model given a looser brief produces a more impressive answer to a different question.
For images, hold the aspect ratio, resolution and reference image constant. If one model needs a different prompt structure to perform — some respond to terse prompts, others to long descriptive ones — then run two pairs: one with a shared prompt, one with each model's best prompt. Report both. The first tells you about out-of-box performance, the second about ceiling, and they often disagree.
Keep pairs short. One prompt, one deliverable, one decision. A pair that asks the rater to compare two whole landing pages produces a judgement about taste, not about the model — narrow the pair until it fits on one screen on Nolvia's side-by-side view.
Set the Criteria Before the Candidates
Write the scoring criteria down before you generate anything. This is the step that turns an opinion into a measurement.
For text, five criteria cover most work: factual accuracy, instruction adherence, structure, concision, and tone fit. Score each on a three-point scale — fails, acceptable, good. Three points is deliberate: a five-point scale invites false precision and raters cannot reliably distinguish a 4 from a 5.
For images, use: prompt adherence, product or subject fidelity, artifact presence, lighting realism, and usability without retouching. Again three points, and define what "good" means in one sentence per criterion before you start.
Weighting matters less than consistency. If accuracy is the thing you actually care about, say so up front and let it dominate the total. Deciding the weights after seeing the results is how a test becomes a rationalisation.
Rater Mechanics
Three raters is the practical minimum, five is comfortable, and more than eight rarely changes the conclusion. Fewer than three and one strong opinion decides everything.
Give each rater the pairs in a different random order. If everyone rates pair 1 first, fatigue contaminates that pair across the whole sample.
Cap the session. Judgement quality falls off a cliff after about forty minutes, and a tired rater defaults to whichever output is longer or prettier. Twenty pairs per session is plenty; if you have more, split across days.
Do not let raters discuss pairs mid-session. Collect independently, then compare. Discussion before collection is how a group converges on the loudest person's view.
Record per rater, per pair, per criterion — a shared sheet works better than three people annotating screenshots of Nolvia outputs. Aggregate later. A single averaged score per pair hides the interesting case: four raters split evenly, which usually means the outputs are equivalent for your use case and you should choose on cost.
Two Protocols: Pairwise and Absolute
Pairwise comparison — A versus B, which is better on this criterion — is more reliable per judgement and the right default. People are better at comparing than at rating in the abstract.
Absolute scoring — rate each output on the criteria without seeing the alternative — is better when you have many candidates, because pairwise scales quadratically. Six models is fifteen pairs, which is manageable; twelve models is sixty-six, which is not.
The practical hybrid: absolute scoring to eliminate the bottom half, then pairwise among the survivors. Two rounds, and you spend the expensive judgements only where they matter.
A third protocol is worth knowing because it is the most honest for images: the forced choice under time pressure. Show each output for three seconds, ask which one a customer would stop scrolling for. It measures the thing that actually matters in a feed and it is immune to over-analysis.
Reading the Results Without Fooling Yourself
Small samples produce confident nonsense. With five raters and ten pairs, a consistent six-to-two split on one criterion is a signal. A five-to-three split is noise, and treating it as a result is how teams end up migrating workflows for no reason.
Three checks before you believe your own table:
- Did the winner win on the criterion you care about, or on the total? A model can lose on accuracy and win overall because it scored well on tone. Look at the per-criterion rows, not the summary.
- Is the margin bigger than the disagreement between raters? If raters disagree with each other more than the models differ, you have no result.
- Would the result survive a re-run? Run the same protocol again with fresh generations a week later. If the winner flips, the models are equivalent and your earlier conclusion was sampling noise.
Report the raw counts, not just the winner. "Seven of nine raters preferred A on adherence, four of nine on tone" is a useful sentence. "A is better" is not.
Where Blind Tests Still Mislead
Blinding fixes preference bias. It does not fix these:
The prompt is doing the work. A prompt tuned for one model will flatter that model. This is why the two-pair design matters — and why a single-prompt test should never be used to conclude that one model is fundamentally better.
The sample is not your workload. Twenty generic prompts tell you about generic prompts. If your real work is contract review or product photography, test on that.
Quality is not the only axis. Cost, latency, rate limits and reliability under load decide more real decisions than a two-point quality gap. A blind test that ignores them produces a correct answer to the wrong question.
Raters are not your customers. A designer spots the artifact your buyer will never notice. If the output is going in front of customers, the relevant rater is someone who thinks like one.
The last one bites hardest in marketing work, where the internal team's aesthetic and the audience's attention are not the same thing.
A Template You Can Run This Week
Pick one task you actually do weekly. Write five prompts that represent it. Generate each prompt on three candidate models. Re-export everything through one pipeline with names stripped to random codes. Build a sheet with one row per pair, five criteria, three-point scale. Hand it to three raters who do not know which models are in play, in three different random orders, in sessions under forty minutes.
Aggregate, then look at two things: which model wins on your primary criterion, and how much raters disagreed. If the disagreement is large, stop and test on your real workload before changing anything.
The whole exercise takes about two hours for one task, and it settles arguments that otherwise run for months. Running the candidates in one workspace makes the generation half of it trivial — on Nolvia you can put the same prompt through three different model families in a single session, then export the outputs for blinding, without managing three accounts and three billing pages.
Do it once a quarter. Model releases move fast enough that last quarter's winner is not this quarter's, and the protocol costs less than the meeting where someone argues from memory.
Try Every Major AI Model in One Workspace
Nolvia gives you GPT, Claude, Gemini, Grok and 40+ models in a single subscription — no API keys, no juggling five tabs.
Start Free on Nolvia
