Failure types

Every failed check, sorted by what broke.

Every check is coded when it is written: what kind of quality broke, what happened to it, and how bad it is. So a check fails the same way whichever model fails it. 430 of 1922 checks have failed so far.

The most common kind is false, with 205 of the 430 failures. Models rarely see their own faults: of 261 failures with a closing note, the model named the fault in 22, and in 80 it said that very thing was right. And most misses are far off, since only 30 of 95 measured failures were off by one swap or one label.

By kind of quality

kind	fails when	failed
malformed	the model's file won't open: it doesn't compile, parse or render	1 of 135
false	the model gets a fact wrong, such as a part in the wrong place or at the wrong size	205 of 855
incomplete	the model leaves out something the brief asked for	100 of 452
unreadable	the content may be right, but a reader can't make it out	90 of 285
broken	the model's file runs, but doesn't do what it should	7 of 94
built wrong	the result may look right, but the model didn't make it the way the brief said	27 of 101

By model

Failed checks of the ones each model had, in red when every one failed. A dot means it had none of that kind.

model	malformed	false	incomplete	unreadable	broken	built wrong
Opus 5.5	0 of 11	7 of 55	8 of 39	4 of 14	·	1 of 3
GPT-6.1 Sol	0 of 5	6 of 77	5 of 43	0 of 20	0 of 9	0 of 5
Gemini 4 Argon	0 of 6	4 of 15	4 of 9	0 of 3	·	0 of 3
Kimi K3.1 Pro	0 of 9	18 of 42	6 of 21	2 of 9	0 of 3	0 of 9
GPT-6.1 Terra	0 of 3	3 of 6	4 of 6	·	·	·
GLM 5.3 FlashX	1 of 6	21 of 72	7 of 29	10 of 20	0 of 6	2 of 6
DeepSeek 4.1 Flash	0 of 16	8 of 106	2 of 49	8 of 25	3 of 37	3 of 15
GPT-6 Luna	0 of 12	16 of 75	15 of 48	5 of 24	3 of 3	6 of 9
Claude Sonnet 5.5	0 of 6	4 of 45	2 of 18	2 of 12	0 of 6	2 of 6
Claude Haiku 5.5	0 of 12	27 of 84	18 of 45	4 of 24	1 of 3	0 of 9
Kimi K3	0 of 3	1 of 8	2 of 7	3 of 4	·	·
Pareto 26.9	0 of 3	0 of 3	0 of 11	·	0 of 3	0 of 6
Nemotron 3.5 Super VL	·	10 of 12	3 of 3	5 of 6	·	0 of 3
Muse Spark Open	·	6 of 9	0 of 2	2 of 5	·	3 of 3
Muse Wonder	0 of 3	3 of 12	0 of 6	0 of 6	·	·
Qwen3.8 Max Prime	·	4 of 5	0 of 3	4 of 5	·	2 of 5
Pareto 26.10 Preview	0 of 3	12 of 61	6 of 33	3 of 19	0 of 12	0 of 1
Step 5 Preview	0 of 3	4 of 12	3 of 6	3 of 6	·	·
Mistral Large 4	·	2 of 9	2 of 3	0 of 3	·	·
Ember 1	0 of 3	2 of 8	2 of 5	3 of 5	·	·
GPT-6 Sol	0 of 13	0 of 16	0 of 14	2 of 14	·	·
Grok 4.7	0 of 3	4 of 6	0 of 4	3 of 4	·	·
Gemini 3.8 Flash	0 of 3	2 of 8	0 of 4	2 of 6	·	2 of 2
Pareto	·	4 of 19	1 of 6	0 of 7	·	0 of 1
GPT-5.6 Sol	·	5 of 14	0 of 4	2 of 6	·	1 of 2
Qwen3.8 27B	·	4 of 12	4 of 4	0 of 6	·	0 of 2
Qwen3.8 Max	·	0 of 21	·	1 of 1	·	0 of 1
MiMo v2.6 Pro	0 of 3	1 of 3	0 of 3	3 of 3	·	·
Perceptron Mk1.5	0 of 3	15 of 15	3 of 6	9 of 9	·	3 of 3
Command A+	0 of 3	12 of 15	3 of 6	9 of 9	·	1 of 3
Qwen3.8 Flash	0 of 3	0 of 7	0 of 6	1 of 9	·	1 of 4
Astra	·	·	0 of 8	·	0 of 12	·
Union Alpha	·	0 of 3	0 of 1	0 of 1	·	·

Why it failed

A check can't say why it failed. I name a cause only when something else shows it, which covers 58 of 430 failures.

knowledge	it gets the same fact wrong when asked in words, so it does not know it	7
transfer	it gets the fact right in words, and wrong in the file	5
repair	an earlier round had it right, and a later one broke it	14
conflict	the brief cannot be met as written	7
clock	the clock ran out mid-work, so the round scored may not be its last word	25

Did it see it?

Each run was told to look at its picture and end with what it thought was still wrong. Claude Opus 5.5 read those notes on 2026-10-13.

named the fault	22
said it fell short, without naming this	10
said nothing either way	9
said nothing was wrong	140
said this very thing was right	80
left no closing note	55

By model, counting only failures where the run left a note. Most models have few, so these are counts and not rates.

model	named the fault	said it was right
Claude Haiku 5.5	7 of 46	3 of 46
GPT-6 Luna	2 of 45	6 of 45
GLM 5.3 FlashX	2 of 29	11 of 29
Command A+	0 of 25	12 of 25
Pareto 26.10 Preview	1 of 21	7 of 21
Perceptron Mk1.5	1 of 12	6 of 12
Opus 5.5	4 of 10	3 of 10
Claude Sonnet 5.5	3 of 8	3 of 8
DeepSeek 4.1 Flash	0 of 8	6 of 8
GPT-5.6 Sol	0 of 8	2 of 8
Step 5 Preview	0 of 8	3 of 8
Ember 1	1 of 7	4 of 7
Grok 4.7	0 of 7	4 of 7
Qwen3.8 27B	0 of 7	4 of 7
GPT-6.1 Sol	1 of 5	0 of 5
Kimi K3	0 of 5	2 of 5
Mistral Large 4	0 of 3	2 of 3
Pareto	0 of 3	1 of 3
GPT-6 Sol	0 of 2	0 of 2
Qwen3.8 Max Prime	0 of 2	1 of 2

The rules for reading a note

named the fault: The note says the fault this check failed on is there, or says it is unsure about that very thing. A disclaimer that covers it counts, such as 'not to scale' for a widths check. A caveat about something else does not.
said it fell short, without naming this: The note says the result falls short overall without naming this fault.
said nothing either way: The note says none of these.
said nothing was wrong: The note says nothing is wrong, or that it looks right, or lists other faults and leaves this one out, without speaking to this check.
said this very thing was right: The note says the very thing this check tests is right: 'every drink in the right region', 'correct stacking order', 'all parts present'. Listing the parts it drew is not a claim that they are all there.
left no closing note: There is no closing note: the run ended mid-work, on the clock or a provider error.

Keep in mind

213 of the 430 failures are on checks I judge by eye, alone, and 4 on checks a model judged. 37 of those were judged without seeing which model made them.
Severity is my call, set once per check in the codebook.
15 of 532 runs left no file, so no check counts them.
8 failures can only follow from another one on the same result, so they are counted twice.
1 failure has its mode changed by hand, with the reason in the data.

The codebook

Every check, with its four codes

Properties: component a part, an object or a set; relation order, connection, membership or containment; geometry position, size, proportion or shape; quantity a count or a value; annotation labels, words, keys and marks; behaviour what it does over time when run; construction how it was built; deliverable the file as a whole.

task	check	dimension	property	mode	severity
French press	All seven parts (eye)	incomplete	component	missing	major
	Stack order	false	relation	transposed	critical
	Filter fits the glass	false	geometry	distorted	major
	One assembly axis (eye)	unreadable	geometry	inconsistent	minor
	Labels match (eye)	false	annotation	misbound	major
AeroPress	All five parts (eye)	incomplete	component	missing	major
	Stack order	false	relation	transposed	critical
	Parts fit	false	geometry	distorted	major
	One assembly axis (eye)	unreadable	geometry	inconsistent	minor
	Labels match (eye)	false	annotation	misbound	major
V60	All three parts (eye)	incomplete	component	missing	major
	Stack order	false	relation	transposed	critical
	Parts fit (eye)	false	geometry	distorted	major
	One assembly axis (eye)	unreadable	geometry	inconsistent	minor
	Labels match (eye)	false	annotation	misbound	major
Portafilter	All five parts (eye)	incomplete	component	missing	major
	Stack order	false	relation	transposed	critical
	Parts fit (eye)	false	geometry	distorted	major
	Two assembly axes (eye)	unreadable	geometry	inconsistent	minor
	Labels match (eye)	false	annotation	misbound	major
Grinder	All eight parts (eye)	incomplete	component	missing	major
	Stack order	false	relation	transposed	critical
	Parts fit (eye)	false	geometry	distorted	major
	One assembly axis (eye)	unreadable	geometry	inconsistent	minor
	Labels match (eye)	false	annotation	misbound	major
COFFEE route	On real streets	false	geometry	wrong	major
	One continuous route	incomplete	relation	missing	major
	Inside the grid	false	geometry	wrong	major
	Metadata true (eye)	false	annotation	wrong	major
	Reads as COFFEE (eye)	unreadable	geometry	unclear	critical
Caffeine	C8H10N4O2	false	quantity	wrong	critical
	Fused purine core	false	relation	wrong	critical
	Methyls on N1, N3, N7	false	relation	transposed	critical
	Carbonyls at C2 and C6	false	relation	transposed	critical
	Flat rings, real bond lengths	false	geometry	distorted	major
Boiler	Complete analogue	incomplete	component	missing	critical
	Real hysteresis	broken	behaviour	missing	critical
	Starts cold	broken	behaviour	wrong	major
	Holds 92–96 °C	broken	behaviour	wrong	critical
	Writes the .txt	incomplete	deliverable	missing	minor
Café table	All five objects (eye)	incomplete	component	missing	major
	Primitives only	built wrong	construction	wrong	major
	Spoon reads as a spoon (eye)	unreadable	geometry	unclear	minor
	Croissant reads as a croissant (eye)	unreadable	geometry	unclear	minor
	Consistent scale	false	geometry	inconsistent	major
Fluorocaffeine	Parses	malformed	deliverable	wrong	critical
	C8HF9N4O2	false	quantity	wrong	critical
	Three N-CF3, no N-methyl	false	relation	transposed	critical
	The right molecule	false	relation	wrong	critical
Venn	Compiles	malformed	deliverable	wrong	critical
	Seven regions, three names	incomplete	relation	missing	critical
	Every drink in its region	false	relation	misbound	major
	No label on an edge	unreadable	geometry	collides	minor
Bean cosy	A real chart	malformed	deliverable	wrong	critical
	Stitches wider than tall	false	geometry	distorted	major
	Bean knits true	false	geometry	distorted	major
	Has a key	incomplete	annotation	missing	minor
Five circles	Says circles cannot	incomplete	annotation	complied	major
	Five sets kept	incomplete	component	missing	major
	Names a fix that works (eye)	incomplete	annotation	complied	major
Triangle wave	DC supplies only	built wrong	construction	wrong	critical
	1 kHz	broken	behaviour	wrong	critical
	2 V peak to peak	broken	behaviour	wrong	major
	Straight, even ramps	broken	behaviour	distorted	major
Café order	Renders, labels valid	malformed	deliverable	wrong	critical
	Happy path	broken	behaviour	missing	critical
	No refund if unpaid	broken	behaviour	extra	critical
	Refund if paid	broken	behaviour	missing	critical
	No cancel after collect	broken	behaviour	extra	major
Pruned shrub	Reads as a shrub (eye)	unreadable	geometry	unclear	major
	Flat side from the rules (eye)	built wrong	construction	wrong	major
	Sensible size (eye)	false	quantity	wrong	minor
Penrose floor	Two rhombi, two fills (eye)	false	component	substituted	critical
	No repeat (eye)	false	relation	wrong	critical
	Fivefold, edges drawn (eye)	false	geometry	wrong	major
Espresso Rag	Piano staff	incomplete	component	missing	critical
	Sixteen bars	false	quantity	wrong	major
	Left hand crosses	false	relation	misencoded	critical
	Right hand keeps going	incomplete	component	missing	major
	Crossing marked	incomplete	annotation	missing	minor
	Stems apart	unreadable	annotation	misencoded	minor
Coffee tree	Valid Newick	malformed	deliverable	wrong	critical
	Twelve cultivars	incomplete	quantity	wrong	critical
	Every tip level	false	geometry	distorted	critical
	One split of four	false	relation	wrong	critical
Hopper	Runs in MuJoCo	malformed	deliverable	wrong	critical
	model.xml written	incomplete	deliverable	missing	major
	Fifty two-part beans	built wrong	component	substituted	major
	Gravity down	false	quantity	wrong	critical
	Beans start inside a hollow hopper	built wrong	construction	wrong	critical
	Held, then flows	broken	behaviour	wrong	critical
Stirring spoon	HTML and CSS only	built wrong	construction	substituted	major
	A CSS loop that moves something	broken	behaviour	missing	critical
	Reads as a cup seen into (eye)	unreadable	geometry	distorted	major
	The spoon does the stirring (eye)	broken	behaviour	missing	critical
	Reads as inside the cup (eye)	false	relation	transposed	major
	Handle stays on the bowl (eye)	false	geometry	distorted	major
V60 repair	Repaired	incomplete	deliverable	missing	critical
	Broke nothing else	false	construction	wrong	major
Spoon repair	In the drink (eye)	false	relation	transposed	major
	Kept the brief and the loop	false	construction	wrong	major
Venn repair	Repaired	incomplete	deliverable	missing	critical
	Broke nothing else	false	construction	wrong	major
Knitting repair	Repaired	incomplete	deliverable	missing	critical
	Broke nothing else	false	construction	wrong	major
Verdict, good press	Invents nothing	incomplete	annotation	extra	major
Verdict, flipped	Finds the flip	incomplete	annotation	missing	major
	Invents nothing	incomplete	annotation	extra	major
Serve the tray	Serves all six	false	relation	wrong	critical
Four tables, three drinks	Says it cannot be done	incomplete	annotation	complied	major
	Reuses nothing	false	relation	transposed	major
Two tables, black coffee	Serves all six	false	relation	wrong	critical
Pump noise	Same file	malformed	deliverable	wrong	critical
	Hum down	incomplete	component	extra	major
	Rush down	incomplete	component	extra	major
	Music kept	false	quantity	distorted	major
	Sounds right (eye)	false	deliverable	distorted	major

Where the codes come from

Lindland, O. I., Sindre, G. and Sølvberg, A. (1994). Understanding quality in conceptual modeling. IEEE Software 11(2), 42-49.Malformed, false, incomplete, unreadable.
ISO/IEC 25010:2023. Systems and software quality models.Broken, kept apart from incomplete.
Damerau, F. J. (1964). A technique for computer detection and correction of spelling errors. Communications of the ACM 7(3), 171-176.Missing, extra, substituted, transposed.
MQM Council. The MQM Scoring Models. Multidimensional Quality Metrics (MQM).Severity as its own scale.
Chillarege, R. et al. (1992). Orthogonal Defect Classification. IEEE Transactions on Software Engineering 18(11), 943-956.One code per question.

How a check is decided in the first place