Failure types

Incomplete

These checks fail when the model leaves out something the brief asked for.

Checks failed100 of 452
Results92
Models20
Tasks15
Dates2026-09-20 to 2026-11-09
What went wrong
Missing69
There when it should not be22
Drew it anyway when it could not work8
The wrong thing in its place1
How bad
Would not work5
Misleading88
Cosmetic7
Where a model ran a task 3 times or more
Failed every time16
Failed only sometimes29
The coffee machine taken out of the café's music, by GPT-6.1 Terra
GPT-6.1 TerraThe coffee machine taken out of the café's music · 2026-11-09 · 2 of 5 checksHum down, component extra, major; Rush down, component extra, majorAlso failed: Sounds right (false)
The coffee machine taken out of the café's music, by Kimi K3.1 Pro
Kimi K3.1 ProThe coffee machine taken out of the café's music · 2026-11-09 · 1 of 5 checksHum down, component extra, major; Rush down, component extra, majorAlso failed: Music kept (false); Sounds right (false)
The coffee machine taken out of the café's music, by Kimi K3.1 Pro
Kimi K3.1 ProThe coffee machine taken out of the café's music · 2026-11-09 · 1 of 5 checksHum down, component extra, major; Rush down, component extra, majorAlso failed: Music kept (false); Sounds right (false)
The coffee machine taken out of the café's music, by Gemini 4 Argon
Gemini 4 ArgonThe coffee machine taken out of the café's music · 2026-11-09 · 2 of 5 checksHum down, component extra, major; Rush down, component extra, majorAlso failed: Sounds right (false)
The coffee machine taken out of the café's music, by GPT-6.1 Sol
GPT-6.1 SolThe coffee machine taken out of the café's music · 2026-11-09 · 2 of 5 checksHum down, component extra, majorAlso failed: Music kept (false); Sounds right (false)
The coffee machine taken out of the café's music, by GPT-6.1 Sol
GPT-6.1 SolThe coffee machine taken out of the café's music · 2026-11-09 · 2 of 5 checksHum down, component extra, majorAlso failed: Music kept (false); Sounds right (false)
The coffee machine taken out of the café's music, by GPT-6.1 Terra
GPT-6.1 TerraThe coffee machine taken out of the café's music · 2026-11-09 · 2 of 5 checksHum down, component extra, major; Rush down, component extra, majorAlso failed: Sounds right (false)
The coffee machine taken out of the café's music, by Gemini 4 Argon
Gemini 4 ArgonThe coffee machine taken out of the café's music · 2026-11-09 · 2 of 5 checksHum down, component extra, major; Rush down, component extra, majorAlso failed: Sounds right (false)
The coffee machine taken out of the café's music, by Opus 5.5
Opus 5.5The coffee machine taken out of the café's music · 2026-11-09 · 3 of 5 checksRush down, component extra, majorAlso failed: Sounds right (false)
The coffee machine taken out of the café's music, by Kimi K3.1 Pro
Kimi K3.1 ProThe coffee machine taken out of the café's music · 2026-11-09 · 1 of 5 checksHum down, component extra, major; Rush down, component extra, majorAlso failed: Music kept (false); Sounds right (false)
The coffee machine taken out of the café's music, by Opus 5.5
Opus 5.5The coffee machine taken out of the café's music · 2026-11-09 · 3 of 5 checksRush down, component extra, majorAlso failed: Sounds right (false)
A serving tray where four tables will only take three drinks, by Claude Haiku 5.5
Claude Haiku 5.5A serving tray where four tables will only take three drinks · 2026-11-08 · 1 of 2 checksSays it cannot be done, annotation complied, major
Exploded portafilter, by DeepSeek 4.1 Flash
DeepSeek 4.1 FlashExploded portafilter · 2026-10-21 · 3 of 4 checks (1 undecided)All five parts, component missing, major
Exploded portafilter, by Opus 5.5
Opus 5.5Exploded portafilter · 2026-10-21 · 4 of 5 checksAll five parts, component missing, major
Exploded portafilter, by GLM 5.3 FlashX
GLM 5.3 FlashXExploded portafilter · 2026-10-21 · 2 of 4 checks (1 undecided)All five parts, component missing, majorAlso failed: Two assembly axes (unreadable)
Autopsy, asked if it is correct: Claude Opus 5.5's French press as a picture, by Opus 5.5
Opus 5.5Autopsy, asked if it is correct: Claude Opus 5.5's French press as a picture · 2026-10-19 · 1 of 2 checksFinds the flip, annotation missing, major
Repair: a Venn diagram with a drink on a circle's edge, by GLM 5.3 FlashX
GLM 5.3 FlashXRepair: a Venn diagram with a drink on a circle's edge · 2026-10-19 · 1 of 2 checksRepaired, deliverable missing, critical
Repair: a knitting chart whose bean knits too long, by Claude Haiku 5.5
Claude Haiku 5.5Repair: a knitting chart whose bean knits too long · 2026-10-19 · 1 of 2 checksRepaired, deliverable missing, critical
Repair: a Venn diagram with a drink on a circle's edge, by Kimi K3
Kimi K3Repair: a Venn diagram with a drink on a circle's edge · 2026-10-19 · 1 of 2 checksRepaired, deliverable missing, critical
Autopsy, asked if it is correct: Claude Opus 5.5's French press as a picture, by Opus 5.5
Opus 5.5Autopsy, asked if it is correct: Claude Opus 5.5's French press as a picture · 2026-10-19 · 1 of 2 checksInvents nothing, annotation extra, major
Autopsy, asked if it is correct: Claude Opus 5.5's French press as a picture (control), by Opus 5.5
Opus 5.5Autopsy, asked if it is correct: Claude Opus 5.5's French press as a picture (control) · 2026-10-19 · 0 of 1 checksInvents nothing, annotation extra, major
Autopsy, asked if it is correct: Claude Opus 5.5's French press as a picture (control), by Opus 5.5
Opus 5.5Autopsy, asked if it is correct: Claude Opus 5.5's French press as a picture (control) · 2026-10-19 · 0 of 1 checksInvents nothing, annotation extra, major
Repair: a knitting chart whose bean knits too long, by Claude Haiku 5.5
Claude Haiku 5.5Repair: a knitting chart whose bean knits too long · 2026-10-19 · 1 of 2 checksRepaired, deliverable missing, critical
Autopsy, asked if it is correct: Claude Opus 5.5's French press as a picture (control), by Opus 5.5
Opus 5.5Autopsy, asked if it is correct: Claude Opus 5.5's French press as a picture (control) · 2026-10-19 · 0 of 1 checksInvents nothing, annotation extra, major
Exploded French press, by Nemotron 3.5 Super VL
Nemotron 3.5 Super VLExploded French press · 2026-10-16 · 2 of 5 checksAll seven parts, component missing, majorAlso failed: Stack order (false); Filter fits the glass (false)
Exploded French press, by Nemotron 3.5 Super VL
Nemotron 3.5 Super VLExploded French press · 2026-10-16 · 1 of 5 checksAll seven parts, component missing, majorAlso failed: Stack order (false); Filter fits the glass (false); One assembly axis (unreadable)
Exploded French press, by Nemotron 3.5 Super VL
Nemotron 3.5 Super VLExploded French press · 2026-10-16 · 0 of 5 checksAll seven parts, component missing, majorAlso failed: Stack order (false); Filter fits the glass (false); One assembly axis (unreadable); Labels match (false)
Beans leaving a grinder hopper, in MuJoCo, by GPT-6 Luna
GPT-6 LunaBeans leaving a grinder hopper, in MuJoCo · 2026-10-15 · 1 of 6 checksmodel.xml written, deliverable missing, major, cause: repair; its note said this very thing was rightAlso failed: Fifty two-part beans (built wrong); Gravity down (false); Beans start inside a hollow hopper (built wrong); Held, then flows (broken)No model.xml beside the script, so the checker has no model to read. The hopper is about a metre tall, and 44 beans are out by 1 s.
Exploded French press, by Step 5 Preview
Step 5 PreviewExploded French press · 2026-10-09 · 3 of 5 checksAll seven parts, component missing, major; its note said nothing was wrongAlso failed: Stack order (false)
Exploded French press, by Step 5 Preview
Step 5 PreviewExploded French press · 2026-10-09 · 4 of 5 checksAll seven parts, component missing, major; its note said nothing was wrong
Exploded French press, by Step 5 Preview
Step 5 PreviewExploded French press · 2026-10-09 · 3 of 5 checksAll seven parts, component missing, major, cause: clock; its note left no closing noteAlso failed: Stack order (false)
A five-set Venn in circles, which cannot be drawn, by Pareto 26.10 Preview
Pareto 26.10 PreviewA five-set Venn in circles, which cannot be drawn · 2026-10-09 · 2 of 3 checksNames a fix that works, annotation complied, major, cause: conflict; its note said this very thing was right
A five-set Venn in circles, which cannot be drawn, by GPT-6.1 Sol
GPT-6.1 SolA five-set Venn in circles, which cannot be drawn · 2026-10-09 · 2 of 3 checksNames a fix that works, annotation complied, major, cause: conflict; its note said it fell short, without naming this
A five-set Venn in circles, which cannot be drawn, by GPT-6.1 Sol
GPT-6.1 SolA five-set Venn in circles, which cannot be drawn · 2026-10-09 · 2 of 3 checksNames a fix that works, annotation complied, major, cause: conflict; its note said it fell short, without naming this
A five-set Venn in circles, which cannot be drawn, by GPT-6.1 Sol
GPT-6.1 SolA five-set Venn in circles, which cannot be drawn · 2026-10-09 · 2 of 3 checksNames a fix that works, annotation complied, major, cause: conflict; its note said it fell short, without naming this
A five-set Venn in circles, which cannot be drawn, by GPT-6 Luna
GPT-6 LunaA five-set Venn in circles, which cannot be drawn · 2026-10-09 · 2 of 3 checksNames a fix that works, annotation complied, major, cause: conflict; its note said it fell short, without naming this
A five-set Venn in circles, which cannot be drawn, by GPT-6 Luna
GPT-6 LunaA five-set Venn in circles, which cannot be drawn · 2026-10-09 · 2 of 3 checksNames a fix that works, annotation complied, major, cause: conflict; its note said it fell short, without naming this
A five-set Venn in circles, which cannot be drawn, by GPT-6 Luna
GPT-6 LunaA five-set Venn in circles, which cannot be drawn · 2026-10-09 · 2 of 3 checksNames a fix that works, annotation complied, major, cause: conflict; its note said it fell short, without naming this
Exploded French press, by Claude Haiku 5.5
Claude Haiku 5.5Exploded French press · 2026-10-08 · 0 of 5 checksAll seven parts, component missing, major; its note said nothing was wrongAlso failed: Stack order (false); Filter fits the glass (false); One assembly axis (unreadable); Labels match (false)
Exploded French press, by Claude Haiku 5.5
Claude Haiku 5.5Exploded French press · 2026-10-08 · 3 of 5 checksAll seven parts, component missing, major; its note said nothing was wrongAlso failed: Stack order (false)
Exploded French press, by Claude Haiku 5.5
Claude Haiku 5.5Exploded French press · 2026-10-08 · 1 of 5 checksAll seven parts, component missing, major; its note said nothing was wrongAlso failed: Stack order (false); Filter fits the glass (false); Labels match (false)
Espresso Rag, with the left hand crossing over, by Claude Haiku 5.5
Claude Haiku 5.5Espresso Rag, with the left hand crossing over · 2026-10-08 · 4 of 6 checksRight hand keeps going, component missing, major, a near miss; its note named the fault; Crossing marked, annotation missing, minor; its note said nothing was wrong
Espresso Rag, with the left hand crossing over, by Claude Haiku 5.5
Claude Haiku 5.5Espresso Rag, with the left hand crossing over · 2026-10-08 · 5 of 6 checksCrossing marked, annotation missing, minor; its note said nothing was wrong
Espresso Rag, with the left hand crossing over, by Claude Haiku 5.5
Claude Haiku 5.5Espresso Rag, with the left hand crossing over · 2026-10-08 · 5 of 6 checksCrossing marked, annotation missing, minor; its note said nothing was wrong
Exploded AeroPress, by Claude Haiku 5.5
Claude Haiku 5.5Exploded AeroPress · 2026-10-08 · 2 of 4 checks (1 undecided)All five parts, component missing, major; its note said nothing was wrongAlso failed: Parts fit (false)
Exploded AeroPress, by Claude Haiku 5.5
Claude Haiku 5.5Exploded AeroPress · 2026-10-08 · 3 of 5 checksAll five parts, component missing, major; its note said nothing was wrongAlso failed: Stack order (false)
Exploded hand grinder, by Claude Haiku 5.5
Claude Haiku 5.5Exploded hand grinder · 2026-10-08 · 3 of 5 checksAll eight parts, component missing, major; its note said nothing was wrongAlso failed: Stack order (false)
Exploded hand grinder, by Claude Haiku 5.5
Claude Haiku 5.5Exploded hand grinder · 2026-10-08 · 2 of 5 checksAll eight parts, component missing, major; its note said nothing was wrongAlso failed: Stack order (false); Parts fit (false)
Exploded hand grinder, by Claude Haiku 5.5
Claude Haiku 5.5Exploded hand grinder · 2026-10-08 · 2 of 5 checksAll eight parts, component missing, major; its note said nothing was wrongAlso failed: Stack order (false); Parts fit (false)
Exploded portafilter, by Claude Haiku 5.5
Claude Haiku 5.5Exploded portafilter · 2026-10-08 · 2 of 4 checks (1 undecided)All five parts, component missing, major; its note said nothing was wrongAlso failed: Stack order (false)
Exploded portafilter, by Claude Haiku 5.5
Claude Haiku 5.5Exploded portafilter · 2026-10-08 · 2 of 5 checksAll five parts, component missing, major; its note said nothing was wrongAlso failed: Stack order (false); Parts fit (false)
Exploded portafilter, by Claude Haiku 5.5
Claude Haiku 5.5Exploded portafilter · 2026-10-08 · 2 of 5 checksAll five parts, component missing, major; its note said nothing was wrongAlso failed: Stack order (false); Parts fit (false)
Exploded French press, by Mistral Large 4
Mistral Large 4Exploded French press · 2026-10-07 · 4 of 5 checksAll seven parts, component missing, major; its note said this very thing was right
Exploded AeroPress, by Pareto 26.10 Preview
Pareto 26.10 PreviewExploded AeroPress · 2026-10-07 · 2 of 5 checksAll five parts, component missing, major; its note said this very thing was rightAlso failed: Stack order (false); Parts fit (false)
Exploded AeroPress, by Pareto 26.10 Preview
Pareto 26.10 PreviewExploded AeroPress · 2026-10-07 · 3 of 5 checksAll five parts, component missing, major; its note said this very thing was rightAlso failed: Stack order (false)
Exploded AeroPress, by Pareto 26.10 Preview
Pareto 26.10 PreviewExploded AeroPress · 2026-10-07 · 1 of 5 checksAll five parts, component missing, major; its note said nothing was wrongAlso failed: Stack order (false); Parts fit (false); One assembly axis (unreadable)
Exploded portafilter, by Mistral Large 4
Mistral Large 4Exploded portafilter · 2026-10-07 · 3 of 5 checksAll five parts, component missing, major; its note said this very thing was rightAlso failed: Stack order (false)
Espresso Rag, with the left hand crossing over, by GPT-6 Luna
GPT-6 LunaEspresso Rag, with the left hand crossing over · 2026-10-05 · 4 of 6 checksCrossing marked, annotation missing, minor, a near miss; its note said this very thing was rightAlso failed: Stems apart (unreadable)
Espresso Rag, with the left hand crossing over, by GPT-6 Luna
GPT-6 LunaEspresso Rag, with the left hand crossing over · 2026-10-05 · 4 of 6 checksCrossing marked, annotation missing, minor, a near miss; its note said nothing was wrongAlso failed: Stems apart (unreadable)
Espresso Rag, with the left hand crossing over, by GPT-6 Luna
GPT-6 LunaEspresso Rag, with the left hand crossing over · 2026-10-05 · 4 of 6 checksCrossing marked, annotation missing, minor; its note said nothing either wayAlso failed: Left hand crosses (false)
Espresso Rag, with the left hand crossing over, by Pareto 26.10 Preview
Pareto 26.10 PreviewEspresso Rag, with the left hand crossing over · 2026-10-05 · 4 of 6 checksCrossing marked, annotation missing, minor, a near miss; its note said this very thing was rightAlso failed: Stems apart (unreadable)
Exploded AeroPress, by GPT-6 Luna
GPT-6 LunaExploded AeroPress · 2026-10-05 · 3 of 5 checksAll five parts, component missing, major; its note said nothing was wrongAlso failed: Parts fit (false)
Exploded AeroPress, by GPT-6 Luna
GPT-6 LunaExploded AeroPress · 2026-10-05 · 3 of 5 checksAll five parts, component missing, major; its note said nothing was wrongAlso failed: Parts fit (false)
Exploded hand grinder, by GPT-6 Luna
GPT-6 LunaExploded hand grinder · 2026-10-05 · 2 of 5 checksAll eight parts, component missing, major; its note said nothing either wayAlso failed: Stack order (false); Parts fit (false)
Exploded hand grinder, by GPT-6 Luna
GPT-6 LunaExploded hand grinder · 2026-10-05 · 3 of 5 checksAll eight parts, component missing, major; its note said nothing was wrongAlso failed: Stack order (false)
Exploded hand grinder, by GPT-6 Luna
GPT-6 LunaExploded hand grinder · 2026-10-05 · 3 of 5 checksAll eight parts, component missing, major; its note said nothing was wrongAlso failed: Parts fit (false)
Exploded portafilter, by GPT-6 Luna
GPT-6 LunaExploded portafilter · 2026-10-05 · 3 of 4 checks (1 undecided)All five parts, component missing, major; its note said nothing was wrong
Exploded portafilter, by GPT-6 Luna
GPT-6 LunaExploded portafilter · 2026-10-05 · 3 of 4 checks (1 undecided)All five parts, component missing, major; its note said nothing was wrong
Exploded French press, by Ember 1
Ember 1Exploded French press · 2026-10-04 · 2 of 5 checksAll seven parts, component missing, major; its note said this very thing was rightAlso failed: Stack order (false); Filter fits the glass (false)
Exploded French press, by Ember 1
Ember 1Exploded French press · 2026-10-04 · 3 of 4 checks (1 undecided)All seven parts, component missing, major; its note said this very thing was right
Exploded French press, by Kimi K3
Kimi K3Exploded French press · 2026-10-04 · 2 of 4 checks (1 undecided)All seven parts, component missing, major; its note said this very thing was rightAlso failed: Filter fits the glass (false)
Exploded French press, by GPT-6 Luna
GPT-6 LunaExploded French press · 2026-10-03 · 2 of 5 checksAll seven parts, component missing, major; its note said this very thing was rightAlso failed: Stack order (false); Filter fits the glass (false)
Exploded French press, by DeepSeek 4.1 Flash
DeepSeek 4.1 FlashExploded French press · 2026-10-03 · 3 of 5 checksAll seven parts, component missing, major; its note said this very thing was rightAlso failed: Stack order (false)
Exploded French press, by Pareto 26.10 Preview
Pareto 26.10 PreviewExploded French press · 2026-10-02 · 3 of 5 checksAll seven parts, component missing, major; its note said this very thing was rightAlso failed: Stack order (false)
Exploded French press, by Pareto
ParetoExploded French press · 2026-10-01 · 3 of 5 checksAll seven parts, component missing, major; its note said this very thing was rightAlso failed: Stack order (false)
Exploded French press, by Claude Sonnet 5.5
Claude Sonnet 5.5Exploded French press · 2026-09-29 · 3 of 5 checksAll seven parts, component missing, major; its note named the faultAlso failed: Stack order (false)
A three-set coffee Venn diagram, by Claude Sonnet 5.5
Claude Sonnet 5.5A three-set coffee Venn diagram · 2026-09-29 · 1 of 4 checksSeven regions, three names, relation missing, critical; its note said this very thing was rightAlso failed: Every drink in its region (false); No label on an edge (unreadable)Overlaping labels with lines.
Exploded French press, by Perceptron Mk1.5
Perceptron Mk1.5Exploded French press · 2026-09-27 · 0 of 5 checksAll seven parts, component missing, major; its note said this very thing was rightAlso failed: Stack order (false); Filter fits the glass (false); One assembly axis (unreadable); Labels match (false)Terrible
Exploded French press, by Perceptron Mk1.5
Perceptron Mk1.5Exploded French press · 2026-09-27 · 0 of 5 checksAll seven parts, component missing, major; its note left no closing noteAlso failed: Stack order (false); Filter fits the glass (false); One assembly axis (unreadable); Labels match (false)Not great and off axis
Exploded French press, by Perceptron Mk1.5
Perceptron Mk1.5Exploded French press · 2026-09-27 · 0 of 5 checksAll seven parts, component missing, major; its note said this very thing was rightAlso failed: Stack order (false); Filter fits the glass (false); One assembly axis (unreadable); Labels match (false)
Exploded French press, by Command A+
Command A+Exploded French press · 2026-09-27 · 0 of 5 checksAll seven parts, component missing, major; its note said this very thing was rightAlso failed: Stack order (false); Filter fits the glass (false); One assembly axis (unreadable); Labels match (false)
Exploded French press, by Command A+
Command A+Exploded French press · 2026-09-27 · 0 of 5 checksAll seven parts, component missing, major; its note said this very thing was rightAlso failed: Stack order (false); Filter fits the glass (false); One assembly axis (unreadable); Labels match (false)
Exploded French press, by Command A+
Command A+Exploded French press · 2026-09-27 · 0 of 5 checksAll seven parts, component missing, major; its note said this very thing was rightAlso failed: Stack order (false); Filter fits the glass (false); One assembly axis (unreadable); Labels match (false)Terrible
Exploded French press, by GLM 5.3 FlashX
GLM 5.3 FlashXExploded French press · 2026-09-22 · 2 of 5 checksAll seven parts, component missing, major; its note said this very thing was rightAlso failed: Stack order (false); Filter fits the glass (false)A plate missing, the cross-plate above the mesh, the lid below the filter, and plates wider than the carafe.
Exploded French press, by Qwen3.8 27B
Qwen3.8 27BExploded French press · 2026-09-22 · 3 of 5 checksAll seven parts, component missing, major; its note said this very thing was rightAlso failed: Stack order (false)No spiral plate and no separate knob, and the cross retainer sits above the mesh.
Exploded French press, by Qwen3.8 27B
Qwen3.8 27BExploded French press · 2026-09-22 · 2 of 5 checksAll seven parts, component missing, major; its note said this very thing was rightAlso failed: Stack order (false); Filter fits the glass (false)A coil spring and a mesh, no cross-plate, the lid under the filter, and a mesh wider than the carafe.
Exploded French press, by Qwen3.8 27B
Qwen3.8 27BExploded French press · 2026-09-22 · 3 of 4 checks (1 undecided)All seven parts, component missing, major; its note said this very thing was rightMesh and a flange disc, no spiral plate or cross-plate. Cut off after round 2.
A run that spells COFFEE, by GLM 5.3 FlashX
GLM 5.3 FlashXA run that spells COFFEE · 2026-09-21 · 3 of 5 checksOne continuous route, relation missing, major; its note said this very thing was rightAlso failed: Reads as COFFEE (unreadable)20.5 km of boxes, with two diagonals cut straight across the blocks, and no word.
A run that spells COFFEE, by GLM 5.3 FlashX
GLM 5.3 FlashXA run that spells COFFEE · 2026-09-21 · 3 of 5 checksOne continuous route, relation missing, major; its note said nothing was wrongAlso failed: Reads as COFFEE (unreadable)Twenty-one separate pieces of street, each one a stroke, with nothing joining them into a run.
Exploded French press, by GLM 5.3 FlashX
GLM 5.3 FlashXExploded French press · 2026-09-20 · 3 of 5 checksAll seven parts, component substituted, major; its note said nothing was wrongAlso failed: Stack order (false)Left the cross-plate out altogether, and still put the spiral plate under the mesh.
Exploded French press, by Qwen3.8 27B
Qwen3.8 27BExploded French press · 2026-09-20 · 3 of 4 checks (1 undecided)All seven parts, component missing, major; its note said this very thing was rightMesh drawn, spiral plate and cross-plate missing, and a frame smaller than the carafe it holds.
A run that spells COFFEE, by GLM 5.3 FlashX
GLM 5.3 FlashXA run that spells COFFEE · 2026-09-20 · 0 of 5 checksOne continuous route, relation missing, major; its note said this very thing was rightAlso failed: On real streets (false); Inside the grid (false); Metadata true (false); Reads as COFFEE (unreadable)Drawn north-up across the blocks and out of the grid entirely, and still not readable as COFFEE.

The other kinds: malformed · false · unreadable · broken · built wrong

All of them, with counts and the codebook · How a check is decided