Broken

These checks fail when the model's file runs, but doesn't do what it should.

What went wrong
Checks failed	7 of 94
Results	7
Models	3
Tasks	2
Dates	2026-10-15 to 2026-10-19
Wrong in some other way	4
Missing	3
How bad
Would not work	7
Where a model ran a task 3 times or more
Failed every time	2
Failed only sometimes	1

A spoon stirring in a cup, in HTML and CSS, by DeepSeek 4.1 Flash — DeepSeek 4.1 FlashA spoon stirring in a cup, in HTML and CSS · 2026-10-19 · 3 of 6 checksThe spoon does the stirring, behaviour missing, critical, cause: clockAlso failed: Reads as inside the cup (false); Handle stays on the bowl (false)

A spoon stirring in a cup, in HTML and CSS, by DeepSeek 4.1 Flash — DeepSeek 4.1 FlashA spoon stirring in a cup, in HTML and CSS · 2026-10-19 · 4 of 5 checks (1 undecided)The spoon does the stirring, behaviour missing, critical, cause: clock

A spoon stirring in a cup, in HTML and CSS, by DeepSeek 4.1 Flash — DeepSeek 4.1 FlashA spoon stirring in a cup, in HTML and CSS · 2026-10-19 · 5 of 6 checksThe spoon does the stirring, behaviour missing, critical

Beans leaving a grinder hopper, in MuJoCo, by Haiku 5.5 — Haiku 5.5Beans leaving a grinder hopper, in MuJoCo · 2026-10-15 · 5 of 6 checksHeld, then flows, behaviour wrong, critical, cause: clock; its note left no closing noteOnly 10 of the 50 beans get out by 5 s, and beans 19, 20 and 24 fall through the tray and out of the model.

Beans leaving a grinder hopper, in MuJoCo, by GPT-6 Luna — GPT-6 LunaBeans leaving a grinder hopper, in MuJoCo · 2026-10-15 · 5 of 6 checksHeld, then flows, behaviour wrong, critical; its note said nothing was wrongThe gate opens at 1 s, but beans stay packed in the throat and nothing gets out until 3.3 s. It reports 2 mm of wall penetration.

Beans leaving a grinder hopper, in MuJoCo, by GPT-6 Luna — GPT-6 LunaBeans leaving a grinder hopper, in MuJoCo · 2026-10-15 · 4 of 6 checksHeld, then flows, behaviour wrong, critical; its note named the faultAlso failed: Beans start inside a hollow hopper (built wrong)All 50 beans are through by 1 s, before the gate opens, and every one leaves the model. Its eighth and last round crashed before drawing, so round 7 is scored.

Beans leaving a grinder hopper, in MuJoCo, by GPT-6 Luna — GPT-6 LunaBeans leaving a grinder hopper, in MuJoCo · 2026-10-15 · 1 of 6 checksHeld, then flows, behaviour wrong, critical; its note said nothing was wrongAlso failed: model.xml written (incomplete); Fifty two-part beans (built wrong); Gravity down (false); Beans start inside a hollow hopper (built wrong)No model.xml beside the script, so the checker has no model to read. The hopper is about a metre tall, and 44 beans are out by 1 s.

The other kinds: malformed · false · incomplete · unreadable · built wrong

All of them, with counts and the codebook · How a check is decided