Three shots
Every model gets three shots at the same frozen prompt.
There are 16 tasks on the board, each with a right answer I can check. Three runs is too few for a pass rate, so nothing here claims one. The prompts, the checkers and every scored file are in the public repo.
The board
One row per run, one square per check: passed failed undecided judged by eye judged by a model⏱ stopped by the clock
- French press
- blank
- Pruned shrub
- ⏱
- Stirring spoon
- ⏱⏱
- COFFEE route
- blankblankblank
- Café table
- ⏱⏱
- Grinder
- ⏱
- Pruned shrub
- ⏱⏱
- Pruned shrub
- ⏱
- French press
- ⏱
- Café table
- ⏱⏱blank
A run holds up to 8 rounds or twelve minutes. How a run becomes a row.
9 runs were given 1 round: Verdict, flipped, Verdict, good press. Those tasks ask for an answer in words, so a second round would only be a second attempt at the same sentence.
9 runs were given 8 rounds and 30 minutes: Pump noise. Those runs had 30 minutes, because a careful run takes many short steps and the clock was measuring the provider more than the model.
Of the 66 checks on the board, 34 are measured by a script, 32 I judge by eye.
Every run and check as data: CSV or JSON.
Retired once two models passed every check: Caffeine (9 runs), Boiler (7 runs), Fluorocaffeine (6 runs), Venn (61 runs), Five circles (16 runs), Triangle wave (6 runs), Café order (6 runs), Penrose floor (6 runs), Coffee tree (15 runs), Hopper (13 runs), V60 repair (9 runs), Knitting repair (9 runs), Serve the tray (21 runs), Four tables, three drinks (24 runs), Two tables, black coffee (24 runs).
Saturated tasks are not fully retired but are kept to check for regression and also to allow comparison over time and provide some stability of the board.
Passed by the fewest models
1 of the 117 checks have never passed. These are the ones at most 60% of the models tried have passed, counting runs on the task's current prompt.
| task | check | models that passed |
|---|---|---|
| Pump noise | Sounds right | 0 of 5 |
| Grinder | All eight parts | 2 of 4 |
| Espresso Rag | Crossing marked | 2 of 4 |
| Café table | Spoon reads as a spoon | 1 of 2 |
| French press | Stack order | 13 of 24 |
| Venn | No label on an edge | 10 of 17 |
| AeroPress | Parts fit | 3 of 5 |
| Pump noise | Hum down | 3 of 5 |
| Pump noise | Rush down | 3 of 5 |
| Pump noise | Music kept | 3 of 5 |
By kind of quality
| kind | fails when | failed |
|---|---|---|
| malformed | the model's file won't open: it doesn't compile, parse or render | 1 of 135 |
| false | the model gets a fact wrong, such as a part in the wrong place or at the wrong size | 205 of 855 |
| incomplete | the model leaves out something the brief asked for | 100 of 452 |
| unreadable | the content may be right, but a reader can't make it out | 90 of 285 |
| broken | the model's file runs, but doesn't do what it should | 7 of 94 |
| built wrong | the result may look right, but the model didn't make it the way the brief said | 27 of 101 |
By model
Failed checks of the ones each model had, in red when every one failed. A dot means it had none of that kind.
| model | malformed | false | incomplete | unreadable | broken | built wrong |
|---|---|---|---|---|---|---|
| Opus 5.5 | 0 of 11 | 7 of 55 | 8 of 39 | 4 of 14 | · | 1 of 3 |
| GPT-6.1 Sol | 0 of 5 | 6 of 77 | 5 of 43 | 0 of 20 | 0 of 9 | 0 of 5 |
| Gemini 4 Argon | 0 of 6 | 4 of 15 | 4 of 9 | 0 of 3 | · | 0 of 3 |
| Kimi K3.1 Pro | 0 of 9 | 18 of 42 | 6 of 21 | 2 of 9 | 0 of 3 | 0 of 9 |
| GPT-6.1 Terra | 0 of 3 | 3 of 6 | 4 of 6 | · | · | · |
| GLM 5.3 FlashX | 1 of 6 | 21 of 72 | 7 of 29 | 10 of 20 | 0 of 6 | 2 of 6 |
| DeepSeek 4.1 Flash | 0 of 16 | 8 of 106 | 2 of 49 | 8 of 25 | 3 of 37 | 3 of 15 |
| GPT-6 Luna | 0 of 12 | 16 of 75 | 15 of 48 | 5 of 24 | 3 of 3 | 6 of 9 |
| Claude Sonnet 5.5 | 0 of 6 | 4 of 45 | 2 of 18 | 2 of 12 | 0 of 6 | 2 of 6 |
| Claude Haiku 5.5 | 0 of 12 | 27 of 84 | 18 of 45 | 4 of 24 | 1 of 3 | 0 of 9 |
| Kimi K3 | 0 of 3 | 1 of 8 | 2 of 7 | 3 of 4 | · | · |
| Pareto 26.9 | 0 of 3 | 0 of 3 | 0 of 11 | · | 0 of 3 | 0 of 6 |
| Nemotron 3.5 Super VL | · | 10 of 12 | 3 of 3 | 5 of 6 | · | 0 of 3 |
| Muse Spark Open | · | 6 of 9 | 0 of 2 | 2 of 5 | · | 3 of 3 |
| Muse Wonder | 0 of 3 | 3 of 12 | 0 of 6 | 0 of 6 | · | · |
| Qwen3.8 Max Prime | · | 4 of 5 | 0 of 3 | 4 of 5 | · | 2 of 5 |
| Pareto 26.10 Preview | 0 of 3 | 12 of 61 | 6 of 33 | 3 of 19 | 0 of 12 | 0 of 1 |
| Step 5 Preview | 0 of 3 | 4 of 12 | 3 of 6 | 3 of 6 | · | · |
| Mistral Large 4 | · | 2 of 9 | 2 of 3 | 0 of 3 | · | · |
| Ember 1 | 0 of 3 | 2 of 8 | 2 of 5 | 3 of 5 | · | · |
| GPT-6 Sol | 0 of 13 | 0 of 16 | 0 of 14 | 2 of 14 | · | · |
| Grok 4.7 | 0 of 3 | 4 of 6 | 0 of 4 | 3 of 4 | · | · |
| Gemini 3.8 Flash | 0 of 3 | 2 of 8 | 0 of 4 | 2 of 6 | · | 2 of 2 |
| Pareto | · | 4 of 19 | 1 of 6 | 0 of 7 | · | 0 of 1 |
| GPT-5.6 Sol | · | 5 of 14 | 0 of 4 | 2 of 6 | · | 1 of 2 |
| Qwen3.8 27B | · | 4 of 12 | 4 of 4 | 0 of 6 | · | 0 of 2 |
| Qwen3.8 Max | · | 0 of 21 | · | 1 of 1 | · | 0 of 1 |
| MiMo v2.6 Pro | 0 of 3 | 1 of 3 | 0 of 3 | 3 of 3 | · | · |
| Perceptron Mk1.5 | 0 of 3 | 15 of 15 | 3 of 6 | 9 of 9 | · | 3 of 3 |
| Command A+ | 0 of 3 | 12 of 15 | 3 of 6 | 9 of 9 | · | 1 of 3 |
| Qwen3.8 Flash | 0 of 3 | 0 of 7 | 0 of 6 | 1 of 9 | · | 1 of 4 |
| Astra | · | · | 0 of 8 | · | 0 of 12 | · |
| Union Alpha | · | 0 of 3 | 0 of 1 | 0 of 1 | · | · |
Why each one failed, whether the model saw it, and the codebook are on the failure page.
How a run works
A run is one model starting from nothing on one task. A round is one picture it asks for: it writes the file, runs render and looks at what came out. A model's three runs differ only by their seed.
The last round that finished rendering is the one scored. If the model edits the file after that, I keep the file and leave it unscored, because the brief tells every model that only render counts.
A run gets 8 rounds and twelve minutes. In the first week, 14 to 20 September, 15 single runs had 14 or 20 rounds or a 28 minute clock, which is why one row in the data shows 12 rounds. None of them is on the board.
A cell holds only runs given the current prompt, scored by the current checker and run through the current harness, which is the instructions around the prompt, the render tool and the sandbox. Older runs stay in the gallery and in the data, where the prompt_version, checks_version, harness_version and on_board columns say which is which.
Every model gets the same brief and the same tools. Some check their picture by eye and some write code to measure it. That choice is the model's, so it is part of what the board measures.
The clock ended 12 of the 232 runs on the board, marked ⏱, and my spend cap ended 4. The middle run took about 2 minutes. So a slow model sits closer to the clock, and the board partly measures speed. Every run's wall time is in the data.