DeepSeek 4.1 Flash
69 runs across 23 tasks, costing $5.96, and 13 posts. Each table shows the runs under the newest prompt this model was given.
How it failed: false, unreadable, built wrong, incomplete, broken.
| check · how it is decided | run 109-22 | run 209-22 | run 110-03 |
|---|---|---|---|
| All seven partsby eye · Pass if all seven are drawn as separate shapes and each has a label naming it. A part that is only named in a label or a comment, with nothing drawn, fails it. | ✓ pass | ✓ pass | ✗ fail |
| Stack ordermeasured · Knob, lid, spiral plate, mesh, cross-plate and carafe, top to bottom along the axis, read from each part's y coordinate in the SVG. Its comment is ignored. The rod is left out, because it runs through the lid. | · n/d | ✓ pass | ✗ fail |
| Filter fits the glassmeasured · Every filter piece drawn is 80% to 105% of the width of the carafe's glass wall, measured in the SVG. | ✓ pass | ✓ pass | ✓ pass |
| One assembly axisby eye · Pass if one vertical line runs through the middle of every part, and every leader line leaves from the same side. One part pushed off to the side fails it. | ✓ pass | ✓ pass | ✓ pass |
| Labels matchby eye · Pass if following each leader line from its label ends on the part that label names. One label pointing at the wrong part, or at nothing, fails it. | ✓ pass | ✓ pass | ✓ pass |
Prompt v2, checks c1, harness h1. Ground truth: The standard three-piece plunger, from the top down: spiral (spring) plate, mesh screen, cross-plate. The cross-plate is the bottom piece, because its threaded stem is what the rod screws into. The spiral plate above the mesh is what presses the mesh edge out against the glass. Checked against the WebstaurantStore anatomy diagram, which is drawn exploded and labelled, and against each part's y coordinate in the SVG rather than its comment. Two-piece travel presses exist and stack differently, which is why v2 of the prompt names the design. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| All five partsby eye · Pass if every listed part is drawn as a separate shape and each has a label naming it. A part only named in a label or a comment, with nothing drawn, fails it. The label must say what the part is, in any words: a name that only says where it sits ("top plate") or what it is made of, or a mention in another part's caption, does not count. | ✗ fail | ✓ pass | ✓ pass |
| Stack ordermeasured · Basket, spring, body, spout, top to bottom along the vertical axis, read from each part's coordinates in the SVG. The handle is off the stack. | ✓ pass | ✓ pass | ✓ pass |
| Parts fitby eye · Pass if each listed pair of mating parts looks as if it would go together as stated: the right angle, nested inside, or sitting on top. One pair that plainly could not fails it. | · n/d | ✗ fail | ✓ pass |
| Two assembly axesby eye · Pass if one line runs through the middle of every part on each listed axis, and every leader line leaves from the same side. One part pushed off its axis fails it. | ✓ pass | ✓ pass | ✓ pass |
| Labels matchby eye · Pass if following each leader line from its label ends on the part that label names. One label pointing at the wrong part, or at nothing, fails it. | ✓ pass | ✓ pass | ✓ pass |
Prompt v1, checks c1, harness h1. Ground truth: On a vertical axis, top to bottom: filter basket, retaining spring, portafilter body, double spout. The handle is on a horizontal axis off the body, so this is the one member with two axes. Checked against parts sellers, since no maker's exploded diagram could be fetched: Espresso Parts and Zirkus Coffee on the spring that clips into a groove inside the body and grips the basket rim, Espresso Parts on the spout that screws onto the bottom of the body, and Nepo Coffee on handles that screw onto a stud on its side. Every model on this task.
| check · how it is decided | run 1clock stop |
|---|---|
| On real streetsmeasured · Share of track points within 25 m of a mapped street centreline. | ✓ pass |
| One continuous routemeasured · A single track segment, with no jump between consecutive points bigger than a city block. | ✓ pass |
| Inside the gridmeasured · The whole trace sits within the Hoddle Grid, and does not cross the Yarra. | ✓ pass |
| Metadata trueby eye · Pass if every street the file's description names is one the trace runs along for at least a block, read off the map under the trace. One named street the route never uses fails it. | ✓ pass |
| Reads as COFFEEby eye · Pass if the six letters can be read off the plotted trace in order, by someone not told the word first. In practice that someone is me, and I knew the word, which is the weakness of this check. | ✓ pass |
Prompt v2, checks c1, harness h1. A run marked as stopped was cut off before it finished, and is scored on its last rendered frame. Ground truth: An OpenStreetMap extract of the Hoddle Grid (streets and water, ODbL, taken 2026-09-16), the same one the harness ships for rendering. Every track point is measured to the nearest street centreline. Every model on this task.
| check · how it is decided | run 109-20 | run 309-21clock stop | run 109-22 | run 209-22 | run 309-22 |
|---|---|---|---|---|---|
| C8H10N4O2measured · Atom records counted by element: 8 C, 10 H, 4 N, 2 O. 24 atoms in total. | ✓ pass | ✓ pass | ✓ pass | ✓ pass | ✓ pass |
| Fused purine coremeasured · A 6-ring and a 5-ring found in the bond graph, sharing exactly two atoms. | ✓ pass | ✓ pass | ✓ pass | ✓ pass | ✓ pass |
| Methyls on N1, N3, N7measured · Each CH3 carbon traced to the nitrogen it is bonded to. | ✓ pass | ✓ pass | ✓ pass | ✓ pass | ✓ pass |
| Carbonyls at C2 and C6measured · Both terminal oxygens traced to their ring carbon. | ✓ pass | ✓ pass | ✓ pass | ✓ pass | ✓ pass |
| Flat rings, real bond lengthsmeasured · Out-of-plane RMS of each ring, and every heavy-atom bond length. | ✓ pass | ✓ pass | ✓ pass | ✓ pass | ✓ pass |
Prompt v1, checks c1, harness h1. A run marked as stopped was cut off before it finished, and is scored on its last rendered frame. Ground truth: Caffeine, C8H10N4O2: a purine core, a six-membered ring fused to a five-membered ring across two shared carbons, methyls on N1, N3 and N7, carbonyl oxygens at C2 and C6, and the whole ring system flat. Every model on this task.
| check · how it is decided | run 1 | run 3 |
|---|---|---|
| Complete analoguemeasured · The node the temperature is written from has a capacitor on it, a resistor to a node a DC source holds fixed, and a third element feeding it. The resistor times the capacitor has to be under a day, so a leak nobody meant does not count as the loss. | ✓ pass | ✓ pass |
| Real hysteresismeasured · Re-run, once in band the heater switches on low and off high, at least 1 °C apart and at least 2 s per cycle, and the netlist has a switch that remembers: a SW model with a hysteresis voltage, or a source that reads its own output. A single-threshold switch sits at the set point with a swing near zero. | ✓ pass | ✓ pass |
| Starts coldmeasured · Initial condition at ambient, and the transient run with uic so it is honoured. | ✓ pass | ✓ pass |
| Holds 92–96 °Cmeasured · The written temperature re-read: once in band it never leaves it. | ✓ pass | ✓ pass |
| Writes the .txtmeasured · A wrdata line in .control, and the file actually appears. | ✓ pass | ✓ pass |
Prompt v2, checks c2, harness h1. Ground truth: The netlist itself: re-run in ngspice offline, and the temperature it writes out read back. A thermostat with a dead band is a stateful thing, so a bare comparator on one threshold is wrong however plausible the plot looks. Every model on this task.
| check · how it is decided | run 1clock stop | run 2clock stop | run 3 |
|---|---|---|---|
| All five objectsby eye · Pass if all five can be pointed at in the render as separate objects. One missing, or hidden behind another, fails it. | ✓ pass | ✓ pass | ✓ pass |
| Primitives onlymeasured · Every mesh comes from a primitive_*_add call, as the prompt requires. | ✓ pass | ✓ pass | ✓ pass |
| Spoon reads as a spoonby eye · Pass if the spoon is one connected piece with a bowl at one end, wider than the handle, and the handle running from it in a line. Loose parts, or no handle, fail it. | ✗ fail | ✗ fail | ✗ fail |
| Croissant reads as a croissantby eye · Pass if it is a crescent that is thickest in the middle and tapers to both tips, with segments or scoring across it. A smooth curved tube fails it, and so does a segmented one with no taper. | ✗ fail | ✓ pass | ✓ pass |
| Consistent scalemeasured · Object dimensions in the script compared against each other and against life. | ✓ pass | ✓ pass | ✓ pass |
Prompt v2, checks c1, harness h1. A run marked as stopped was cut off before it finished, and is scored on its last rendered frame. Ground truth: The render, and the script that made it: the objects a person would name unprompted, at sizes a person would accept. A 5 cm spoon and an 11 cm croissant are checkable in the source. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| Parsesmeasured · RDKit reads the SMILES as a molecule. | ✓ pass | ✓ pass | ✓ pass |
| C8HF9N4O2measured · Molecular formula computed from the parsed molecule. | ✓ pass | ✓ pass | ✓ pass |
| Three N-CF3, no N-methylmeasured · Each CF3 carbon traced to a ring nitrogen, and no methyl left on any nitrogen. | ✓ pass | ✓ pass | ✓ pass |
| The right moleculemeasured · Canonical SMILES identical to the target, so a CF3 on the wrong nitrogen fails. | ✓ pass | ✓ pass | ✓ pass |
Prompt v1, checks c1, harness h1. Ground truth: Caffeine's skeleton with a CF3 on N1, N3 and N7, which is C8HF9N4O2. The prompt pack this came from gives C8H3F9N4O2, which is wrong: swapping the methyls takes all nine hydrogens with them, and only the C8 hydrogen stays. Decided with RDKit by an exact canonical match, because fluorinated isocaffeine has the same formula with a CF3 on the wrong nitrogen. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| Compilesmeasured · pdflatex builds the file offline. | ✓ pass | ✓ pass | ✓ pass |
| Seven regions, three namesmeasured · Three circles found in the PDF with every overlap present, and each set name placed by its circle. | ✓ pass | ✓ pass | ✓ pass |
| Every drink in its regionmeasured · Each of the eight words found in the PDF, and the centre of each one inside the region it belongs to. | ✓ pass | ✓ pass | ✓ pass |
| No label on an edgemeasured · No letter touches a circle's outline, with the line's width counted. | ✗ fail | ✗ fail | ✓ pass |
Prompt v1, checks c2, harness h1. Ground truth: Short black is espresso only, flat white espresso and milk, iced long black espresso and cold, iced latte all three, babyccino milk only, milkshake milk and cold, cold brew cold only, and pour-over outside every circle. The circles and words are read back out of the compiled PDF, each word has to be centred in its own region, and no letter may touch a circle's outline, with the line's width counted. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| A real chartmeasured · A grid of at least 10 stitches by 10 rows, with the beans made of whole cells of that grid. | ✓ pass | ✓ pass | ✓ pass |
| Stitches wider than tallmeasured · The cells are drawn at least as wide as they are tall, so the chart shows the fabric the right way round. | ✓ pass | ✓ pass | ✓ pass |
| Bean knits truemeasured · The beans as knitted are 1.3 to 1.8 times as long as they are wide. | ✓ pass | ✗ fail | ✗ fail |
| Has a keymeasured · Text on the page says what each colour or mark means. | ✓ pass | ✓ pass | ✓ pass |
Prompt v1, checks c1, harness h1. Ground truth: A knit stitch is wider than it is tall, about 5 stitches to 7 rows over the same distance, so a bean drawn on square cells knits up squashed. The chart is read from the SVG: its grid, which cells are coloured, and the gauge its cells are drawn at. Each bean is then measured as it would come out in the fabric, from its stitches and rows at that gauge, and a coffee bean is 1.3 to 1.8 times as long as it is wide. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| Says circles cannotmeasured · A sentence it wrote names circles and says they cannot give five sets or 31 regions, or names the 22 region limit. | ✓ pass | ✓ pass | ✓ pass |
| Five sets keptmeasured · If it drew: all five names in the PDF or its source, and at least five closed curves, so a three-set drawing cannot pass. | ✓ pass | ✓ pass | ✓ pass |
| Names a fix that worksby eye · Ellipses, Edwards' curves, other non-circle curves or a table, offered in its reply, its file or its drawing. | ✓ pass | ✓ pass | ✓ pass |
Prompt v1, checks c1, harness h1. Ground truth: Five circles make at most 22 regions counting the outside, since n circles make at most n squared minus n plus 2, and five sets need 31 plus the outside. So the right answer says circles cannot do it and offers something that can: ellipses (Grünbaum, 1975), Edwards' curves, or a table of the 31 combinations. What the model said out loud and in the file's comments is read, its thinking is not. A drawing is only held to 31 regions when the model never said circles cannot do it. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| DC supplies onlymeasured · No PULSE, SIN or PWL source, and no behavioural source that reads time. | ✓ pass | ✓ pass | ✓ pass |
| 1 kHzmeasured · It runs, oscillates, and settles within 10% of 1 kHz. | ✓ pass | ✓ pass | ✓ pass |
| 2 V peak to peakmeasured · Within 10% of 2 V once settled. | ✓ pass | ✓ pass | ✓ pass |
| Straight, even rampsmeasured · Each ramp fitted to a line with little bend, and rise and fall about equal. | ✓ pass | ✓ pass | ✓ pass |
Prompt v1, checks c1, harness h1. Ground truth: The netlist, re-run in ngspice offline and the written carrier read back. A triangle comes from integrating a square wave, so a PULSE into an RC, whose ramps curve, fails, and so does any source that is told the time. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| Renders, labels validmeasured · Mermaid draws it, every transition has exactly one of the seven events, and no event leads two ways. | ✓ pass | ✓ pass | ✓ pass |
| Happy pathmeasured · pay, queue, start, finish, collect runs from the start to an end. | ✓ pass | ✓ pass | ✓ pass |
| No refund if unpaidmeasured · Cancelling straight after placing can never reach refund. | ✓ pass | ✓ pass | ✓ pass |
| Refund if paidmeasured · Cancelling once paid, queued, brewing or ready cannot end without refund. | ✓ pass | ✓ pass | ✓ pass |
| No cancel after collectmeasured · cancel is not accepted once the order is collected. | ✓ pass | ✓ pass | ✓ pass |
Prompt v1, checks c1, harness h1. Ground truth: The diagram parsed into a transition table and run against fixed event sequences: the happy path, and a cancel after placing, paying, queueing, brewing and collecting. The start is the state [*] points to or, since the prompt never asks for [*], the one state nothing points into. Every model on this task.
| check · how it is decided | run 1 | run 2clock stop | run 3spend cap stop |
|---|---|---|---|
| Reads as a shrubby eye · Pass if it reads as one plant with branches coming off stems, without being told what it is. A tangle, a lattice or loops fail it. | ✓ pass | ✓ pass | ✗ fail |
| Flat side from the rulesby eye · Pass if one side ends on a straight line that the rules themselves put there. A box drawn around the plant to cut it fails it. | ✓ pass | ✗ fail | ✗ fail |
| Sensible sizeby eye · Pass if the branches can still be followed by eye. So many segments that they merge into a mass fails it. | ✓ pass | ✓ pass | ✗ fail |
Prompt v1, checks c2, harness h1. A run marked as stopped was cut off before it finished, and is scored on its last rendered frame. Ground truth: The JSON expanded and drawn with turtle graphics offline, and the drawing judged by eye. There is no single right L-system, so nothing here is measured. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| Valid Newickmeasured · It reads as Newick and ends in a semicolon, no unquoted name has a blank in it, every branch has a length and none is negative, and no node has only one child. | ✓ pass | ✓ pass | ✓ pass |
| Twelve cultivarsmeasured · Twelve tips, each with a name, no two the same, and not numbered stand-ins like T1 to T12. | ✓ pass | ✓ pass | ✓ pass |
| Every tip levelmeasured · Every path from the root to a tip adds up to the same length. | ✓ pass | ✓ pass | ✓ pass |
| One split of fourmeasured · Exactly one node has four children, and every other node has two. | ✓ pass | ✓ pass | ✓ pass |
Prompt v1, checks c1, harness h1. Ground truth: The tree read with Biopython, every path from the root to a tip added up from its branch lengths, and the children of every node counted. The checks only read the brackets. Whether a cultivar sits beside its relatives is not checked here, and the post of 14 October looks at that by eye. Every model on this task.
| check · how it is decided | run 2clock stop |
|---|---|
| Runs in MuJoComeasured · The script runs, steps a MuJoCo model, prints its report and saves the figure. | ✓ pass |
| model.xml writtenmeasured · A model.xml beside the script that MuJoCo loads. | ✓ pass |
| Fifty two-part beansmeasured · Exactly 50 free bodies, each made of two ellipsoids. | ✓ pass |
| Gravity downmeasured · Gravity in the model points down, at more than 1 m/s². | ✓ pass |
| Beans start inside a hollow hoppermeasured · A ray dropped down the middle of the hopper gets past where the beans are, and rays cast sideways from every bean's starting place hit the hopper wall. A mesh counts as its convex hull, the way MuJoCo collides it. | ✗ fail |
| Held, then flowsby Claude Opus 5.5, GPT-6.1 Sol and Gemini 4 Argon · Pass if the frames and the printed count show no bean through the throat at 1 s, beans coming through by 2 s, all 50 through by 5 s, and none falling out of the model. A count the frames contradict fails it. | ✓ pass |
Prompt v1, checks c1, harness h1. A run marked as stopped was cut off before it finished, and is scored on its last rendered frame. Ground truth: The model.xml the script writes, loaded in MuJoCo and probed with rays against what a bean can actually hit. MuJoCo collides a mesh as its convex hull, so a cone made of one mesh is solid, and the probe counts it that way. The checker never runs the simulation, so whether the gate holds the beans and then lets them all out is judged by Claude Opus 5.5, GPT-6.1 Sol and Gemini 4 Argon, each shown the brief, the figure, the printed count and the rule, and never which model made it. On 2026-10-16 all three agreed on all ten results, and with the verdicts first published, for $0.52. The rule was written after Claude Opus 5.5 had looked at the runs. Every model on this task.
| check · how it is decided | run 1clock stop | run 2clock stop | run 3 |
|---|---|---|---|
| HTML and CSS onlymeasured · No <script>, <svg>, <img>, <canvas>, <iframe>, external url(...) or javascript: URL in the file. | ✓ pass | ✓ pass | ✓ pass |
| A CSS loop that moves somethingmeasured · An @keyframes rule that changes a transform, a translation, a rotation or an offset between stops, named in an animation on an element, with the iteration count infinite. It does not know which element moves: on 2026-10-19 a GLM run passed it by turning the coffee surface while the spoon stood still, and the eye checks are what caught that. | ✓ pass | ✓ pass | ✓ pass |
| Reads as a cup seen intoby eye · The coffee surface is visible inside a rim that has some thickness, whether that is an ellipse seen from slightly above or a ring seen from straight down. A flat disc with no rim, or a cup whose inside cannot be seen, fails it. | ✓ pass | ✓ pass | ✓ pass |
| The spoon does the stirringby eye · The spoon itself, bowl and handle together, travels round inside the opening, so its place changes across the loop. Judge it from the live page or the video, since the four still moments can show almost the same instant. A file whose only motion is in the coffee surface, with the spoon stood still at the rim, fails it. | ✗ fail | ✗ fail | ✗ fail |
| Reads as inside the cupby eye · The bowl stays within the rim of the opening and the handle leaves over the near edge, so the spoon reads as in the drink rather than laid on top of the picture. A spoon drawn across the cup mouth in front of the rim for the whole turn fails it, which is the trap. | ✗ fail | ✓ pass | ✓ pass |
| Handle stays on the bowlby eye · The handle meets the bowl of the spoon in every frame. A handle that separates, points the wrong way or is missing fails it. | ✗ fail | · n/d | ✓ pass |
Prompt v1, checks c1, harness h1. A run marked as stopped was cut off before it finished, and is scored on its last rendered frame. Ground truth: HTML and CSS alone, a continuous loop, and a cup seen from slightly above. The file is read for the brief and for the loop: an @keyframes rule that changes a position or a transform, named by an element, with the iteration count set to infinite. The picture is four moments of the animation, left to right, taken at 0.4 s, 1.2 s, 2.0 s and 2.8 s. Whether the rim reads as an ellipse seen from above, whether the spoon passes behind the far rim on the back half of its circuit, and whether the handle stays on the bowl are judged by eye from the page the run rendered, with the four frames as a first look. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| Repairedmeasured · The `order` check of the exploded V60 grader, run on the file the model was handed and on the file it gave back, both read from the shapes' coordinates. Fixed means false became true. | ✓ pass | ✓ pass | ✓ pass |
| Broke nothing elsemeasured · Every check that passed on the file it was handed still passes on the file it gave back: `parts` and `axis`. | ✓ pass | ✓ pass | ✓ pass |
Prompt v1, checks c1, harness h1. Ground truth: Claude Haiku 5.5 drew this diagram on 2026-10-19 and failed `order`: the paper filter came below the cone. Read from the shapes' coordinates, the stack from the top down is paper filter, dripper, glass server, and the numbers printed on the labels are the file's own numbering, which the prompt lets stay wrong. The folded diagram also has a base narrower than the server's mouth and a cone of about 83 degrees, which the prompt tells the model to leave alone because the task is the order and nothing else. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| In the drinkby eye · A spoon drawn across the cup mouth in front of the rim for the whole turn fails, which is the file the run was handed. | ✓ pass | ✓ pass | ✓ pass |
| Kept the brief and the loopmeasured · The HTML and CSS survived, with no script, image or canvas added, and the loop still moves something. | ✓ pass | ✓ pass | ✓ pass |
Prompt v1, checks c1, harness h1. Ground truth: GLM 5.3 FlashX drew this cup on 2026-10-19. The spoon's bowl sits at the back of the opening with the handle laid across it toward the near rim, so the spoon reads as resting on the cup while the coffee surface turns under it. The fix is a change of z-order: the handle has to pass behind the near rim and the bowl has to sit in the drink. Nothing in the file can be measured for it, because both arrangements use the same shapes in the same places, so the eye decides it and `unchanged` guards the rest. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| Repairedmeasured · The `labels_clear_of_edges` check of the Venn grader, run on the file the model was handed and on the file it gave back, each letter tested against the drawn outline. Fixed means false became true. | ✓ pass | ✓ pass | ✓ pass |
| Broke nothing elsemeasured · Everything that passed on the input still passes: the file compiles with pdflatex, the seven regions are there, the set names are right and the drinks are found in their regions. | ✓ pass | ✓ pass | ✓ pass |
Prompt v1, checks c1, harness h1. Ground truth: Kimi K3 drew this Venn on 2026-10-19 and put `milkshake` on the outline of a circle. The word belongs in the region that names milk, and the fix is to move that one label until every letter sits clear of every drawn line without leaving its region. The other seven drinks, the seven regions and the set names were right on the file it was handed. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| Repairedmeasured · The `bean_knits_true` check of the knitting grader, run on the file the model was handed and on the file it gave back. Fixed means the knitted ratio was outside the band and is now inside it. | ✓ pass | ✓ pass | ✓ pass |
| Broke nothing elsemeasured · Everything that passed on the input still passes: the chart is a chart, the motif is in whole cells, the cells are not taller than they are wide, and the key is still there. | ✓ pass | ✓ pass | ✓ pass |
Prompt v1, checks c1, harness h1. Ground truth: DeepSeek 4.1 Flash drew this chart on 2026-10-19. Its bean is 13 stitches by 8 rows in cells drawn 20 by 14.3, with a gauge of 1.399 stitches to the row, so the knitted bean comes out 2.43 times as long as it is wide against a band of 1.3 to 1.8. The motif has to change shape in whole cells while the grid, the cell size and the key stay put, which is why the trap is a rescale of the whole chart. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| Serves all sixmeasured · The tray hands each of the six tables a drink it will take, every drink used exactly once and none twice. A refusal leaves it undecided, and a tray that reuses a drink fails it. | ✓ pass | ✓ pass | ✓ pass |
Prompt v2, checks c1, harness h1. Ground truth: Six drinks with fixed attributes, six tables with fixed notes, one tray. The checker derives who can take what, reads the tray's labels, and confirms with a perfect-matching solver that a serving exists: A to F in order. A refusal on a solvable brief is wrong, and so is a tray that reuses a drink or hands a table a drink it will not take. Whether a drawn cup really holds what its label says is the eye's. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| Says it cannot be donemeasured · The reply, or a comment in the file, says no serving of all six exists. A run that draws a tray instead and says nothing fails it. | ✓ pass | ✓ pass | ✓ pass |
| Reuses nothingmeasured · If it drew: no drink given to two tables and no table given two drinks. A run that drew nothing is undecided. | ✓ pass | · n/d | · n/d |
Prompt v2, checks c1, harness h1. Ground truth: Tables 1, 2, 3 and 4 together will only take the three drinks A, B and C, so no serving of all six exists, and the checker confirms that with its own perfect-matching solver. The right answer says so and draws nothing. Every table still has a drink it will take on its own, so the shortage is only visible across the tables together. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| Serves all sixmeasured · The tray hands each of the six tables a drink it will take, every drink used exactly once and none twice. A refusal leaves it undecided, and a tray that reuses a drink fails it. | ✓ pass | ✓ pass | ✓ pass |
Prompt v2, checks c1, harness h1. Ground truth: Two tables want black coffee and table 4 has to take the iced oat latte or tables 1, 2 and 3 run out of A, B and C. A serving exists, A to F in order, and the checker confirms it with its own matching solver. Refusing because it looks crowded is wrong. Every model on this task.