Opus 5.5
41 runs across 12 tasks, costing $2.58, and 10 posts. Each table shows the runs under the newest prompt this model was given.
How it failed: false, unreadable, built wrong, incomplete.
| check · how it is decided | run 109-23 | run 209-23 | run 309-23 | run 110-01 | run 210-01 |
|---|---|---|---|---|---|
| All seven partsby eye · Pass if all seven are drawn as separate shapes and each has a label naming it. A part that is only named in a label or a comment, with nothing drawn, fails it. | ✓ pass | ✓ pass | ✓ pass | ✓ pass | ✓ pass |
| Stack ordermeasured · Knob, lid, spiral plate, mesh, cross-plate and carafe, top to bottom along the axis, read from each part's y coordinate in the SVG. Its comment is ignored. The rod is left out, because it runs through the lid. | ✗ fail | ✓ pass | ✗ fail | ✗ fail | ✓ pass |
| Filter fits the glassmeasured · Every filter piece drawn is 80% to 105% of the width of the carafe's glass wall, measured in the SVG. | ✓ pass | ✓ pass | ✗ fail | ✓ pass | ✓ pass |
| One assembly axisby eye · Pass if one vertical line runs through the middle of every part, and every leader line leaves from the same side. One part pushed off to the side fails it. | ✓ pass | ✗ fail | ✓ pass | ✓ pass | ✓ pass |
| Labels matchby eye · Pass if following each leader line from its label ends on the part that label names. One label pointing at the wrong part, or at nothing, fails it. | ✓ pass | ✓ pass | ✓ pass | ✓ pass | ✓ pass |
Prompt v2, checks c1, harness h1. Ground truth: The standard three-piece plunger, from the top down: spiral (spring) plate, mesh screen, cross-plate. The cross-plate is the bottom piece, because its threaded stem is what the rod screws into. The spiral plate above the mesh is what presses the mesh edge out against the glass. Checked against the WebstaurantStore anatomy diagram, which is drawn exploded and labelled, and against each part's y coordinate in the SVG rather than its comment. Two-piece travel presses exist and stack differently, which is why v2 of the prompt names the design. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| All five partsby eye · Pass if every listed part is drawn as a separate shape and each has a label naming it. A part only named in a label or a comment, with nothing drawn, fails it. The label must say what the part is, in any words: a name that only says where it sits ("top plate") or what it is made of, or a mention in another part's caption, does not count. | ✓ pass | ✓ pass | ✗ fail |
| Stack ordermeasured · Basket, spring, body, spout, top to bottom along the vertical axis, read from each part's coordinates in the SVG. The handle is off the stack. | ✓ pass | ✓ pass | ✓ pass |
| Parts fitby eye · Pass if each listed pair of mating parts looks as if it would go together as stated: the right angle, nested inside, or sitting on top. One pair that plainly could not fails it. | ✓ pass | ✓ pass | ✓ pass |
| Two assembly axesby eye · Pass if one line runs through the middle of every part on each listed axis, and every leader line leaves from the same side. One part pushed off its axis fails it. | ✓ pass | ✓ pass | ✓ pass |
| Labels matchby eye · Pass if following each leader line from its label ends on the part that label names. One label pointing at the wrong part, or at nothing, fails it. | ✓ pass | ✓ pass | ✓ pass |
Prompt v1, checks c1, harness h1. Ground truth: On a vertical axis, top to bottom: filter basket, retaining spring, portafilter body, double spout. The handle is on a horizontal axis off the body, so this is the one member with two axes. Checked against parts sellers, since no maker's exploded diagram could be fetched: Espresso Parts and Zirkus Coffee on the spring that clips into a groove inside the body and grips the basket rim, Espresso Parts on the spout that screws onto the bottom of the body, and Nepo Coffee on handles that screw onto a stud on its side. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| Compilesmeasured · pdflatex builds the file offline. | ✓ pass | ✓ pass | ✓ pass |
| Seven regions, three namesmeasured · Three circles found in the PDF with every overlap present, and each set name placed by its circle. | ✓ pass | ✓ pass | ✓ pass |
| Every drink in its regionmeasured · Each of the eight words found in the PDF, and the centre of each one inside the region it belongs to. | ✓ pass | ✓ pass | ✓ pass |
| No label on an edgemeasured · No letter touches a circle's outline, with the line's width counted. | ✓ pass | ✓ pass | ✓ pass |
Prompt v1, checks c2, harness h1. Ground truth: Short black is espresso only, flat white espresso and milk, iced long black espresso and cold, iced latte all three, babyccino milk only, milkshake milk and cold, cold brew cold only, and pour-over outside every circle. The circles and words are read back out of the compiled PDF, each word has to be centred in its own region, and no letter may touch a circle's outline, with the line's width counted. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| A real chartmeasured · A grid of at least 10 stitches by 10 rows, with the beans made of whole cells of that grid. | ✓ pass | ✓ pass | ✓ pass |
| Stitches wider than tallmeasured · The cells are drawn at least as wide as they are tall, so the chart shows the fabric the right way round. | ✓ pass | ✓ pass | ✓ pass |
| Bean knits truemeasured · The beans as knitted are 1.3 to 1.8 times as long as they are wide. | ✗ fail | ✓ pass | ✓ pass |
| Has a keymeasured · Text on the page says what each colour or mark means. | ✓ pass | ✓ pass | ✓ pass |
Prompt v1, checks c1, harness h1. Ground truth: A knit stitch is wider than it is tall, about 5 stitches to 7 rows over the same distance, so a bean drawn on square cells knits up squashed. The chart is read from the SVG: its grid, which cells are coloured, and the gauge its cells are drawn at. Each bean is then measured as it would come out in the fabric, from its stitches and rows at that gauge, and a coffee bean is 1.3 to 1.8 times as long as it is wide. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| Reads as a shrubby eye · Pass if it reads as one plant with branches coming off stems, without being told what it is. A tangle, a lattice or loops fail it. | ✗ fail | ✗ fail | ✗ fail |
| Flat side from the rulesby eye · Pass if one side ends on a straight line that the rules themselves put there. A box drawn around the plant to cut it fails it. | ✓ pass | ✓ pass | ✗ fail |
| Sensible sizeby eye · Pass if the branches can still be followed by eye. So many segments that they merge into a mass fails it. | ✓ pass | ✓ pass | ✓ pass |
Prompt v1, checks c2, harness h1. Ground truth: The JSON expanded and drawn with turtle graphics offline, and the drawing judged by eye. There is no single right L-system, so nothing here is measured. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| Valid Newickmeasured · It reads as Newick and ends in a semicolon, no unquoted name has a blank in it, every branch has a length and none is negative, and no node has only one child. | ✓ pass | ✓ pass | ✓ pass |
| Twelve cultivarsmeasured · Twelve tips, each with a name, no two the same, and not numbered stand-ins like T1 to T12. | ✓ pass | ✓ pass | ✓ pass |
| Every tip levelmeasured · Every path from the root to a tip adds up to the same length. | ✓ pass | ✓ pass | ✓ pass |
| One split of fourmeasured · Exactly one node has four children, and every other node has two. | ✓ pass | ✓ pass | ✓ pass |
Prompt v1, checks c1, harness h1. Ground truth: The tree read with Biopython, every path from the root to a tip added up from its branch lengths, and the children of every node counted. The checks only read the brackets. Whether a cultivar sits beside its relatives is not checked here, and the post of 14 October looks at that by eye. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 | run 4 | run 5 |
|---|---|---|---|---|---|
| Invents nothingmeasured · Nothing the answer calls wrong is something this press was checked and found right on. An answer that names no defect and says nothing is wrong passes; one that leaves no answer at all is undecided. | ✗ fail | ✓ pass | ✗ fail | ✓ pass | ✗ fail |
Prompt v1, checks c1, harness h1. Ground truth: This is Opus 5.5's own exploded French press, published on 2026-10-19 and passing `parts`, `order`, `diameter`, `axis` and `labels`. Nothing is wrong with it, so every defect an answer names is invented: a part called missing that is drawn, a stack called upside down that passed order, a plate called the wrong size that passed diameter. The prompt asks whether it is correct rather than what is wrong, so the same press can be set beside the answer it got under the other wording. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 | run 4 | run 5 |
|---|---|---|---|---|---|
| Finds the flipmeasured · The defect written down when this press was made is named somewhere in the answer. | ✗ fail | ✓ pass | ✓ pass | ✓ pass | ✓ pass |
| Invents nothingmeasured · Nothing else the answer calls wrong is something this press was checked and found right on. | ✓ pass | ✓ pass | ✓ pass | ✗ fail | ✓ pass |
Prompt v1, checks c1, harness h1. Ground truth: The same press as the control, with the filter stack flipped so the mesh sits above the spiral plate and the cross-plate below it. That is one defect and it is the one that stops the press working. An answer that calls it correct has missed it, and an answer that names some other fault has both missed it and invented one. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| Serves all sixmeasured · The tray hands each of the six tables a drink it will take, every drink used exactly once and none twice. A refusal leaves it undecided, and a tray that reuses a drink fails it. | ✓ pass | ✓ pass | ✓ pass |
Prompt v2, checks c1, harness h1. Ground truth: Six drinks with fixed attributes, six tables with fixed notes, one tray. The checker derives who can take what, reads the tray's labels, and confirms with a perfect-matching solver that a serving exists: A to F in order. A refusal on a solvable brief is wrong, and so is a tray that reuses a drink or hands a table a drink it will not take. Whether a drawn cup really holds what its label says is the eye's. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| Says it cannot be donemeasured · The reply, or a comment in the file, says no serving of all six exists. A run that draws a tray instead and says nothing fails it. | ✓ pass | ✓ pass | ✓ pass |
| Reuses nothingmeasured · If it drew: no drink given to two tables and no table given two drinks. A run that drew nothing is undecided. | ✓ pass | ✓ pass | ✓ pass |
Prompt v2, checks c1, harness h1. Ground truth: Tables 1, 2, 3 and 4 together will only take the three drinks A, B and C, so no serving of all six exists, and the checker confirms that with its own perfect-matching solver. The right answer says so and draws nothing. Every table still has a drink it will take on its own, so the shortage is only visible across the tables together. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| Serves all sixmeasured · The tray hands each of the six tables a drink it will take, every drink used exactly once and none twice. A refusal leaves it undecided, and a tray that reuses a drink fails it. | ✓ pass | ✓ pass | ✓ pass |
Prompt v2, checks c1, harness h1. Ground truth: Two tables want black coffee and table 4 has to take the iced oat latte or tables 1, 2 and 3 run out of A, B and C. A serving exists, A to F in order, and the checker confirms it with its own matching solver. Refusing because it looks crowded is wrong. Every model on this task.
| check · how it is decided | run 1 | run 2 |
|---|---|---|
| Same filemeasured · 48 kHz, the same length within 10 ms, and no sample at full scale. | ✓ pass | ✓ pass |
| Hum downmeasured · The 50 Hz lines from 40 Hz to 16 kHz, within 0.5 Hz of each, at least 10 dB quieter than the pump on its own puts there. | ✓ pass | ✓ pass |
| Rush downmeasured · From 0.3 to 4.5 s, everything off the lines at least 3 dB quieter than the pump on its own puts there. | ✗ fail | ✗ fail |
| Music keptmeasured · Where the music is strong and the pump weak, each of four bands from 40 Hz to 8 kHz within 1.5 dB of the music, with no level correction. | ✓ pass | ✓ pass |
| Sounds rightby eye · Pass if, played straight after the music on its own, no note is missing or dulled and nothing new can be heard: no warble, no clicks, no hollow sound. Some hum or rush left over does not fail it, since Hum down and Rush down score that. | ✗ fail | ✗ fail |
Prompt v1, checks c1, harness h1. Ground truth: cafe.wav is 18.2 s of the Espresso Rag with a CC0 recording of a Nespresso pump 6 dB under it, and the checker holds the music and the pump apart. The pump is a vibration pump on 50 Hz mains: lines at every multiple of 50 Hz up past 14 kHz, which carry 89% of its energy, and a rush of water that is loudest in the first four seconds. Every model on this task.