Finding
Mistral Large 4 got the filter the right way up in one of its two drawings. In its thinking, two of its three runs had the order upside down before drawing anything.
Mistral Large 4 went up on OpenRouter on 6 October, at $0.68 per million tokens in and $2.09 out, and it has vision as well, so it can look at its own drawing.
The prompt, v2, frozen
Draw an exploded assembly diagram of a French press, as an SVG. Use the standard three-piece plunger filter rather than a two-piece travel press.
Run by run
Run 1 never drew anything. Run 2 wrote the whole file in one go, rendered it once and looked at it. It knew the hole was missing but did not have time to correct. Run 3 got the order right but missed a part.
Checks
| check · how it is decided | run 2clock stop | run 3 |
|---|---|---|
| All seven partsby eye · Pass if all seven are drawn as separate shapes and each has a label naming it. A part that is only named in a label or a comment, with nothing drawn, fails it. | ✓ pass | ✗ fail |
| Stack ordermeasured · Knob, lid, spiral plate, mesh, cross-plate and carafe, top to bottom along the axis, read from each part's y coordinate in the SVG. Its comment is ignored. The rod is left out, because it runs through the lid. | ✗ fail | ✓ pass |
| Filter fits the glassmeasured · Every filter piece drawn is 80% to 105% of the width of the carafe's glass wall, measured in the SVG. | ✓ pass | ✓ pass |
| One assembly axisby eye · Pass if one vertical line runs through the middle of every part, and every leader line leaves from the same side. One part pushed off to the side fails it. | ✓ pass | ✓ pass |
| Labels matchby eye · Pass if following each leader line from its label ends on the part that label names. One label pointing at the wrong part, or at nothing, fails it. | ✓ pass | ✓ pass |
Prompt v2, checks c1, harness h1. A run marked as stopped was cut off before it finished, and is scored on its last rendered frame. Ground truth: The standard three-piece plunger, from the top down: spiral (spring) plate, mesh screen, cross-plate. The cross-plate is the bottom piece, because its threaded stem is what the rod screws into. The spiral plate above the mesh is what presses the mesh edge out against the glass. Checked against the WebstaurantStore anatomy diagram, which is drawn exploded and labelled, and against each part's y coordinate in the SVG rather than its comment. Two-piece travel presses exist and stack differently, which is why v2 of the prompt names the design. Every model on this task.
17 of the 40 French press drawings where the order could be decided have it the right way up, so Mistral sits about where everyone else does.
Verdict
Run 2 had every part but the filter upside down, and run 3 had the filter the right way up but missed a part.
n = 3 runs, 2 with a drawing. No test.
The 2 runs cost $0.21. Run 1 was cut off before its reply finished, so it recorded no cost, and I estimate it at $0.06.