Finding
Both cost under a cent a run, so I gave them the same eight tasks, three runs each. Haiku 5.5 is new on OpenRouter and pi has no settings for it so I borrowed its settings from Haiku 4.5.
Taking things apart
| Parts in the right order | Luna | Haiku 5.5 |
|---|---|---|
| French press | 2 of 3 | 0 of 3 |
| AeroPress | 3 of 3 | 1 of 2 |
| V60 | 3 of 3 | 0 of 3 |
| Portafilter | 3 of 3 | 0 of 3 |
| Grinder | 1 of 3 | 0 of 3 |
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| All three partsby eye · Pass if every listed part is drawn as a separate shape and each has a label naming it. A part only named in a label or a comment, with nothing drawn, fails it. The label must say what the part is, in any words: a name that only says where it sits ("top plate") or what it is made of, or a mention in another part's caption, does not count. | ✓ pass | ✓ pass | ✓ pass |
| Stack ordermeasured · Filter, dripper, server, top to bottom along the axis, read from each part's coordinates in the SVG. | ✗ fail | ✗ fail | ✗ fail |
| Parts fitby eye · Pass if each listed pair of mating parts looks as if it would go together as stated: the right angle, nested inside, or sitting on top. One pair that plainly could not fails it. | ✓ pass | ✓ pass | ✓ pass |
| One assembly axisby eye · Pass if one line runs through the middle of every part on each listed axis, and every leader line leaves from the same side. One part pushed off its axis fails it. | ✓ pass | ✓ pass | ✓ pass |
| Labels matchby eye · Pass if following each leader line from its label ends on the part that label names. One label pointing at the wrong part, or at nothing, fails it. | ✓ pass | ✓ pass | ✓ pass |
Prompt v1, checks c1, harness h1. Ground truth: Top to bottom: paper filter, dripper, server. The dripper is a 60 degree cone with spiral ribs and one large hole, and it sits directly on the server. Checked against Hario's V60 and Range Server pages. The server's lid is optional, so it is not required and not marked down. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| All three partsby eye · Pass if every listed part is drawn as a separate shape and each has a label naming it. A part only named in a label or a comment, with nothing drawn, fails it. The label must say what the part is, in any words: a name that only says where it sits ("top plate") or what it is made of, or a mention in another part's caption, does not count. | ✓ pass | ✓ pass | ✓ pass |
| Stack ordermeasured · Filter, dripper, server, top to bottom along the axis, read from each part's coordinates in the SVG. | ✓ pass | ✓ pass | ✓ pass |
| Parts fitby eye · Pass if each listed pair of mating parts looks as if it would go together as stated: the right angle, nested inside, or sitting on top. One pair that plainly could not fails it. | ✓ pass | ✗ fail | ✓ pass |
| One assembly axisby eye · Pass if one line runs through the middle of every part on each listed axis, and every leader line leaves from the same side. One part pushed off its axis fails it. | ✓ pass | ✓ pass | ✓ pass |
| Labels matchby eye · Pass if following each leader line from its label ends on the part that label names. One label pointing at the wrong part, or at nothing, fails it. | ✓ pass | ✓ pass | ✓ pass |
Prompt v1, checks c1, harness h1. Ground truth: Top to bottom: paper filter, dripper, server. The dripper is a 60 degree cone with spiral ribs and one large hole, and it sits directly on the server. Checked against Hario's V60 and Range Server pages. The server's lid is optional, so it is not required and not marked down. Every model on this task.
The Venn goes the other way
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| Compilesmeasured · pdflatex builds the file offline. | ✓ pass | ✓ pass | ✓ pass |
| Seven regions, three namesmeasured · Three circles found in the PDF with every overlap present, and each set name placed by its circle. | ✓ pass | ✓ pass | ✓ pass |
| Every drink in its regionmeasured · Each of the eight words found in the PDF, and the centre of each one inside the region it belongs to. | ✓ pass | ✗ fail | ✓ pass |
| No label on an edgemeasured · No letter touches a circle's outline, with the line's width counted. | ✓ pass | ✗ fail | ✓ pass |
Prompt v1, checks c2, harness h1. Ground truth: Short black is espresso only, flat white espresso and milk, iced long black espresso and cold, iced latte all three, babyccino milk only, milkshake milk and cold, cold brew cold only, and pour-over outside every circle. The circles and words are read back out of the compiled PDF, each word has to be centred in its own region, and no letter may touch a circle's outline, with the line's width counted. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| Compilesmeasured · pdflatex builds the file offline. | ✓ pass | ✓ pass | ✓ pass |
| Seven regions, three namesmeasured · Three circles found in the PDF with every overlap present, and each set name placed by its circle. | ✓ pass | ✓ pass | ✓ pass |
| Every drink in its regionmeasured · Each of the eight words found in the PDF, and the centre of each one inside the region it belongs to. | ✓ pass | ✓ pass | ✓ pass |
| No label on an edgemeasured · No letter touches a circle's outline, with the line's width counted. | ✗ fail | ✗ fail | ✗ fail |
Prompt v1, checks c2, harness h1. Ground truth: Short black is espresso only, flat white espresso and milk, iced long black espresso and cold, iced latte all three, babyccino milk only, milkshake milk and cold, cold brew cold only, and pour-over outside every circle. The circles and words are read back out of the compiled PDF, each word has to be centred in its own region, and no letter may touch a circle's outline, with the line's width counted. Every model on this task.
The shrub and the rag
Neither passes either one. Luna grows a full shrub and never prunes it, while Haiku prunes all three and gets the size wrong. On the rag, Haiku splits the stems in all three and Luna in one.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| Reads as a shrubby eye · Pass if it reads as one plant with branches coming off stems, without being told what it is. A tangle, a lattice or loops fail it. | ✓ pass | ✗ fail | ✗ fail |
| Flat side from the rulesby eye · Pass if one side ends on a straight line that the rules themselves put there. A box drawn around the plant to cut it fails it. | ✓ pass | ✓ pass | ✓ pass |
| Sensible sizeby eye · Pass if the branches can still be followed by eye. So many segments that they merge into a mass fails it. | ✗ fail | ✗ fail | ✗ fail |
Prompt v1, checks c2, harness h1. Ground truth: The JSON expanded and drawn with turtle graphics offline, and the drawing judged by eye. There is no single right L-system, so nothing here is measured. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| Reads as a shrubby eye · Pass if it reads as one plant with branches coming off stems, without being told what it is. A tangle, a lattice or loops fail it. | ✓ pass | ✓ pass | ✓ pass |
| Flat side from the rulesby eye · Pass if one side ends on a straight line that the rules themselves put there. A box drawn around the plant to cut it fails it. | ✗ fail | ✗ fail | ✗ fail |
| Sensible sizeby eye · Pass if the branches can still be followed by eye. So many segments that they merge into a mass fails it. | ✓ pass | ✓ pass | ✓ pass |
Prompt v1, checks c2, harness h1. Ground truth: The JSON expanded and drawn with turtle graphics offline, and the drawing judged by eye. There is no single right L-system, so nothing here is measured. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| Piano staffmeasured · A treble staff and a bass staff, both with notes. | ✓ pass | ✓ pass | ✓ pass |
| Sixteen barsmeasured · The last note is in bar 16. | ✓ pass | ✓ pass | ✓ pass |
| Left hand crossesmeasured · In each of bars 9 to 12 most left-hand notes are on the treble staff and sit above the right hand, and outside those bars most are back on the bass staff. A left hand written high on the bass staff fails. | ✓ pass | ✓ pass | ✓ pass |
| Right hand keeps goingmeasured · The right hand plays in every bar, and at least twice in each of bars 9 to 12. | ✗ fail | ✓ pass | ✓ pass |
| Crossing markedmeasured · Words naming a hand or the crossing at bar 8 or 9, and a release at bar 12 or 13. | ✗ fail | ✗ fail | ✗ fail |
| Stems apartmeasured · In bars 9 to 12, where both hands share a staff, at least three in four of their notes have stems pinned apart: left hand up, right hand down. | ✓ pass | ✓ pass | ✓ pass |
Prompt v1, checks c1, harness h1. Ground truth: LilyPond's own record of every note: its hand, staff, bar, pitch and stem, read while the file is engraved. A voice belongs to the hand whose staff it starts on, unless its notes are labelled with a hand's name. The script was written after hearing Sol's and Pareto's runs, so Luna was its first fair test. How good the music sounds is rated by ear. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| Piano staffmeasured · A treble staff and a bass staff, both with notes. | ✓ pass | ✓ pass | ✓ pass |
| Sixteen barsmeasured · The last note is in bar 16. | ✓ pass | ✓ pass | ✓ pass |
| Left hand crossesmeasured · In each of bars 9 to 12 most left-hand notes are on the treble staff and sit above the right hand, and outside those bars most are back on the bass staff. A left hand written high on the bass staff fails. | ✓ pass | ✗ fail | ✓ pass |
| Right hand keeps goingmeasured · The right hand plays in every bar, and at least twice in each of bars 9 to 12. | ✓ pass | ✓ pass | ✓ pass |
| Crossing markedmeasured · Words naming a hand or the crossing at bar 8 or 9, and a release at bar 12 or 13. | ✗ fail | ✗ fail | ✗ fail |
| Stems apartmeasured · In bars 9 to 12, where both hands share a staff, at least three in four of their notes have stems pinned apart: left hand up, right hand down. | ✗ fail | ✓ pass | ✗ fail |
Prompt v1, checks c1, harness h1. Ground truth: LilyPond's own record of every note: its hand, staff, bar, pitch and stem, read while the file is engraved. A voice belongs to the hand whose staff it starts on, unless its notes are labelled with a hand's name. The script was written after hearing Sol's and Pareto's runs, so Luna was its first fair test. How good the music sounds is rated by ear. Every model on this task.
Luna’s 24 runs cost $0.13 and Haiku’s 24 cost $0.16, so $0.30 for the 48.