All Posts
How often is my checker wrong?
More often than I'd like, so now I check it before each post.
Do models stop when a task is impossible?
Most models notice the task is impossible, but most still hand something over.
How good are models at separating music from background coffee machine noise?
They don’t sound right to me. The model really struggled.
Do models say no when a set of coffee order is impossible?
They notice, but more than half still serve something.
How does Kimi K3.1 Pro do on the five busiest tasks?
It built the hopper right every time.
How good are models at fixing things?
Fixing is a different skillset than making
Do models know how to stir a coffee?
Some models have the spoon flat on the surface to stir and others have the spoon leaving the cup while it stirs.
What do models like to do when given the opportunity?
Three ideas came up more than once: moths at a light, slime moulds and paper boats.
How do models fail when simulating a coffee hopper?
Pareto 26.9 got all three runs right. Haiku 5.5 got two right, and Gemini 4 Argon handed nothing in.
Are models better at tasks where the grader can be automated?
It checks out, across the board, measured checks pass more often.
Does Gemini 4 Argon know the coffee family tree?
Gemini 4 Argon got the Bourbon family right in all three runs.
Are models as bad as humans in the spatial orientation test?
They are really good and within a few degrees of the answer.
Does a model know a knit stitch is wider than it is tall?
Five of six models know it. GLM 5.3 flashx drew stitches tall and said so.
Opus 5.5 says a right French press is upside down
When shown french presses with the filter the right way up, Opus 5.5 thinks they are the wrong way about half the time.
How do models respond to an impossible task?
They knew it couldn't be done, and 15 of the 16 drawings are five circles anyway.
Luna against Haiku 5.5, which model gets the coffee tasks right?
Luna gets the parts in order in 12 of 15 drawings and Haiku in 1 of 14.
Can Mistral Large 4 make a French press?
It’s a French model after all, and it got one drawing upside down, one the right way up and one unfinished.
Can models follow a musical instruction?
Only Sol so far marks the crossing and keeps the stems apart in every run.
How jagged is a model?
Some models are great at doing hard things but poor at doing things that most models find easy, they are the most jagged.
Four tasks got too easy including the new Penrose cafe floor task!
Whenever a task passes every check in three runs for two different models, it comes off the board.
Opus 5.5 knows the order of filter in a french press, but can it draw it?
Asked in words, Opus 5.5 got the filter order right in 5 of 5 runs, and asked to draw it, in 2 of 5.
Four cheap models grow a coffee shrub pruned on one side
Models drawing a half pruned coffee tree
Sonnet 5.5 against Opus 5.5, which one costs more and does best on the Venn, the French press and the shrub?
Opus passed the Venn in 3 of 3 runs and Sonnet in 2. Sonnet passed the French press in 2 of 3 and the shrub in 1 of 3, and Opus passed neither in 3 runs each. Sonnet is cheaper a run on all three.
Does measuring help a model lay out a Venn? With a bit more statistics.
Sol kept the labels clear nine times out of ten whether I gave it a list, told it to check itself or said nothing.
Does measuring help a model lay out a Venn?
Telling the models to keep labels off the lines worked about as well as telling them to measure.
Qwen3.8 Flash vs DeepSeek 4.1 Flash draw a coffee Venn
Qwen gets all three runs right and DeepSeek only one.
Qwen3.8 Max Prime draws three coffee shrubs
Three very different shrubs, and only one looks right.
Ten models draw a coffee Venn, some check with vision others with code
Almost all put the drinks in the right place, but only a few keep the labels off the lines.
DeepSeek found the grader and ran it
It found my grading script and ran it on its own answer, so no score.
Opus 5.5 got the French press right, then changed it
Nicest drawings so far, but the stack is right in only one of three runs.
Two of my headlines were wrong after three runs
The croissant and the boiler headlines didn't hold up with more runs.
Bonsai vs Qwen3.8 27B draw a French press
Neither of them drew a filter that works.
GLM 5.3 FlashX draws a French press without the cross-plate
It used a base and a handle instead of the cross-plate.
Pareto knows the part order but can't draw it
It lists the parts in the right order, but the drawing is upside down.
Someone ran the control on the pelican test
The labs don't seem to train on the pelican.
Pareto vs Sol draw a nice french press with the three filter piece upside down
Both models know all the parts, but the order of the filter was wrong.
Union Alpha spells COFFEE on the streets of Melbourne
All checks out
Astra vs DeepSeek 4.1 Flash both build a boiler
All checks out with temps 92–96 °C
DeepSeek 4.1 Flash can't make croissants
Not a nice croissant
Qwen3.8 Max builds a caffeine molecule
Every check passes
Welcome to Jagged
Controlled tests of what new models can and can't do, checked against ground truth.