The jagged frontier

The same model can be very good at one task and strangely bad at the one next to it.

So I give new AI models tasks with a right answer and look for those edges. More specifically, I test whether AI-generated artifacts actually satisfy the instructions, and publish the evidence.

In the example below, Pareto lists in text the parts of the French press filter in the right order. But then when it drew them, it drew them in the wrong order.

Pareto's exploded French press, with the filter stack upside down — Fig. 1. Pareto’s French press, with the filter stack upside down. The note

Method

Prompts	Each task fits in one frozen, versioned prompt, and two results compare only on the same version. That is where AI does best, so the board says little about work that needs context you cannot paste in.
Runs	Three per model, because one run tells you what happened once.
Checks	Three to five pass or fail checks per task against a known answer, each decided by a number or by my eye. They run afterwards in a separate container with no network, and the model never sees them.
Agent	pi 0.83.0, pinned, with four tools: shell, read (pictures included), write and edit. No web search.
Model	Through OpenRouter at medium thinking and the provider’s default temperature, unless the result page says otherwise. OpenRouter names the model and not the snapshot, so a model can change under the same name.
Container	A fresh Docker container per run, with 2 CPUs, 2 GB and the renderers installed.
Loop	The model writes a file, renders it, looks at the picture and fixes it. It gets eight renders or twelve minutes, and the last rendered frame is scored.
Costs	I pay for every run, and each result shows its cost, failed runs included.

The full rules Every prompt, checker and scored file, on GitHub

Who writes it

I’m Fiacre Rougieux, an engineer and academic. I work on solar energy, data science and machine learning, and I use AI a lot in my research and with industry partners. This is where I check what it can actually do.

Corrections

Research is my day job, so when I get something wrong, I say so at the top of the post, dated, with the original wording left in. A run that breaks the rules, like the two caffeine runs that found the grader, is voided and written up. If you think a check is wrong, tell me the right answer and I’ll rerun it.

Follow along

A new model ships, I run the tasks on it, and the result goes up here. You can get one email a month, or every post by feed.

Paste the feed into any reader. No account needed.