Finding
I gave each model these two pictures and asked: "Name everything wrong with it, most serious first. Say if you think nothing is wrong." Neither press has anything wrong with it.
| Says the filter is upside down | Opus's press | Pareto's press |
|---|---|---|
| Opus 5.5 | 3 of 5 | 2 of 5 |
| Pareto | 0 of 5 | 1 of 5 |
Verdict
So about half the time, Opus thinks the filter is upside down, which is about as often as it draws it upside down. Pareto said it was upside down in 1 of 10, and it never is.
n = 5 runs a model on each press. No test.
Looking back at when it drew the french press, every time it drew it upside down, it also said it upside down. So it is consistent between its words and its drawing. Its answer about the order just changes from run to run.
The 20 runs cost $2.26.