Finding
Every score here comes from a script or from me looking at it, so if the checker is wrong the board is wrong. Here are the times I caught it.
| What went wrong | What I did |
|---|---|
| I graded 20 random runs by hand without seeing the checker, and disagreed on 5, all French presses | Looked again and 2 were still off, both on labels |
| The Venn checker failed labels that were clear by eye | Changed it to test each letter against the line, and re-graded |
| DeepSeek found the checker and ran it on its own answer | Threw out both runs, and the checkers now run after the model is done |
| The tray checker misread some trays and missed a no verdict | Had every reply and tray read again |
Now before each post I grade one run per model by hand and compare with the checker. The checkers are in the evidence repo.