Steps
Path 3 · 5 steps · about 58 min, plus the exercises
Evals: the product skill nobody taught you
Why trying a model feature by hand stopped telling you anything, and what an eval is instead: real inputs, a definition of good, a grader, and a score you can rerun.
Before the next step
Pick one AI feature you own or use. Write down ten inputs real people sent it, not ones you made up. Mark the three you would bet it gets wrong.
The rubric that replaced the spec
How to write the definition of good: a criterion with a name, a plain definition, a type and a severity, and why a blocking failure never averages out.
Before the next step
Write three criteria for your feature, and make at least one of them blocking. Hand them and two outputs to a colleague, and see whether you reach the same verdict.
Where an eval lives
One section of the toolset piece: why a spreadsheet and a short script is the right place to start, and what would make a dedicated platform worth it.
Before the next step
Put your ten inputs and three criteria in a spreadsheet, one row per input and one column per criterion. Score every cell by hand.
Rubric, the case study
The tool built from the two essays above: the problem it was for, the screen it exists for, and what it cannot do.
Before the next step
Find one thing the case study says the tool cannot do, and decide whether it matters for your feature.
Run your first eval
Your cases and criteria in the tool, graded by hand or by a model with your own API key. The project stays in your browser; nothing is sent anywhere except the grading calls you start.
Before the next step
Change one thing in your feature, the prompt, the model or one criterion’s wording, paste the new answers and run again. Write down which cases moved in the diff, not just whether the score did.
The Bets & Ships newsletter
Get the next one by email
For product leaders and hiring managers: how product work is changing after AI, with the evidence attached.
- One essay every TuesdayHow product work is changing.
- The evidence attachedCharts, numbers, every source.
- Case studies from the buildThe tools, and what broke.
Free. You’ll get one email to confirm first.