Executive summary
- “I tried it” stopped working. Model features give a different answer every time and fail quietly and articulately. Fifteen hand-picked inputs are not random, not repeatable and produce no number.
- An eval has four parts. A set of real inputs weighted toward the hard cases, a definition of good (usually a rubric), a grader (code, a model, and a human checking the model) and a score you rerun.
- It’s product work. Engineering owns the harness; product owns the definition of good. An eval is acceptance criteria for a system that isn’t exactly repeatable, and every customer complaint becomes a permanent test case.
- A week, in a spreadsheet. Collect fifty real inputs, write three to five criteria, score by hand, automate the grading and audit it, then change one thing and compare case by case.
- How it goes wrong. Easy test sets, rubrics that measure fluency, trusting the model grader blindly, overfitting, one number for everything, abandoning the eval and starting with a platform.
Evals measure only what you thought to test, and they don’t say whether the feature should exist. They are still the only thing that lets you say “this got better” and mean it.
The first AI feature I shipped, I tested like every feature before it. I opened it, tried ten or fifteen inputs, decided it was good, and shipped it.
It was good. For those fifteen inputs. I had no way of knowing how it behaved on the fifteen hundred I hadn’t tried. Or what would happen the next time someone changed a line in the prompt. Or whether the version we shipped beat the version from three days earlier.
Inputs I tried
15
Chosen by me, on one afternoon, from the cases I had designed for. All fine.
Inputs I did not
1,500
No idea how it behaved on any of them, or whether the version we shipped beat the one from three days earlier.
I’d been a product manager for a decade. I had strong opinions about acceptance criteria. And I had shipped something whose quality I could not put a number on, could not defend in a meeting, and could not tell had slipped until a customer told me.
That gap has a name, and the name is evals. This is the long version: what they are, why they’re a product job rather than an engineering one, how to build your first in a week, and all the ways I’ve seen them done badly — including by me.
Part one: why “it worked when I tried it” stopped working
Deterministic software has a nice property
For most of software history, testing rested on an assumption so basic nobody stated it: the same input produces the same output.
That one property makes traditional QA possible. You write a test: given this input, expect this output. It passes or it fails. You run a thousand of them in a minute. If one breaks, something changed, and you know exactly which one. Acceptance criteria work the same way — “when the user clicks Save, the record persists” is checkable, once, definitively.
As a PM you absorbed this so deeply you stopped seeing it. You wrote acceptance criteria the way you breathe. “Given / when / then.” The button works or it doesn’t.
Model-powered features don’t have that property
Ask a language model the same question twice and you get two different answers. Both may be good. One may be subtly wrong. Change the temperature, the model version, or one word in the prompt, and the answers shift in ways you cannot predict by reading the change.
This breaks the whole testing apparatus, and in a particularly nasty way: the feature keeps looking fine. It doesn’t crash. It doesn’t throw a 500. It produces fluent, confident, plausible output that is wrong in a way only domain knowledge can detect.
Traditional software fails loudly. AI features fail quietly and articulately. That’s the whole problem in one sentence.
What “I tried it” is actually measuring
When you open a feature and try fifteen inputs, you’re taking a sample. A tiny one, chosen by you, on a day when certain things were on your mind. That sample has three problems:
It’s not random. You test the cases you thought of, and those are the cases you designed for. The failures live in the cases you didn’t think of — the user who pastes in a whole email thread, the transcript with three people talking over each other, the question in Turkish when you built it in English.
It’s not repeatable. Try the same fifteen inputs tomorrow and you’ll get fifteen different outputs. Your judgment of them will differ too: you’ll be in a different mood, and you’ll have forgotten what yesterday’s looked like.
It produces no number. So you cannot answer the only two questions that matter in a product meeting: is this good enough to ship? and is the new version better than the old one? You can only say “it seems good,” which in a room with a skeptical engineering lead is worth nothing.
The moment it bites
It bites at one of four moments, and everyone who ships AI features hits at least one:
The model updates
Your provider ships a new version. Nothing in your repository changed and your prompt behaves differently.
How you find out: from a customer.
Someone edits the prompt
A small change to fix one case quietly breaks four others.
How you find out: you don’t. Nobody checked the four.
You want to cut costs
A cheaper model would save real money. Is it good enough?
How you find out: you either don’t switch, or you switch and hope.
A customer complains
Someone reports a bad output and you fix it.
How you find out whether the fix broke anything else: you don’t, and the same class of failure returns in three months.
Every one of these is the same problem: no measurement, so no basis for a decision.
Part two: what an eval actually is
Strip away the tooling and the jargon and an eval has four parts. That’s it.
1. A test set
A collection of real inputs. Not invented ones — real. For a summarizer, documents people summarized. For a support bot, customer questions, including the badly written ones. For a synthesizer, transcripts with crosstalk and tangents.
Size: fifty is enough to start and far better than zero. Two hundred is comfortable. A thousand is a mature system. People delay starting because they think they need thousands; they don’t.
The essay’s guidance. Composition matters more than size: bias the set toward the hard cases.
The numbers
| Stage | Cases |
|---|---|
| To start | 50 |
| Comfortable | 200 |
| Mature | 1,000 |
Composition matters more than size. A good test set leans on purpose toward the hard cases: the ambiguous, the adversarial, the wrong language, the empty input, the enormous input, the one where the right answer is “I don’t know.”
2. A definition of good
This part is yours, and it’s the heart of the thing.
Sometimes “good” means a reference answer — for a classification task, the correct label. More often, especially for generative features, it’s a rubric: a short list of criteria you can judge one by one.
A rubric for a meeting-summary feature might be:
A decision is anything the room settled: a date, an owner, a choice between two options. Not settled is not a decision.
The name in the summary is the name in the transcript. An unowned action item named as unowned is a pass.
No inferred deadlines, no tidied-up reasoning, no context the meeting did not contain.
Mechanical, so no judgment and no model decides it.
3 — they know what was decided and what is theirs. 2 — they know roughly, and would have to ask. 1 — they would have to read the transcript.
Five criteria. Anyone can apply them. No machine learning knowledge required — but deep knowledge of what the user needs, which is why this is product work.
3. A grader
Something that applies the rubric to each output and produces a score.
A human — you, at the start. Slow, expensive, and the most accurate. You will do this for your first fifty, and it will teach you more than anything else you do.
A model — a second LLM reading the output against your rubric. This is how you scale past a few hundred cases. It’s surprisingly good, and it needs auditing: spot-check whether its judgments match yours.
Code — for anything mechanical. Valid JSON? Under 200 words? Contains a citation that resolves to a real source? Cheap, fast, perfectly reliable. Use code wherever code can do the job.
Most real evals use all three: code for the mechanical criteria, a model for the judgment ones, and human spot-checks to keep the model grader honest.
4. A score you rerun
The number. Percentage of cases passing, or average rubric score, or pass rate per criterion. What matters is that it’s the same number, computed the same way, every time, so when you change something you can see whether it got better.
That’s an eval. A set of real inputs, a definition of good, something that applies it, and a number you can compare. Everything else in this essay is detail.
Part three: why this is the PM’s job
I want to be precise here, because “AI evals are the new PRD” has been thrown around so loosely it means nothing.
The split
Engineering owns the harness. Running the cases, calling the models, storing results, wiring it into CI, making it fast. That’s infrastructure, and it’s engineering work.
Product owns the definition of good. What goes in the test set. What the criteria are. What weight each carries. What the threshold for shipping is. What counts as a failure severe enough to block a release.
That second list is not a technical list. Every item on it is a judgment about what the user needs and what the business can tolerate.
The concrete argument
Take the meeting summarizer. Suppose an output is fluent, accurate in every sentence, well under the word limit — and omits the one moment where two people disagreed about the deadline.
Is that a pass or a fail?
An engineer can’t answer that. It depends on what the summary is for. If it exists so someone who missed the meeting knows what was decided, the omission is catastrophic — the disagreement is the most important thing in the transcript. If it exists as a searchable record, the omission is minor.
Someone has to make that call, write it into the rubric, and own it. That’s a product decision in exactly the way “what does done mean” always was.
The continuity
Here’s the framing I found most useful: an eval is acceptance criteria for a system that isn’t exactly repeatable.
You already know how to do this. You’ve been writing “given / when / then” for years. Now, instead of one expected output you have a rubric, and instead of pass/fail on one case you have a rate across many. The underlying skill — deciding what good means and writing it down precisely enough that someone else can apply it — is one you already have.
The PMs who struggle with evals usually aren’t struggling with the technical part. They struggle because they never had to define quality that precisely; a deterministic system did it for them.
The other reason it’s yours
Practical, not philosophical: the test set is where customer complaints go to become permanent.
A customer reports a bad output. You fix the prompt. Without an eval, that fix is a one-off and the same class of failure comes back in three months. With an eval, the complaint becomes a test case, and that failure can never silently return.
That loop — complaint to test case to permanent protection — is the most valuable thing an eval gives you, and it’s a product loop.
Part four: building your first one in a week
Concrete: five days, no ML background, no budget approval. Start with a spreadsheet. Graduate later, if ever.
Day 1 — Collect fifty real inputs
Go to where real inputs live: support tickets, logs, the documents people upload, the questions people ask. Copy fifty into a spreadsheet, one per row.
Rules:
- Real, not invented. Invented inputs are always cleaner than reality.
- Bias toward hard. If it’s all easy cases your score will be high and meaningless.
- Include at least five that you expect to fail. If you can’t think of five, you don’t understand the feature yet.
- Include the degenerate ones: empty, enormous, wrong language, wrong format, hostile.
This is the asset. Everything else is replaceable.
Day 2 — Write the rubric
Three to five criteria. Each one yes/no or a 1–3 scale. Write them so two people applying them to the same output would agree.
Holds
- “Every factual claim in the output appears in the source document.”
- “Under 200 words.” Mechanical, so it never needs a judgment.
- “Names the person each action item belongs to.” One thing, one verdict.
- “Says so when the source does not answer the question.”
Collapses
- “The output is high quality.”
- “Accurate, complete and well-formatted.” Three criteria, one verdict.
- “Reads naturally.” Models are already good at this; it will never move.
- “Captures the important parts.” Important to whom, decided when?
The test for a criterion: could someone who’s never used your product apply it correctly after reading one sentence of explanation? If not, sharpen it.
Put first the criteria that catch the failures that would embarrass you. Fluency takes care of itself; fabrication doesn’t.
Day 3 — Score the current version by hand
Run all fifty inputs through your feature as it is today. Paste the outputs into the spreadsheet. Score each against the rubric, yourself, by hand.
This day is unpleasant, and it’s the most valuable one. Two things happen. You get your baseline — the number everything after is compared to. And you learn how your feature really fails, which is not how you assumed it fails.
Expect to be humbled. My first baseline was far lower than my impression of the feature, and the whole gap was in cases I’d never have thought to try.
Day 4 — Automate the grading
Now make a model apply the same rubric, with a prompt like: here is an output, here is the source, here are five criteria; for each, answer yes or no and give a one-line reason; respond as JSON.
Then audit it: take ten cases you scored by hand and compare the model’s scores to yours. If they agree on eight or nine, the grader is usable. If they disagree wildly, your rubric is ambiguous — useful news about the rubric, not the model.
Your coding agent will write the script that loops through the spreadsheet in twenty minutes. This is not hard engineering.
Day 5 — Change one thing and rerun
Edit the prompt. Switch the model. Adjust the retrieval. One change.
Rerun. Compare. Look not just at whether the number moved but at which cases moved — three fixed and two broken is a different story from five fixed, and only the case-level view tells you which.
You now have an eval. It took a week, cost nothing, and you can answer both of the questions you couldn’t answer before.
Part five: how evals go wrong
I’ve done most of these.
Test sets that are too easy
You write the inputs yourself, so they are clean, well-formed and just what you imagined. You score 95% and ship with confidence. Reality is messier, and the 95% meant nothing.
Fix: every input comes from production or a real user. If you must write some, write them adversarially.
Rubrics that measure fluency
“Is the output well-written?” Models are very good at well-written. You will score high on every version of everything and learn nothing.
Fix: measure what models are bad at — not fabricating, not omitting, following constraints, admitting uncertainty.
Trusting the model grader blindly
Model-as-judge is the only way to scale, and it has biases. They are stronger in some judges than others: it can prefer longer answers, it tends to favor whichever answer it sees first, and it is generous with its own output.
Fix: audit ten cases against your own scores whenever the rubric or the grader changes. Never stop entirely.
Over-fitting to the test set
You iterate on the prompt until the number goes up, and up, and up. Then you ship, and users hit problems you never saw. You optimized for fifty cases, not the feature.
Fix: hold back a slice you never look at while iterating, and keep adding new cases from production.
One number for everything
A single 87% hides that fabrication is at 99% and completeness at 61%. Those are wildly different problems, and the average tells you to work on neither.
The essay’s example. Report per criterion: the aggregate is for the slide, the breakdown is for the work.
The numbers
| Measure | Score |
|---|---|
| Overall | 87% |
| No fabrication | 99% |
| Completeness | 61% |
Fix: report per criterion. The aggregate is for the slide; the breakdown is for the work.
Building it and abandoning it
The most common failure by far. Someone builds a beautiful eval, the team uses it for two weeks, then a deadline hits. Eight months later it is stale and nobody trusts it.
Fix: make adding a case take under a minute, put the score where people see it, and tie a run to a release ritual.
Starting with a platform
The tools are good. They are also a way to spend three weeks on buying and setup and never learn how your feature fails.
Fix: spreadsheet first, always. Graduate when a second person edits the set or a second model is in play.
Part six: what evals can’t do
I want to be honest about the limits, because the enthusiasm has gotten ahead of itself.
They measure what you thought to test. Every failure outside your test set is invisible, and the failures that hurt most are, almost by definition, the ones nobody expected. You can’t fix this, only soften it: sample real production output by hand, regularly, forever, and feed what you find back in.
They don’t tell you if the feature should exist. A perfect score on a feature nobody wants is a well-tested waste of time. Evals measure execution, not whether the bet was right. The metric does that job; they’re different instruments.
They can be gamed, even by accident. If the score is the goal, you’ll optimize the score. Keep the eval as an instrument, not a target — the moment a team is judged by the eval number, the number stops telling you anything.
They don’t replace judgment. The rubric is your judgment, written down. It’s only as good as the thinking behind it, and it goes stale as the product and its users change. Revisit it quarterly.
They cost real time. Building the first one takes a week. Maintaining it takes an hour or two a month. That’s cheap next to shipping a silently broken feature, but it isn’t free, and pretending otherwise is how evals end up abandoned.
Where to start, if you start anywhere
Pick the AI feature you’re least sure about. Collect fifty real inputs this week. Write five criteria. Score them by hand.
You’ll learn more about your product in that one day of hand-scoring than in a month of dashboards. And you’ll have the start of the only thing that lets you say “this got better” and mean it.
I’m building one for each of the tools in my own series — the interview synthesizer’s test set is a folder of studies I once synthesized by hand, and the bet scorer’s is a set of bets whose outcomes I now know. Both will produce numbers I’d rather not see. That’s the point of measuring.
— Chris
Sources
Where the evidence comes from. Each row opens the document itself; the tag says which part of the argument it supports, and ↩︎ goes back to the sentence that uses it.
- An Empirical Study of the Non-determinism of ChatGPT in Code GenerationOuyang, Zhang, Harman, Wang (ACM TOSEM; first posted as “LLM is Like a Box of Chocolates”)LLM non-determinismPaper2023arXiv:2308.02828same prompt, different outputs across runs
- How is ChatGPT's behavior changing over time?Lingjiao Chen, Matei Zaharia, James ZouModel driftPaper2023arXiv:2307.09009same model name, behavior changed in months
- Judging LLM-as-a-Judge with MT-Bench and Chatbot ArenaLianmin Zheng et al. (NeurIPS 2023 Datasets & Benchmarks)LLM-judge biasesPaper2023arXiv:2306.05685Position, verbosity and self-enhancement bias in judges
- LLM Evaluators Recognize and Favor Their Own GenerationsArjun Panickssery, Samuel R. Bowman, Shi Feng (NeurIPS 2024)Self-preference biasPaper2024arXiv:2404.13076judges score their own outputs higher
- Define success criteria and build evaluationsAnthropic (Claude Platform Docs)Grader typesDocscurrentcode, LLM and human grading; edge cases
- The reusable holdout: Preserving validity in adaptive data analysisDwork, Feldman, Hardt, Pitassi, Reingold, RothReused holdout overfittingJournal2015Science 349(6248):636-638adaptive reuse of test data overfits
- Categorizing Variants of Goodhart's LawDavid Manheim, Scott GarrabrantGoodhart's lawPaper2018arXiv:1803.04585optimizing a metric degrades it
- Your AI Product Needs EvalsHamel HusainPractitioner eval guideArticle2024-03evals beyond vibe checks; three-level system
Series
The job, rewritten
What product management became once building stopped being the expensive part — one argument at a time.
- 1The product manager, before and after
- 2What replaces the PRD
- 3Evals: the product skill nobody taught you
- 4Should a PM know how to read code in 2026?
- 5How to hire a PM now
+The 2026 product leader's toolsetCompanion
The tool
Rubric
The four parts above, built: a test set, criteria you write, a grader, and a score that survives to the next run. The case study shows the results screen with an example eval already scored, and what the thing could not do.
Read the case study →