← All articles

What are Evals?

Evals · · 2 min read

Evals are tests for AI features — the way you check whether a model-powered product is actually doing what you want, because unlike normal software it doesn’t do the same thing every time.

With a traditional feature, you write acceptance criteria and QA checks them: the button either works or it doesn’t. With an LLM feature — a summarizer, a chatbot, a recommender, your interview synthesizer — there’s no single correct output. The same input can produce a hundred slightly different answers, some great, some subtly wrong, and “it worked when I tried it” tells you nothing.

An eval replaces “I tried it” with a repeatable measurement:

  1. A test set. Fifty to a few hundred real inputs — actual customer questions, actual transcripts, actual half-formed ideas — chosen to cover the range of what users will throw at it, including the ugly cases.

  2. A definition of good. For each input, what a good output looks like. Sometimes that’s a reference answer; often it’s a rubric — “must mention X, must not invent Y, must be under 100 words, must link to a quote.”

  3. A grader. Either a human scoring outputs against the rubric, or another model doing it at scale, or a simple check (did it return valid JSON, did it cite a real source).

  4. A score you run every time something changes. New prompt, new model, new data — run the set, compare the number. If it dropped, you know before your users do.

Why it’s the PM’s job now: defining “good” is a product decision, not an engineering one. An engineer can build the harness, but only the person who understands the user can say that a summary which is accurate but misses the one contradiction is a failure. That’s the same skill as writing acceptance criteria, applied to a system that’s never fully deterministic.

Concretely for your tools: the interview synthesizer’s eval is a folder of old transcripts you’ve already synthesized by hand, and a check of whether the tool finds the same patterns and contradictions you found. You said you’d judge it that way in Part 3 — that is an eval. You just hadn’t called it one.