The rubric that replaced the spec

Executive summary

• The spec’s assumption broke. A model-backed feature answers the same input differently each time, so you can’t describe the output. You describe the standard you’ll hold outputs to, precisely enough that two people reach the same verdict.
• A criterion has four parts. A name, a plain-English definition that spells out its edges, a type (yes/no by default, a three-point scale only where a real middle exists) and a severity.
• Two numbers, read together. Blocking failures don’t average. In the worked eval a 76% score hides five release blockers in twelve cases, and the per-criterion breakdown turns that grade into a work queue.
• Audit the grader, fix the criterion. Grade a sample by hand and measure agreement. Treat each disagreement as a hole in the definition, not a fault in the model.
• The diff, not the score. The typical failure is a score that rises while three cases quietly break. Keep every run, compare case by case, and rerun on every change, vendor release and incident.

A rubric doesn’t replace judgment. It writes the judgment down first, specifically enough that you can notice when it’s wrong.

Product specs rest on an assumption that quietly stopped being true: that the same input produces the same output. Write the acceptance criterion, check it once, and it stays checked.

A model-backed feature breaks that. The same support question, asked twice, returns two different replies. Both may be fine. One may invent a returns policy. Nothing in the build changed between them. There is no line of the spec to point at and no single test to run, because there is no single correct output — only a range of acceptable ones and a set of ways to fall outside it.

So the artifact shifts. Instead of describing the output, you describe the standard you will hold outputs to, precisely enough that two people reading it reach the same verdict on the same reply. That document is the rubric. It is the nearest thing this era of product work has to a spec, and unlike a spec it is built to be run again after every change, forever.

The job has changed, and not subtly. The product leader used to deliver a description of the thing to build. Now it is a definition of what counts as good, plus evidence that the current build meets it, plus a record of what changed the day it stopped.

What a criterion actually is

A criterion is not a bullet point. It carries four things, and each one bears weight the moment a disagreement reaches a review.

A name short enough to say out loud. Without one, people refer to criteria by number and nobody remembers which is which.

A definition in plain English, specific enough that two graders agree, and explicit about the edges. Take this one, from an eval of a support assistant:

Invents nothing. No restock dates, tracking details, discounts or policies that were not in the message or the policy above. Saying “I do not know” is a pass.

That last sentence is the whole craft. Without it, half your graders mark a refusal as a failure to help and half mark it as correct caution, and the score is an opinion poll.

A type. Yes/no for facts, a three-point scale for the few qualities that really have a middle. And a severity, the field most rubrics leave out and the one that does the most work.

Yes/no, or a scale

Reach for yes/no by default. A criterion that resists it — “is this helpful?” — is usually two criteria wearing one name. Split it; don’t soften the scale.

The three-point scale is for things like tone, register and formatting, where a real middle exists and you would act on it differently. Anything finer is precision you cannot defend. Nobody has ever explained the difference between a 6 and a 7 out of 10 to a second grader. A scale with invented interior points is worse than no scale, because it produces decimals that look like measurements.

Blocking, and why one number was never enough

Non-blocking criteria average into a score. They describe quality on a gradient, where more is better and a weak case drags the mean down a little.

Blocking criteria do not average. Failing one makes the case unshippable however it scored elsewhere. Fabrication, a misstated policy, a refusal that should have been an answer: a warm tone does not offset these. A reply that invents a sixty-day returns window is not eighty percent correct. It is a liability with a pleasant manner.

An average is the wrong instrument for a failure that cannot be traded against anything else.

That is why a run produces two numbers, and why you read them together.

A worked eval

An online store’s support assistant. Twelve real customer messages from the helpdesk and live chat — a late delivery, a return outside the window, a double charge, a discount code that would not apply. Four criteria: three blocking, one scaled.

The run comes back at 76% overall, with five release blockers.

Read the first number alone and you ship it. Seventy-six is a respectable pass, the sort of figure that survives a steering meeting without a follow-up question. Read the two together and they say something far more useful. The average is fine; the floor is not. The assistant is pleasant, on-brand and broadly helpful. It is also misstating or inventing policy in four cases out of twelve, and answering the wrong question in a fifth.

Broken out by criterion, the grade becomes a work queue:

Answers the question asked · blocking83%

States policy correctly · blocking75%

Invents nothing · blocking83%

Tone fits the brand63%

The worked eval above: 12 support messages, four criteria. Overall 76% is the mean of these four.

The numbers

Criterion	Score	Passing	Failing
Answers the question asked (blocking)	83%	10	2
States policy correctly (blocking)	75%	9	3
Invents nothing (blocking)	83%	10	2
Tone fits the brand	63%	10	2

Tone is the weakest score on the page and the least urgent. The work is states policy correctly: three cases, all blockers, all the same class of error. That is a prompt or retrieval change, a week’s work, not a quarter’s. An aggregate would have told you the feature was at 76% and left you to guess.

How the score is computed

Worth knowing, because every scoring scheme encodes a judgment, this one included.

1. Each verdict normalizes to 0–1. A yes/no criterion is 1 or 0; a three-point scale maps to 0, 0.5, 1.
2. Each criterion scores the mean of its verdicts across every graded case.
3. The overall is the mean of the criterion scores — not the mean of the cases.
4. Separately, a case is marked blocked if any blocking criterion failed on it. The blocker count tallies cases, not failures.

The assumption to notice is in step three. Averaging criteria rather than cases gives every criterion equal weight in the headline, whatever its severity and however many cases it was graded on. A blocking criterion and a tone criterion move the overall by exactly the same amount.

That is deliberate, and it is why the blocker count sits beside the score rather than inside it. Try to weight severity into a single figure and you are back to arguing about weights instead of replies. Two numbers, honestly separated, beat one number with a defensible-looking formula behind it.

Writing criteria that hold

A rubric fails in the writing far more often than in the running. The test is simple: hand it to a colleague with ten replies and no context. If your verdicts diverge, the criterion is the problem. Not the colleague, and not the model.

Criteria that hold share five habits. They name the source of truth — returns window is thirty days, delivery is two to four working days — so “correct” is a fact rather than a memory. They state the pass condition for uncertainty. They test one thing. They describe the reply rather than the intent, because what is on the screen is all the customer gets. And they are written before anyone reads the outputs, since a criterion written afterward describes what you already have. That is where to start, not stop. people refine their criteria once they grade real outputs, so expect a second draft after the first read. The difference: the change is written down and deliberate, not a quiet redefinition to make the run pass.

The ones that collapse are just as easy to spot. “Is the response helpful?” gets two verdicts from two graders every time. “Accurate and complete and well-formatted” is three criteria in a trench coat, and a failure doesn’t tell you which. “Sounds human” has never survived contact with a review. And the worst: the criterion added after a disappointing run, whose real purpose is to explain why the run was fine.

Who checks the grader

If a model grades the outputs, the obvious question is who grades the model. You do, on a sample, and the result is more interesting than it looks.

Grade a handful of cases by hand. Compare your verdicts with the grader’s. Measure the agreement. Then — and this is the part people get backward — treat every disagreement as an indictment of the criterion, not the model.

A criterion precise enough to grade is precise enough that a competent grader, human or otherwise, reaches your verdict. When it doesn’t, you have almost always found a hole in the definition: an edge you never specified, a word doing two jobs, an assumption you hold and never wrote down. Rewriting the criterion fixes the disagreement and improves every run after it. Swapping the grader model fixes nothing and hides the hole.

Agreement is also the number that lets you defend the exercise to someone senior. “The model scores this 76%” invites the question of why anyone should believe it. “The model and I agree on nine cases out of ten, and here are the ones we don’t” does not.

The diff, not the score

The aggregate is the least useful number an eval produces. It is a headline, and like most headlines it hides the story.

What matters is the comparison between two runs: which cases moved, and which way. The typical failure when you iterate on a model-backed feature is not that the number goes down. It is that the number goes up while three cases quietly break.

You tighten the prompt to fix the policy errors. The overall climbs from 76% to 84%. Along the way the assistant stops admitting uncertainty and starts guessing confidently, because the instruction that fixed one thing pushed on another. On paper: better. In fact: worse, in exactly the way that produces an incident three weeks later.

A per-case diff catches that. An aggregate never will. That is also the argument for keeping every run instead of overwriting the last one. The history is the asset, and being able to say this change broke these two cases is the whole reason to build the thing.

When to run it

An eval that runs once is a document. An eval that runs on a trigger is a control. There are five triggers, and each assumes the one before it.

Before the first release. Write the rubric before reading any outputs, gather fifty real cases from actual traffic rather than invented ones, and take the baseline. The number itself doesn’t matter yet. It starts to mean something the second time you run it.

On every prompt or model change. Rerun and read the diff. This is the regression suite. Treat it as optional and a working feature degrades across a quarter without any single change to blame.

When a vendor ships a new version. The model underneath you changes whether or not you asked. A rubric turns “the new one feels better” into a comparison you can put in front of someone who controls the budget.

After every incident. The failure that reached a customer becomes case fifty-one. This is how the suite grows teeth: it collects the specific ways this product has failed, which no generic benchmark contains.

At the quarterly review. Blocker count over time, per-criterion movement, grader agreement. Three numbers that describe whether the thing is getting better, and the only honest answer to “how is the AI feature doing?”

The ways it goes wrong

What it does not do

It does not tell you whether anyone wants the feature. It measures conformance to a standard you wrote, and a standard can be met perfectly by something nobody asked for.

It does not catch what you didn’t think to write down. A rubric is a list of known failure modes. The new one arrives in production, and the honest response is to add it as a criterion, not to claim the eval covered it.

And it does not replace judgment. It makes judgment explicit, repeatable and arguable, which is different and better, but it is still your standard and your call. The point was never to remove the opinion. It was to write it down first, specifically enough that you can notice when it’s wrong.

Sources

Where the evidence comes from. Each row opens the document itself; the tag says which part of the argument it supports, and ↩︎ goes back to the sentence that uses it.

1. Defeating Nondeterminism in LLM InferenceHorace He, Thinking Machines LabLLM nondeterminismWorking paper2025-09↗︎Same prompt 1,000 times, temperature 0: 80 unique completions
2. Judging LLM-as-a-Judge with MT-Bench and Chatbot ArenaLianmin Zheng et al.LLM judge agreementPaper2023-06 (NeurIPS 2023)arXiv:2306.05685↗︎GPT-4 judge matches humans >80%, like humans
3. Q: Why do you recommend binary (pass/fail) evaluations instead of 1-5 ratings (Likert scales)?Hamel Husain and Shreya Shankar (AI Evals FAQ)binary over LikertArticle2025-05↗︎Binary evals force clearer, more consistent labels
4. Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human PreferencesShreya Shankar, J.D. Zamfirescu-Pereira, Björn Hartmann, Aditya Parameswaran, Ian Arawjocriteria driftPaper2024-04 (UIST 2024)doi:10.1145/3654777.3676450↗︎LLM graders need human validation; criteria get refined
5. Interrater reliability: the kappa statisticMary L. McHugh, Biochemia Medica 22(3):276-282inter-rater reliabilityJournal2012doi:10.11613/BM.2012.031↗︎Defines interrater reliability; kappa corrects chance agreement
6. Evaluation best practicesOpenAI API docscontinuous evaluationDocsaccessed 2026-09↗︎Run evals on every change; grow set
7. Define success criteria and build evaluationsAnthropic, Claude Platform Docsreal task distributionDocsaccessed 2026-09↗︎Evals should mirror real-world task distribution. Its own tone example uses a 1–5 scale