← All articles

How to harness Evals

Evals · · 3 min read

This is the one that separates people who ship AI features from people who demo them.

What it is, in practice. A place where your test set, your rubric, and your scores live, and a button — or a command — that runs the whole set against the current version and tells you the number moved. Everything else is packaging.

The three options:

  • A dedicated platform — Braintrust, Langfuse, LangSmith, Promptfoo. They store test cases, run them against your prompt or app, grade outputs, and show you version-over-version comparisons. Some also log production traffic so real failures become new test cases.

  • A spreadsheet plus a script. Inputs in column A, expected outputs or rubric in B, model output in C, score in D. Your agentic coding tool writes the script in twenty minutes. This is where most teams should start.

  • Nothing, and vibes. What most teams actually have. It works until the first confident, wrong answer reaches a customer.

When to graduate from spreadsheet to platform: more than one person editing the test set, more than one prompt or model in play, or production traffic you want to sample from. Before that, the platform is overhead.

The test for choosing one:

  1. Can you add a test case from a real failure in under a minute? The harness only stays useful if the bar to add to it is near zero. A support ticket about a wrong answer should become a test case that afternoon.

  2. Can a non-engineer write the rubric? Defining “good” is your job. If the tool needs code to express “must not invent a source,” you’ll stop maintaining it.

  3. Does it show the diff? Not just “score went from 82 to 79” but which cases changed and how. That’s where you learn.

  4. Can it grade with a model and let you audit the grader? Model-as-judge scales; blind trust in it doesn’t. You need to spot-check that the grader agrees with you.

  5. Does it connect to production? Eventually you want real inputs flowing into the test set. Not day one.

How to actually build your first one, in a week:

  • Day 1: collect 50 real inputs. Real, not invented — the ugly customer questions, the transcripts with crosstalk.

  • Day 2: write the rubric. Three to five criteria, each yes/no or 1–3. “Cites a real quote.” “Doesn’t fabricate a feature.” “Under 150 words.”

  • Day 3: score the current version by hand. This is your baseline and it will be humbling.

  • Day 4: script it so a model grades the same way; check its grades against yours on 10 cases.

  • Day 5: change something — prompt, model — rerun, compare. Now you have an eval.

Two cautions:

  • The test set is the asset, not the tool. Switching platforms is easy; rebuilding a good set of 200 real cases is not. Keep it in a format you own.

  • Evals measure what you thought to test. The failures that hurt are the ones outside the set. Sample production output by hand, regularly, and put what you find back in.

For your series: every one of the six tools needs one of these. The synthesizer’s eval is your old hand-syntheses; the bet scorer’s is bets you made whose outcome you now know. Building the harness once and reusing it across all six is a real product decision, and a good Part 7.