Evaluation: how you know it works

Evaluation is not one number. It is three layers that fail independently, and if you collapse them you will ship a fluent liar. Layer one is retrieval: did we fetch the right passages. Layer two is the answer: given those passages, is the sentence grounded, cited, in schema, and inside policy. Layer three is the product: did the human finish the job. Executives only feel layer three. Layer three lies if you skip the first two.

Retrieval metrics belong to whoever built the index, judged by questions the business wrote. Recall at k is the one to start with: of the documents a reviewer says are necessary, how often is at least one in the top k. MRR (mean reciprocal rank) tells you whether the right thing is near the top. You do not need a research paper. You need a spreadsheet of questions, the page that should be retrieved, and a pass/fail at k equals five.

Answer metrics belong to you and a reviewer. Groundedness: every claim is supported by a retrieved span, or the system refused. Citation correctness: the cited page actually contains the claim. Schema validity: if you asked for JSON or a four-line status, you got that shape. Policy: no toxic output, no out-of-scope advice, no secrets echoed. Fluency is not a metric you pay for. The model is already fluent. Fluency without groundedness is the failure mode.

Product and task metrics belong to the business. Did the PM find the decision in under two minutes. Did the extractor fill the right fields. Did the draft reduce the time to send a status email without increasing rework. Time-on-task, task success, and rework rate are numbers you already understand from process work. Put them on the same page as recall and groundedness so nobody can wave a demo and call it adoption.

Owners are not a RACI decoration. If retrieval is red and the application engineer says 'the answers look fine,' you have a lucky fluent model and a rotten index. That luck expires when the corpus shifts. If task success is red and the model team says 'groundedness is 0.92,' you have a system that is correct and unused, or correct and too slow, or correct and answering the wrong job. Separate the layers so the fix has a name.

Offline, human, and online are how you instrument the three layers over time, not a different taxonomy. Offline is the golden set on every change — your regression suite. Human is a sampled review against a rubric — your UAT. Online is live thumbs, traces, cost, latency, and task telemetry — your production monitoring. All three run. Dropping one is how you either freeze in the lab or fly blind in production.

Do not let a demo standing in for layer three. A steering committee that watches a happy-path question get a pretty answer has seen theatre, not task success. Task success is observed on the job, with a timer, on tasks the business named. If you cannot get thirty minutes with two PMs to watch them work, you do not have a product eval. You have a generator that can complete sentences. Write that gap on the RAID log as a dependency: labeling time and observed UAT, named humans, a date.

Split ownership in the charter so a green retrieval score cannot hide a red task. The index owner reports recall. You and a reviewer report groundedness and refuse. The business owner reports task success. Three lines on a weekly pulse. When one is red, that owner brings the fix. Combined scores with one owner are how a workstream debates a blended number nobody can act on. You already split RAID owners this way. Do it for eval or the layers collapse in the first steering pack.

Thirty to two hundred labeled examples is a start. Each row is an input, context if any, expected behavior (an answer key, a refuse, a JSON shape), and tags: happy, adversarial, empty, injection, ACL, conflict, stale. Version it. When the product changes, the set changes with a changelog, like a test suite. A golden set that nobody owns is a folder of leftover demo questions.

Include cases that must fail closed. No sources. Conflicting sources. Out of scope. User asking the bot to ignore policy. A document that contains an injection. A question whose real answer is 'I do not know, here is how to find the owner.' A golden set of only demo questions is a vanity metric. It will go green while production users hit the cases you never wrote.

The business writes the questions. You facilitate. If engineers write all the items, you will over-fit the index they already tuned. Sit with three PMs for forty minutes and collect the questions they actually ask, including the rude ones and the ones that should be refused. Then sit with the corpus owner and mark the page that should be retrieved, or mark 'refuse.' That labeling session is the most valuable meeting on the workstream.

Versioning is not ceremony. Golden set v3 is what you ran to ship. If someone edits a row in place without a version bump, you cannot explain a score change. Treat it like a test fixture in a repo: pull request, review, tag. The changelog says what was added (three injection rows after the incident) and what was retired (the old stipend amount after SOP-Travel-2024 replaced it).

Coverage beats volume. Forty rows that cover happy path, refuse, empty retrieve, conflict, injection, ACL miss, and sycophancy will teach you more than four hundred paraphrases of the same FAQ. After you have coverage, then grow. Growing without coverage is how you get a set that is expensive to run and still blind to the failure that will land on a steering slide.

Expected behavior is specific. 'A good answer about travel' is not a label. 'Cite SOP-Travel-2024 section 4 with the APAC amount, or refuse if that page is missing' is a label. If two reviewers cannot agree on the expected, the product spec is not done. Eval is how you find that. It is cheaper to find it in a spreadsheet than in a customer email.

Who labels is a named role, not a rotating volunteer. The corpus owner or a trained reviewer marks the expected source and the refuse cases. Engineers may propose rows. They do not solely decide expected behavior, because they will overfit what the current index can do. Put labeling hours in the plan the same way you put UAT hours. If nobody has time to label, you do not have a set. You have a wish list of questions.

Adversarial rows are not optional spice. They are how you learn whether the system fails closed. Empty retrieve, conflict, injection, ACL miss, sycophancy, out of scope. If those six tags are missing, add them before you grow happy-path paraphrases. A set that is ninety percent restated FAQs will go green and still fail the first angry user. Coverage of failure modes is the adult version of 'we tested it.'

Groundedness is the claim-level question: is this sentence supported by the retrieved evidence, or did the model speak from weights. You score it on a sample with a rubric, not with a feeling. A useful rubric is three buckets: supported, unsupported, or refused. Refused on a must-refuse item is a pass. Fluent and unsupported is a fail even if the user liked it. This is the metric that protects you from confident wrongness.

Recall at k is the retrieval question. For each golden item that has a required source, is that source in the top k chunks. Start with k equals five because that is what you probably stuff in the prompt. If recall is 0.6, you do not have an answer problem yet. You have an ingest, chunking, or query problem. Tuning the generator on a 0.6 retrieve is rearranging the furniture while the house is on fire.

Task success is the only number a sponsor will remember, and it has to be defined in their words before you start. 'PM can find the last decision on a workstream in under two minutes, with a citation, on eight of ten sampled tasks' is a definition. 'People like the copilot' is not. Instrument it with a small observed sample, not with a survey of vibes. You already know how to UAT a process. Sit next to the user. Time the job. Record pass/fail.

Secondary metrics earn their place or they leave. Citation correctness is worth keeping because a grounded sentence with a wrong link trains users to stop clicking. Schema validity is worth keeping if a downstream system consumes the output. Toxicity and policy are worth keeping as a tripwire. BLEU, ROUGE, and 'similarity to a reference paragraph' are usually not worth keeping for a copilot. They punish paraphrase and reward copy-paste.

Do not average unlike things into a single 'quality score' you report to steering. A dashboard tile that says 0.81 hides a recall of 0.5 and a groundedness of 0.95, or the reverse. Show the three numbers. The conversation you want is 'retrieval is the constraint' or 'the generator is inventing' or 'the job still takes twelve minutes.' A blended index is how a workstream debates a ghost.

Cost and latency sit next to quality, not underneath it. A grounded answer that arrives in fourteen seconds at two dollars a question will not be used, and unused is a task-success fail. You will write SLOs for those on day 18. Today, put a placeholder on the eval sheet: p95 and dollars per successful task, measured on the same golden run. Quality that ignores cost is a lab result.

Citation correctness is worth its own column when users must click through. A grounded sentence with a dead or wrong link trains people to stop looking at sources, and then you have lost the only cheap check you had. Score 'cited page contains the claim' on the human sample. If that number is red while groundedness is green, the generator is waving at the right neighborhood and hoping. Fix the citation instruction and the retrieve rank before you celebrate groundedness.

Do not report a single quality index to steering. Three numbers: recall, groundedness, task success. Add refuse-pass as a binary on the must-refuse tags. A blended 0.81 is a fog. You will be asked to explain a drop. You cannot explain a fog. You can explain 'retrieve missed SOP-Travel-2024 on four of forty.' That sentence assigns work. The fog assigns a debate.

You can ask a model to score groundedness. It is fast and cheap and sometimes wrong in correlated ways. It loves fluent answers. It under-flags unsupported claims that sound like the source. It over-flags terse answers that are correct. Use it to triage a large set, then have a human sample. The judge is a junior reviewer. You would not let a junior reviewer ship a release unsampled. Do not do it here.

Calibrate. Take forty rows scored by both the judge and a human against the same rubric. Look at disagreement. If the judge and the human disagree on twelve of forty, you do not have a metric. You have a coin with a bias. Tighten the rubric, change the judge prompt, or drop the judge for that dimension and pay for humans. Calibration is a recurring task, not a one-time notebook. When you change the generator, re-check the judge. They drift together.

Never let a judge model grade a sibling model on 'quality' with no rubric. The rubric is the spec: evidence, schema, refusal. Adjectives — good, helpful, high quality — are how you ship sycophancy. A judge asked 'is this helpful' will reward a pleasing paragraph that invents a stipend. A judge asked 'list each claim and mark supported/unsupported/refused against the retrieved spans' has a chance.

Conflicts of family are real. A judge from the same vendor, same generation, same style, will prefer outputs that sound like itself. If your generator is model A, consider a judge from a different family for the dimensions you care about, or do not use a model judge at all on those dimensions. This is not ideology. It is correlated error. You already distrust a tester who only uses the happy path. Same instinct.

Cost of judging can exceed cost of generating if you are sloppy. A long chain-of-thought judge on every production trace is how the bill becomes the news. Use the judge offline on the golden set and on a small online sample, not on every live request, unless you have a reason and a cap. Humans on a twenty-row sample a week will beat an uncalibrated judge on ten thousand rows that nobody looks at.

Write the limits in the eval plan so a future you does not 'just add a judge' under pressure. What it may score (groundedness against retrieved spans, schema). What it may not score (overall quality, brand voice, 'would a user like this'). How often you calibrate. Who reads the disagreements. That paragraph prevents a year of fake green.

Calibrate on a schedule, not when you feel nervous. Forty rows, human and judge, once before go-live and then monthly, or after any generator or judge-prompt change. Log disagreement rate. If it climbs, stop using the judge for the gate until you fix the rubric. A judge you do not calibrate is a second model you are treating as ground truth. You would not treat an untrained junior as ground truth. Do not start here.

Keep humans on the dimensions that money or safety touch. A judge can triage schema and maybe groundedness against spans. A judge should not be the only scorer on 'would we send this to a customer' or on policy refusals you care about in an audit. Pay for the sample. Twenty rows a week is cheap compared with a SEV. Write the split in the eval plan so an engineer under pressure does not silently drop the human sample to save time.

Eval is a loop, not a phase. You change a prompt, a chunking rule, a model pin, or the corpus. You run the golden set. You look at which tags moved. You decide to ship, revert, or waive. Then production traces feed new rows into the set. If this loop is not on the calendar, you will get drive-by prompt edits and a mysterious regression two weeks later.

Run offline on every change that could move the numbers. Prompt, retrieve, model version, system message, tool schema. If you cannot automate it yet, run it by hand from a script and paste the summary into the change ticket. 'I tried it a few times and it seemed better' is not a run. A run has a set version, a prompt version, a model pin, and a table of scores by tag.

Human sample on a cadence, not only when someone complains. Weekly twenty rows is enough for an internal copilot of eighty users. Pull from production traces, stratified: some thumbs-down, some thumbs-up, some refusals, some writes if you have them. Score against the rubric. This is UAT that does not end at go-live. You already know production support needs a pulse. This is that pulse.

Online metrics close the loop. Thumbs are noisy and still useful as a tripwire. Trace anomalies (timeouts, empty retrieve, schema fail) are tickets. Task success, if you instrumented it, is the sponsor number. A drop in any of these opens the loop: pull traces, add rows to the golden set, fix, re-run. The golden set that does not absorb production failures becomes a museum of the demo.

Who decides is the same question as who waives. A named owner looks at the table and says ship, revert, or waive. That owner is you or the business owner, not 'the model will be fine.' Put the decision in the change ticket. This is a go/no-go, the same as any other release. The novelty is the metrics, not the governance.

Do not let the loop become a research project. If recall is the constraint, the next change is ingest or chunking, not a new model. If groundedness is the constraint, the next change is the generator contract or the retrieve quality, not a temperature tweak. If task success is the constraint and the first two are green, the next change is the UI, the latency, or the job definition. The loop should name the layer, then the change.

Change one variable when you can. Prompt or retrieve or model pin, not all three on a Thursday because someone was inspired. Isolation is how you learn. Combined changes are how you ship a coincidence and cannot roll back cleanly. Put the rule on the eval loop diagram: one variable, one run, one decision. You already preach this on defect isolation. The novelty is that prompt edits feel too small to count as a change. They count.

Production misses feed the set the same week, or the set becomes a museum of the demo. A thumbs-down without a new row is a complaint you will hear again. Assign a weekly thirty minutes: pick five live fails, write expected behavior, tag them, bump the set version. That meeting is the cheapest quality program you will run. Skip it and you will tune against a past that no longer matches the users.

Write the bar before you like the demo. Example for an internal wiki copilot: recall at 5 of at least 0.8 on the golden 40; groundedness of at least 0.9 on a human sample of 20; must-refuse items all pass; p95 complete under 4 seconds; error rate under 2 percent. Your numbers will differ. The shape should not: retrieval, answer, refuse, latency, errors. If a dimension is not on the bar, it will be the one that fails in week two.

The bar is a gate on the release checklist, next to regression tests, not a slide you show if the numbers are good. Prompt change, model change, retrieve change — each re-runs the set and each can fail the gate. Engineers who are used to 'just shipping the prompt' will find this irritating. That irritation is the point. You would not let an untested payment flow ship because the engineer was in a flow state.

Waivers exist because sometimes you must ship with a known miss. A waiver has a name, a reason, an expiry, and a rollback. 'Groundedness 0.84 against a bar of 0.9, waived by Priya until Friday, rollback is pin 1.4, we ship because the alternative is a dead FAQ over a holiday weekend.' That is a waiver. 'Let's see how it goes' is not a waiver. You already know how to write these for other systems. Copy the template. Add prompt version and golden-set version.

Who can waive is named, and it is a short list. You, the business owner, maybe the engineering lead. Not the intern who edited the prompt. Not the vendor. Not a Slack emoji. Publish the list on the eval plan. When someone asks you to 'just this once' from a hallway, you have a page to point at. Hallway waivers are how error budgets die.

A bar that is never missed is probably too low, or you are not measuring the hard tags. Look at injection, conflict, and empty retrieve separately. Averages hide them. If those tags are red and the average is green, you do not ship without a waiver that names those tags. This is the same instinct as not letting a 99 percent unit-test pass rate hide the one failing payment test.

Revisit the bar when the product changes intensity. Shadow mode can live on a looser quality bar because humans are not depending on it. Assist mode needs the bar you wrote. Confirm-on-write needs a higher bar on the write path, plus HITL. Do not reuse the v0 shadow numbers for a customer-facing send. The sequence on day 17 and day 20 assumes the bar travels with the intensity.

Who can waive is a short published list. You, the business owner, maybe the engineering lead. Not the vendor. Not a Slack reaction. The waiver sentence is metric, actual, bar, who, until when, rollback pin, why we ship anyway. If you cannot say it, you are hoping. Hallway waivers burn the error budget and teach the team that the bar is decorative. Put the template in the eval plan so the legal kind of waiver is easier than the hallway kind.

A bar that is never missed is probably blind or too low. Look at injection, conflict, and empty retrieve as their own rows, not averaged into the happy path. If those tags are red and the mean is green, you do not ship without a waiver that names the tags. This is the same instinct as not letting a 99 percent unit-test pass rate hide the one failing payment test. Averages are how you fail the case that will be in the ticket.