LLM Evaluation Is Three Different Jobs

Ask three engineers on the same team what they mean by LLM evaluation and you will get three answers, all correct. One is choosing between models. One is deciding whether the change about to merge broke something. One is watching whether the system is behaving right now, on real traffic. These are different measurement problems with different failure modes, and the trouble starts when the artifact from one gets used to answer another.

The most common version: a team picks a model on leaderboard scores, ships, and then has no way to tell whether last week's prompt edit made things worse. The leaderboard answered question one. Nobody built anything for question two.

A leaderboard answers a question you are not asking

Stanford's Holistic Evaluation of Language Models is the most careful attempt at question one. It runs 42 scenarios — 16 core and 26 targeted — across 30 models, measuring seven metric categories rather than one: accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency. All seven land on each core scenario 87.5% of the time.

The number worth sitting with is not any model's score. Before HELM, models had been evaluated on an average of just 17.9% of its core scenarios. The benchmark pushed that to 96.0%. For years the field was comparing models that had largely never been measured on the same things, under the same conditions.

That is a real contribution to question one. It is also the ceiling of what a public benchmark can do for you, because none of those 42 scenarios is your scenario. A model that leads on aggregate accuracy may sit mid-table on calibration, which is the metric that matters if a downstream system routes on the model's confidence. Multi-metric evaluation does not remove the need to know which metric is load-bearing for your product — it just makes it possible to look.

The judge is less consistent than it looks

Once a team accepts that public benchmarks will not answer question two, the usual next move is a model-graded eval: put a strong model in front of the outputs and have it score them. The reference study here is the MT-Bench and Chatbot Arena work, built on 3,000 expert votes and 30,000 conversations with human preferences.

Its headline result is genuinely encouraging. On MT-Bench, GPT-4 agreed with human experts 85% of the time excluding ties, against 81% agreement between the humans themselves. A model judge can match a human panel.

Now the number that gets quoted less. Swap the order of the two answers being compared, and GPT-4 reaches the same verdict only 65.0% of the time. GPT-3.5 manages 46.2%, Claude-v1 23.8%. The judge agrees with humans more reliably than it agrees with itself. Some fraction of that 85% is the judge responding to position rather than to quality — and 85% versus 65% is not a rounding error, it is twenty points of measurement that came from somewhere other than the thing you meant to measure.

The other two biases are sharper still. Padding an answer with a repetitive list that adds no information fooled GPT-4 8.7% of the time; GPT-3.5 and Claude-v1 fell for it at 91.3%. On self-enhancement, GPT-4 gave its own output roughly ten points more win rate than human raters did, and Claude-v1 about twenty-five.

None of this means model-graded evaluation is unusable. It means the naive version is a measurement instrument with a known, quantified offset. The mitigations are ordinary engineering: run every pair in both orders and count only the cases where the verdict holds, strip model identity before grading, penalise length explicitly in the rubric, and never let a model grade its own output — Anthropic's own guidance says to use a different model for grading than the one that generated the text. Each of these costs calls, which is the actual reason teams skip them.

Volume beats polish, and both beat neither

The most useful piece of published guidance on question two is also the most counterintuitive. Anthropic's advice is to prioritise volume over quality: many test cases with slightly noisier automated grading beat a small set graded carefully by hand. That runs against the instinct to build a small, pristine golden set, and it is right for a simple reason — an eval you can run on every pull request changes behaviour, and an eval that needs a human afternoon does not.

The practical ordering falls out of that. Reach for exact match wherever the task has a categorical answer. Use similarity metrics — embeddings, or ROUGE-L for summarisation — where the output is text but a reference exists. Save the model judge for the dimensions that resist both, like tone or whether a response actually used the context it was given. Most teams invert this, reaching for the judge first because it is the one that works on anything, and inheriting its biases across the whole suite. Which tier you can actually reach is decided by what reference data you are able to supply, not by which metric sounds most sophisticated.

Question three, production monitoring, is a different instrument again: sampled, continuous, and measured against drift rather than against a fixed target. It is the one most often skipped, and the one that catches the failure modes nobody wrote a test case for. Getting from a working demo to a system with all three of these in place is most of what separates a proof of concept from production.

The tooling churns; the method does not

One note on where to put the work. OpenAI's Evals platform goes read-only on 31 October 2026 and shuts down on 30 November 2026, with new users pointed at Datasets instead. That is a normal product lifecycle, and it is also a useful test of how you have set things up.

If your eval set is a versioned artifact in your own repository — inputs, expected outputs, rubrics, thresholds, all under review like any other code — a deprecation like that is a migration you schedule. If it lives inside a vendor's interface, it is a rebuild, and the historical scores that told you whether things were getting better do not come with you.

Where to start

Write thirty cases from real traffic before you write any harness. Grade them the cheapest way that works — exact match first, similarity second, judge last and only where the first two cannot reach. Pair each one with the invariance test that goes with it, because a case that only ever sees your own phrasing is testing your phrasing. Run the set on every change, keep the scores in version control next to the code, and only then argue about which model to use — or about whether to fine-tune or retrieve, a question the eval set is the only honest way to settle. Question one is the easiest of the three and the least valuable to answer early, because the answer changes every few months and the eval set is what tells you when it has.