LLM Testing Already Had a Playbook in 2020

Testing a language model feels like a problem without precedent. The output is text, the same input can produce different text twice in a row, and the obvious tools all assume a function that returns the same thing every time. So teams invent something ad hoc, usually a folder of prompts with expected answers, and it works until it does not.

The methodology that fits best was published in 2020, before any of the current models existed. It was aimed at NLP models generally, and almost everything in it transfers.

Three kinds of test, and you are writing one

CheckList proposes three test types, borrowed from software testing and adapted to models whose behaviour you cannot read off the source.

A minimum functionality test asks whether the system does the thing at all. Give it a clear-cut case and check the output. This is the folder of prompts with expected answers, and it is what nearly every team builds first.

An invariance test perturbs the input in a way that should not change the output, then checks that it did not. Rephrase the question, swap a name, reorder two list items, change the whitespace. The assertion is about stability, not correctness.

A directional expectation test perturbs the input in a way that should move the output in a known direction, and checks that it moved that way. Add a clearly negative clause to a review and the sentiment should not become more positive. You are not asserting an exact result, only the sign of the change.

Sit with a typical eval suite and it is minimum functionality tests all the way down. The other two are missing, and they are where the interesting failures live.

Invariance is the type non-determinism makes essential

The standard complaint about testing models is that you cannot assert equality against a sampled output, so the whole exercise feels unstable. That complaint quietly assumes the only available assertion is about the content of a single response.

Invariance tests invert the problem. They take the model's sensitivity to irrelevant changes as the subject of the test rather than as an obstacle to it. A production prompt that gives a different classification when the customer's name changes from one to another has a defect, and it is a defect no amount of correct answers on your happy-path set will surface. The same goes for a retrieval system whose answer quality depends on which order the passages arrived in.

These are also the tests that catch the failures customers actually report, because customers do not send you your happy-path phrasing. They send you the same question in their own words, and the system either holds or it does not.

Directional tests earn their place in a narrower band, but that band is important: anywhere the product makes a claim about how it responds to input. If your assistant is supposed to get more cautious as a request gets riskier, that is a directional expectation, and it is testable without anyone writing an ideal response.

Structure beats inspiration

The reason to adopt someone else's taxonomy rather than brainstorming freely is that unstructured test ideation misses whole categories, and it does so invisibly.

The CheckList study measured this. Practitioners given the structure wrote roughly twice as many tests and found close to three times as many bugs as those without it. Separately, a team applying it to a commercial sentiment analysis system that had already been tested extensively found new and actionable defects in it. Not obscure ones — the point of the finding was that a well-tested production model still had whole capability areas nobody had thought to probe.

The mechanism is simple. When you sit down to write tests from imagination, you write tests for the failures you already have in mind, which are the failures you have already fixed. A matrix of capabilities crossed with test types forces you into the cells you would never have visited.

A failing test is a statistic

Here is the part that has no 2020 precedent, because it comes from sampling.

A test that runs a case once and asserts on the result is a coin flip you have written into your suite. It will pass for weeks and then fail on a Tuesday for no reason anyone can reconstruct, and the team's response will be to rerun it until it goes green, which is the moment the suite stops meaning anything.

Run each case several times and assert on the pass rate against a threshold. Store the rate, not the verdict. A case that passes nine times out of ten is a different thing from one that passes ten out of ten, and the difference is information you want before a release rather than after. It also gives you something to watch: a case whose rate drifts from ten out of ten to eight out of ten over a month has told you about a regression that a pass or fail assertion would have hidden until it crossed the line.

This costs more calls, which is the real reason it is skipped. Weigh that against Anthropic's guidance to prioritise volume over quality — many cases graded automatically beat a few graded painstakingly by hand — and the arithmetic usually favours more runs of cheaper checks over fewer runs of expensive ones.

What the graders look like

Once you know which tests to write, grading is a solved-enough problem. The OpenAI evals model builds an eval from a data source configuration paired with testing criteria, the simplest of which is a plain string check. Anthropic's guidance lists exact match, cosine similarity over sentence embeddings, ROUGE-L, and model-based grading on Likert, binary or ordinal scales.

Reach for them in that order. Where you do end up with a model grader, use a different model than the one that produced the output — Anthropic states this outright as best practice, and the MT-Bench work supplies the reason with a number: a model judge gave its own output roughly ten percentage points more win rate than human raters did for GPT-4, and about twenty-five for Claude-v1. Grading yourself is not a neutral act. The rest of the judge's failure modes, and what they mean for the scores you report, are covered alongside the metrics worth choosing between.

A first suite

Take the ten cases your support queue complains about most. Write each as a minimum functionality test. Then, for each one, write the invariance test that goes with it — same case, rephrased by someone who was not in the room. Run every case five times, assert on pass rate, and commit the rates.

That suite will be small enough to run on every pull request and honest enough that a green result means something. Both properties are rarer than they should be.