Pick LLM Evaluation Metrics by What You Can Supply

Most write-ups of LLM evaluation metrics sort them by mechanism: n-gram overlap here, embedding similarity there, model-as-judge at the end. It is a tidy taxonomy and it is close to useless when you are trying to pick one, because mechanism is not the thing standing between you and a working eval.

What stands between you and a working eval is data. Every metric demands something from you before it will produce a number, and the demands differ enormously. Sort by that instead and the choice mostly makes itself.

Metrics that need a known-correct answer

The cheapest tier, and the one teams skip because it feels unsophisticated. Exact match asks whether the output equals the expected answer. String presence asks whether a required substring appears. Both are deterministic, cost nothing to run, and never disagree with themselves.

Anthropic's guidance on developing test cases puts exact match at the top of its grading options for a reason, and the Ragas metric catalogue ships both exact match and string presence alongside its more elaborate offerings. What they need from you is a right answer for every case — which sounds like a lot until you notice how much of a real system is categorical underneath the prose. Routing decisions, extracted fields, classifications, refusals, whether a tool was called and which one: all of these have correct answers you can write down.

The failure mode here is not the metric. It is designing the system so nothing is checkable. If your pipeline emits a paragraph where it could emit a paragraph plus a structured field, you have converted a tier-one measurement into a tier-three one for no benefit.

Metrics that need a reference text

Now you need a written example of a good output for each case, and you are comparing against it.

The n-gram family lives here — the Ragas catalogue includes BLEU, ROUGE and CHRF, and Anthropic's guidance names ROUGE-L specifically for summarisation. They count overlap, which makes them fast, language-agnostic and blind to paraphrase. Say the same thing in different words and the score falls.

BERTScore was built to fix exactly that. Instead of matching tokens exactly, it computes similarity between each token in the candidate and each token in the reference using contextual embeddings, so a paraphrase that means the same thing scores like one. The paper validates this across 363 machine translation and image captioning systems, reporting better correlation with human judgment and stronger model selection than the metrics it replaces. Cosine similarity over sentence embeddings, which Anthropic's guidance also lists, is the same idea in a simpler package.

The cost of this tier is the reference itself. One good example per test case, written by someone who knows what good looks like, kept current as the product changes. That is real work, and it is why the tier is often skipped straight past.

Metrics that need no reference at all

This is the tier that changed what is practical, and it is worth understanding why it works rather than just which library provides it.

Ragas frames itself as reference-free evaluation of retrieval-augmented pipelines, assessing three things: whether retrieval surfaced relevant and focused passages, whether the model used those passages faithfully, and the quality of the generated output. The trick is that a RAG system hands you a reference for free. The retrieved context is right there. You do not need a human-written ideal answer to ask whether the response is supported by the passages the retriever returned.

That is what the catalogue's core metrics measure. Faithfulness checks whether the answer is grounded in the supplied context. Context precision asks whether the retrieved passages were relevant to the question, context recall whether they contained what was needed, and noise sensitivity how much irrelevant retrieved material degrades the answer. Response relevancy asks whether the answer addressed the question at all. None of them require you to have written anything in advance.

For a retrieval system this is close to a free lunch, and it is the reason a team with no golden set can still have a functioning eval by Friday.

The bill for tier three

There is a catch, and the catalogue names it for you. Among a long list of metrics sits one called non-LLM string similarity. The name is doing quiet work: it exists to distinguish itself from everything around it. The reference-free metrics are computed by a model.

So the judge is back, and with it the measurement offset that comes attached — the reason a model grader's absolute score is worth less than its trend, covered in more detail in the three jobs that LLM evaluation gets asked to do. Reference-free does not mean assumption-free. It means you swapped the cost of writing references for the cost of trusting a grader, and the second cost is less visible, which is exactly what makes it easy to underestimate.

The practical consequence: use the reference-free metrics for breadth and trend, and keep a smaller tier-one or tier-two set as the thing you actually gate a release on.

Reporting one number is a choice not to look

Whichever tier you land in, there is a separate failure that has nothing to do with which metric you picked.

HELM measures seven metric categories rather than one — accuracy, calibration, robustness, fairness, bias, toxicity and efficiency — across 42 scenarios and 30 models, with all seven reported on each core scenario 87.5% of the time. Building that took real effort precisely because the field's habit was to report accuracy and stop.

The habit survives inside individual teams. A dashboard showing one quality score is not a summary of six other things; it is silence about them. Calibration is the one that bites first in production, because it decides whether a confidence value is safe to route on. If a downstream branch trusts the model's certainty, calibration is your load-bearing metric and accuracy is context.

A starting set

Three metrics, chosen to sit in different tiers. One exact-match check on whatever is categorical in your output, run on every change and gating the merge. One faithfulness or groundedness check across a broad sample, watched as a trend rather than a threshold. One dimension beyond accuracy that matters for your product — calibration if you route on confidence, refusal behaviour if you have a safety boundary, format validity if something downstream parses the output.

Add the fourth when one of the three has caught something. Not before, because a metric nobody has ever seen fail is a metric nobody knows how to read.