LLM Observability Is Mostly a Schema Problem

Every team that shops for LLM observability ends up comparing dashboards. Latency charts, token counters, a trace waterfall, a spend graph. They all look roughly the same in a demo, because the dashboard is the easy half of the problem.

The hard half is what your spans actually contain. At two in the morning, when a customer says the assistant gave them a price that has not been correct since March, the question you need to answer is which model version served that request, what context was retrieved for it, and whether the same thing happened to anyone else. If those three facts are not attributes on a span, no dashboard recovers them.

The standard exists, and it is not finished

OpenTelemetry has semantic conventions for generative AI, now maintained in their own repository covering spans, metrics and events for GenAI clients, for the Model Context Protocol, and for provider-specific behaviour. That is the schema worth adopting, because it is the one your vendors are converging on.

It is also, right now, mostly provisional. On an inference span, two attributes are required: gen_ai.operation.name and gen_ai.provider.name. Both carry development status. So do gen_ai.request.model, gen_ai.response.model, and the token counters. Scan the same span definition for attributes marked stable and you find error.type and server.port — the two fields with nothing to do with generative AI at all.

Read that as a maturity signal rather than a reason to wait. The generic parts of tracing settled years ago; the AI-specific parts are still moving. Adopting the convention now costs you a rename or two later and buys you instrumentation that other tools can read. Inventing your own attribute names costs you a migration you will keep postponing.

The field you need most is optional

Look closely at how the requirements are graded, because the gradient is not where you would guess.

Token counts — gen_ai.usage.input_tokens and gen_ai.usage.output_tokens — are recommended, not required. So is gen_ai.response.model, the attribute that records which model actually served the request as opposed to which one you asked for. gen_ai.request.model is conditionally required, meaning it is expected when available.

Those gradings make sense for a spec that has to accommodate every provider. They are wrong for you. The response model is how you answer whether a regression tracks a silent provider-side version change, and the token counters are how you attribute cost to a feature rather than to a monthly invoice. Both should be non-negotiable in your own instrumentation, whatever the spec permits. A default auto-instrumentation that omits them will look fine in a demo and fail on the first real incident.

Cardinality is a decision you make once

The spec names inference and embedding spans with a template: the operation name followed by the request model. Retrieval spans use the operation name plus gen_ai.data_source.id. And the fetch-response span deliberately leaves the response identifier out of its name, because the identifier is high cardinality.

That last choice is the whole lesson in miniature. Span names are grouping keys, and a grouping key with a million distinct values groups nothing while costing a great deal to index. The same discipline has to survive contact with your own code, where the temptation is worse: prompt text, user identifiers, session identifiers, and full retrieved documents all feel like things you want on the trace. They are — as attributes or events on the span, which are stored and searchable, not as part of the name.

Get this wrong early and the bill teaches you before the tooling does.

Retrieval is its own operation

The convention defines operation names for chat, embeddings, content generation, retrieval, memory search and tool execution. Retrieval is a first-class span with its own required data source identifier.

This matters more than it sounds. When a retrieval-augmented system returns a wrong answer, the completion span shows a fluent, confident, wrong response, and it looks exactly like a model failure. The actual defect is usually one layer up: the retriever returned three passages, none of which contained the answer, and the model did the best it could with what it was handed. Without a retrieval span recording what came back and from which source, you will spend the incident tuning a prompt that was never the problem.

Instrument the retriever before you instrument anything clever.

Watching production is a different instrument

The last piece is the one teams bolt on last and should design first: grading a sample of live traffic rather than a fixed test set. It is the only thing that catches failures nobody wrote a case for, and it is a genuinely different measurement problem from the offline suite — one of the three jobs that get confused with each other in LLM evaluation.

The reason it is different is worth being precise about. Offline, when a model judges which of two answers is better, you can defend against its position bias by running both orders and keeping only the verdicts that hold — a real defence, because the MT-Bench work found GPT-4 reaches the same verdict on swapped answers just 65.0% of the time, with GPT-3.5 at 46.2% and Claude-v1 at 23.8%. In production there is no second answer. There is one response, and no counterfactual to compare it against.

So the graders change shape. Pairwise comparison is out. What works on a single live output is a binary or rubric grader answering a narrow question — did the response cite a retrieved passage, did it refuse when it should have, did it stay inside the requested format. The kind of criteria the OpenAI evals model builds an eval from: a data source configuration paired with testing criteria, of which a plain string check is the simplest instance. And the number you watch is drift over time rather than an absolute score, because the absolute score of a model grader carries an offset you cannot fully remove.

Where to start

Instrument one path end to end before you instrument everything shallowly. Emit spans for the retrieval and the completion, pin the response model and both token counters as required in your own code even though the spec calls them recommended, keep identifiers out of span names and on attributes, and only then connect a dashboard. Sample one or two percent of live traffic into a binary grader and chart the pass rate weekly.

The dashboards really are interchangeable. The schema underneath them is what you are choosing.