LLM Observability: Structured Traces, Spans, and Cost Attribution

You can't operate what you can't see. LLM observability borrows from distributed tracing but adds token-level accounting.

The span model

Every user request generates a trace. Inside the trace, spans capture:

• retrieve — query, top-k, latency, recall against ground truth if available
• rerank — candidates in, ordered out, score distribution
• llm.call — model, input tokens, output tokens, cache hit ratio, latency to first token, latency per token
• tool.call — name, arguments, result size, retry count
• validate — schema pass/fail, repair attempts

Each span carries the tenant ID, user journey ID, and a cost value in fractional cents. Sum by any dimension.

Cost attribution that changes behavior

A "$40k monthly OpenAI bill" is not actionable. "$0.18 average cost per resolved support ticket, driven 60% by a 2,000-token system prompt replayed on every turn" is. The moment engineering sees cost per journey, prompts get shorter and caches get turned on.

Quality signals inside traces

• Groundedness score per generation (was the answer supported by retrieved context?)
• Judge score sampled on 1% of traffic
• Repair-loop count as a leading indicator of drift
• Cache hit ratio trend — a sudden drop usually means a prompt change broke the prefix

Alerting

Alert on rates and ratios, not absolutes: cost per journey up 20% week over week, repair rate above 5%, groundedness score below threshold, cache hit ratio below baseline. Absolute latency alerts fire constantly; ratio alerts fire when something actually changed.

From dashboards to decisions

The output of good observability isn't a wall of charts — it's a weekly list of the three most expensive user journeys and the three lowest-quality ones. Every optimization ticket traces back to a span.

For how this fits inside enterprise governance, see the Silverberry AI Platform.