AI observability for production LLM systems: what to record and why
AI observability for a production LLM system means recording every model call, retrieval and tool call as one trace. Each trace holds the inputs, outputs, cost, latency and a quality signal. With that record you can answer two questions every owner asks: why did it say that, and is it getting worse.
What is AI observability for a production LLM system?
AI observability is the record that lets an engineer explain any answer an LLM system gave in production. For each user request it captures one trace. The trace holds the prompt the model saw, the documents it retrieved, and the tools it called with their arguments. It also holds the answer returned and what it cost in tokens, money and time.
Ordinary application monitoring tells you a service is up and how fast it responds. An LLM system can be up, fast and wrong. AI observability adds the content and the quality signal that ordinary monitoring leaves out.
What should you record on every LLM call?
Start with the fields that answer “why did it say that” for a single request:
| Field | Why it matters |
|---|---|
| Trace and step IDs | Ties retrieval, tool calls and the answer to one user request. |
| Model name and version | A provider update can change behaviour without any change on your side. |
| Prompt template version | Lets you tie a drop in quality to the prompt change that caused it. |
| Retrieved sources | Shows whether a wrong answer came from the model or from what it was given. |
| Tool calls and arguments | Shows what an agent actually did, not what it said it did. |
| Tokens, cost and latency per step | Finds the step that is slow or expensive, so you can change that step alone. |
| Outcome or feedback | A thumbs-down, an escalation or a business result such as a booking. |
Prompts and answers can contain personal or confidential data. Decide before launch which fields are stored in full, which are redacted, and where the traces live.
What is the difference between LLM monitoring and LLM evaluation?
Monitoring watches live traffic and tells you what happened: volumes, errors, cost, latency and user feedback. Evaluation runs a fixed set of real cases with known good answers and tells you whether the system is right. Monitoring catches the incident; evaluation tells you whether your fix made things better or worse.
Build the evaluation set from production traces. The cases that went wrong in real use are the ones worth testing every time you change a model or a prompt.
Should you use OpenTelemetry for LLM observability?
OpenTelemetry is the open standard most tracing tools already speak. It has a set of conventions for generative AI, covering spans, metrics and events for model calls, agents and MCP tool calls. Those GenAI conventions moved into their own repository, semantic-conventions-genai, and are still being revised (checked October 2026).
The practical advice is to instrument once, in a standard format, and send the traces to whichever backend you choose. That keeps the record yours if you change vendor, and lets traces stay on your own infrastructure when the data has to.
What does observability change in a real system?
It turns arguments into measurements. Super Cat Technology built a 24/7 AI agent call center that handles 200,000 calls a month across 60 US HVAC brands. Its booking rate is measured against human agents: 67% for the AI against 77% for people. That number says where the system stands and what the next change has to move.
Per-step cost and quality records also make model changes safe. On a private-markets search build, a model about 90% cheaper took over entity grouping, a switch you can only make with confidence when each step is measured on its own.
What are the risks of AI observability?
- Results in testing differ from results in use. We measure on the client's real cases, report the numbers as they are, and scope fixes.
- A model or vendor changes under you. Version fields on every trace show when a provider update moved quality. We check vendor terms at selection and re-route when they change.
- The traces hold sensitive data. Logged prompts are a copy of your users' data. Where policy says it stays on your servers, the traces stay there too.
Who should build it?
The engineer who builds the LLM system should instrument it, because they know which step can fail silently. If your team is shipping its first production LLM system, a forward-deployed AI engineer can build the system and its tracing together, then hand both over. For agents with tool access, pair this with a security review before go-live.
In Cat’s words
“On the call centre my team built, the number that matters is the booking rate against the human team: 67% against 77%. Without a trace on every call, we could not say which change moved it. A system you cannot explain at 2am is still a demo.”
What I’m seeing in October 2026
- Vendors now retire models on fixed dates. Teams with traces and an evaluation set treat the swap as a test run. Teams without them hear about it from users.
- Cost has become an engineering question. When each step is measured on its own, a cheaper model can take over one step, as it did on our private-markets search build.
- Tencent said this month that AI agents it ran showed browsing it did not intend. Tool-call traces are moving into the security review.
Cat Yung, as of October 2026
Sources: OpenAI API deprecations (notice of 22 April 2026) · Anthropic model deprecations (notice of 30 September 2026) · The Standard (Reuters) on Tencent’s AI agents (16 October 2026)
Frequently asked questions
What is AI observability for production LLM systems?
AI observability records each request to an LLM system as one trace. The trace holds the prompt, retrieved documents, tool calls, the answer, its cost and latency, and a quality signal. It lets an engineer explain any answer the system gave and see whether quality moves after a model or prompt change.
What should I log for every LLM call?
Log the trace and step IDs, model name and version, and prompt template version. Add retrieved sources, tool calls with arguments, and tokens, cost and latency per step. Finish with an outcome or feedback signal. Decide before launch which fields are redacted, because prompts often hold personal or confidential data.
Is OpenTelemetry ready for LLM tracing?
OpenTelemetry has GenAI conventions for model calls, agents and MCP tool calls, maintained in their own repository and still being revised as of October 2026. Instrumenting in that format keeps traces portable between backends and lets them stay on your own infrastructure.
How is LLM evaluation different from monitoring?
Monitoring watches live traffic for errors, cost, latency and feedback. Evaluation runs a fixed set of real cases with known good answers to check correctness. Monitoring finds the incident; evaluation shows whether a fix improved the system. Build the evaluation set from production failures.
Sources
- OpenTelemetry, GenAI semantic conventions repository (checked October 2026)
An engineer who builds the system and owns the outcome.
Security review for agents before they get tool access.
One who has shipped LLM systems to production.
The six-point check before an agent gets credentials.
About the author
Cat Yung is the founder of Super Cat Technology, an AI agent engineering team in Hong Kong and London, and works as an AI expert, data scientist and fractional CTO. Over a decade in production AI, NLP and machine learning; Expert-Vetted top 1% on Upwork with a 100% Job Success Score (source: Upwork profile, October 2026).