LLM observability tools record what your AI feature actually did on every request: the prompt, the retrieved context, each model and tool call, the tokens, the cost and the time taken.
Monitor five layers, not one: traces, cost and tokens, latency, output quality, and safety. Most teams start with cost and miss quality.
The best LLM observability tools split into four groups: open-source tracing (Langfuse, Arize Phoenix), framework platforms (LangSmith), evaluation-first platforms (Braintrust, Galileo) and APM add-ons (Datadog). Pick the group before the tool.
The market is moving fast. Helicone was acquired by Mintlify in March 2026 and is in maintenance mode, and Cisco announced plans to acquire Galileo in April 2026. Check a vendor's roadmap before you commit.
Instrument with OpenTelemetry where you can. Its GenAI conventions are still marked "Development", but tracing through a standard keeps you free to switch tools later.
A support assistant has worked well for three months. Then the monthly model bill doubles, and nobody can say why. Was it longer prompts, a retry loop, a new feature, or one customer pasting whole contracts into the chat?
Regular application monitoring cannot answer that. It sees a request that took four seconds and returned a 200. It does not see that the request made three model calls, retrieved the wrong document, and produced an answer that quoted a policy you retired in June.
That gap is what LLM observability fills. This guide explains what to monitor in production, how the main LLM observability tools compare on pricing, hosting and strengths, and how to choose one without locking yourself into a vendor that may not exist in its current form next year.
What is LLM observability?
LLM observability is the practice of recording and analysing every step of an LLM application in production, including prompts, retrieved context, model calls, tool calls, token usage, cost, latency and output quality, so teams can explain why a response was good, bad, slow or expensive and fix the cause.
LLM observability vs LLM monitoring vs LLM evaluation
The three terms overlap, and vendors use them loosely.
LLM monitoring
LLM observability
LLM evaluation
Main question
Is something wrong right now?
Why did this request behave this way?
Is this version good enough to ship?
What it uses
Dashboards and alerts on aggregates
Full traces of individual requests
Test sets, scores and graders
When it runs
Continuously in production
Continuously, plus during debugging
Before release, and on samples of live traffic
Example
Cost per hour jumped 60%
One user's request looped through a tool eight times
Faithfulness dropped from 91% to 84% after a prompt change
Good LLM observability tools now cover all three, because a trace you cannot score and a score you cannot trace back to a request are both half a tool. For the scoring side, see our guide to LLM evaluation metrics.
What to monitor in production LLM applications
Group what you track into five layers. Each one catches a different kind of failure.
1. Traces: what happened inside each request
A trace is the full record of one request: the user input, the system prompt, each retrieval step, every model call with its inputs and outputs, every tool call, and the final answer, with timings for each step.
Track these from day one:
The span tree for every request, so a slow or wrong answer can be opened and read step by step.
The prompt and model version used on each call, so a change can be tied to a shift in behaviour.
Retrieved documents and their IDs, so you can tell a retrieval failure from a generation failure.
Tool calls and their results, including errors and retries.
A session or conversation ID, so multi-turn problems can be followed across requests.
2. Cost and tokens: where the money goes
Model bills grow quietly. Watch:
Input and output tokens per request, broken down by feature, model and customer.
Cost per request and cost per successful task. A cheap call that fails and retries three times is not cheap.
Cached tokens, if your provider supports prompt caching, to confirm the cache is being used.
The heaviest users and prompts, since a small share of requests often drives a large share of spend.
3. Latency: what the user waits for
Average latency hides the problem. Track:
Time to first token for streamed responses, which is what the user feels.
Total response time at the 95th and 99th percentile, not the mean.
Latency per step, so you can see whether the model, the retriever or a slow tool is the bottleneck.
Provider errors, rate limits and timeouts, which show up as latency spikes before they show up as outages.
4. Quality: whether the answers are any good
This is the layer most dashboards skip, and the one users notice first.
Online evaluation scores on a sample of live traffic, such as faithfulness to retrieved context, answer relevance and format validity.
User feedback: thumbs up and down, edits to generated text, and whether the user asked the same question again.
Escalation and fallback rate: how often the system hands off to a person or a default answer.
Quality broken down by topic or customer segment, because a healthy average can hide one category failing badly.
5. Safety and drift: what changes without anyone deploying
Policy and safety flags, such as toxic output, prompt injection attempts and leaked personal data.
Input drift: new kinds of questions the system was never tested on.
Output drift after provider updates, when the same prompt starts producing different answers because the model behind an alias changed.
Alerts worth setting first
A handful of alerts catches most production problems:
Alert
What it catches
Typical starting rule
Cost per hour above baseline
Retry loops, prompt bloat, abuse
50% above the trailing seven-day average for the same hour
95th percentile latency
Slow providers, slow tools
Above your product's acceptable wait, for 10 minutes
Error and timeout rate
Provider outages, rate limits
Above 2% of requests for 5 minutes
Online quality score
Prompt or model regressions
A drop larger than the score's normal day-to-day range
Tool call loops
Agents stuck repeating an action
More than a set number of calls to the same tool in one request
These thresholds are starting points, not benchmarks. Set them from your own first two weeks of data.
Best LLM observability tools compared
The tools below cover most production use. Pricing is the vendors' published pricing as of mid-2026, collected from a July 2026 comparison of 12 platforms. Prices change often, so confirm before you buy.
Tool
Best for
Open source or self-host
Free tier
Paid starting price
Langfuse
Teams wanting open-source tracing with evals and prompt management
MIT core, self-host yes
50,000 units a month on cloud
From $29 a month
Arize Phoenix
Teams wanting a free, self-hosted tracing and eval workbench
Teams already running Datadog for the rest of their stack
Proprietary, no self-host
Not listed
About $8 per 10,000 LLM spans, billed annually
Portkey
Teams wanting a gateway with logging, routing and fallbacks
Gateway MIT, control plane hosted
10,000 logs a month
$49 a month for 100,000 logs
OpenLLMetry (Traceloop)
Teams wanting OpenTelemetry-based instrumentation
Apache 2.0, self-host yes
Free SDK; 50,000 spans a month on cloud
Not listed
Langfuse
Langfuse is the most widely adopted open-source option. It covers tracing, cost tracking, prompt management, datasets and evaluation in one product, and it can be self-hosted, which matters when prompts contain customer data that cannot leave your infrastructure. The trade-off is that self-hosting means you run and upgrade it.
Arize Phoenix
Phoenix is free to run and strong on tracing and evaluation, with good support for OpenTelemetry-based instrumentation. It suits teams that want a local workbench for debugging and evaluation. Note the licence: Elastic License 2.0 is source-available, not open source in the strict sense, which some legal teams care about.
LangSmith
LangSmith is the natural choice if your application is built on LangChain or LangGraph, because tracing works with almost no setup. It also works with other frameworks. Pricing is per seat plus per trace, which can add up for larger teams with high traffic, and self-hosting needs an Enterprise contract.
Braintrust
Braintrust starts from evaluation: datasets, experiments, scoring and comparison of prompt or model versions, with production logging attached. It suits teams whose main question is "is the new version better?" The entry price is higher than tracing-first tools.
Datadog LLM Observability
If Datadog already monitors your servers, databases and APIs, its LLM module puts model traces next to everything else, which makes it easier to see when a slow database is the real cause of a slow AI response. It is priced per LLM span and suits teams that value one dashboard over specialist depth in evaluation.
Portkey
Portkey is a gateway first. Every model call passes through it, so it can log, route between providers, retry, apply fallbacks and enforce budgets in one place. Logging comes as a side effect of routing. It suits teams using several model providers.
OpenLLMetry by Traceloop
OpenLLMetry is an instrumentation library built on OpenTelemetry. It does not lock you into a backend: you can send its traces to Traceloop's cloud, to an existing observability stack, or to several tools at once. It suits teams that want to keep their options open.
Tools that changed hands in 2026
Two well-known names are in transition. Helicone was acquired by Mintlify on March 3, 2026 and is in maintenance mode with no active roadmap. Cisco announced its intent to acquire Galileo on April 9, 2026. Existing users are not necessarily affected, but new projects should weigh where each product is heading.
Choosing a tool is the easy decision. Deciding what to trace, which quality scores to run on live traffic and where the alerts should fire takes longer, and it is where most setups stay shallow. Codiste designs that monitoring layer around your own application and says plainly when an open-source tool covers it without custom work.
How to choose an LLM observability tool
Answer these questions in order. The first two usually narrow the field to two or three tools.
Can prompts and outputs leave your infrastructure? If customer data, health data or contracts flow through prompts and policy says no, start with self-hostable tools such as Langfuse, Phoenix or OpenLLMetry with your own backend.
What do you already run? A team on Datadog gains a lot from keeping LLM traces in the same place. A team on LangChain gets the fastest start from LangSmith.
Is your main question "what happened" or "is it better"? Tracing-first tools answer the first; evaluation-first tools such as Braintrust answer the second. Most teams need both eventually, so check how each tool handles the other half.
How will cost grow? Model the bill at ten times today's traffic. Per-seat, per-trace, per-span and per-unit pricing diverge sharply at scale.
Can you leave? Prefer tools that accept OpenTelemetry data and let you export traces. Instrumentation is the expensive part to redo.
Will the vendor exist in this form next year? Check the roadmap, recent releases and ownership. 2026 has already seen one acquisition into maintenance mode.
Why OpenTelemetry matters for LLM observability
OpenTelemetry is the open standard most of the industry uses for traces, metrics and logs. Its GenAI semantic conventions define common names for LLM data, such as the model requested, the provider, input and output token counts, and agent and tool calls.
One caveat matters. As of September 2026 the GenAI conventions are still at "Development" status, the first of three stability levels, which means attribute names can still change between releases. Prompt and response text is kept out of span attributes by default and recorded separately as opt-in content, because it often contains personal data.
The practical advice is still to instrument through OpenTelemetry where your stack supports it. A trace written to a common standard can be sent to a new tool without rewriting your application, and that is the cheapest insurance against a vendor changing direction.
Common LLM observability mistakes
Logging everything, forever. Full prompts and outputs fill storage and often hold personal data. Set retention periods and mask sensitive fields before data leaves the application.
Tracking cost but not quality. A dashboard that shows spend and latency looks healthy while answers get worse. Run online evaluation on a sample from the first week.
Using averages. Mean latency and mean scores hide the slow tail and the failing topic. Use percentiles and break scores down by segment.
No link between traces and releases. If a trace does not record the prompt version and model version, you cannot connect a regression to the change that caused it.
Sampling away the problems. Sample routine traffic if you must, but keep every trace with an error, a low score, negative feedback or an escalation.
Treating the model alias as fixed. A model name can point to an updated version. Record the exact model version the provider returns.
How to set up LLM observability in five steps
Instrument the main path first. Trace the user request, retrieval, each model call and each tool call, with prompt and model versions attached.
Add cost and latency dashboards broken down by feature and model, using percentiles.
Turn on online evaluation for one or two quality scores that match your biggest risk, such as faithfulness for a document assistant.
Set five alerts from the table above, using your first two weeks of data as the baseline.
Close the loop. Every bad trace a person reviews should become a test case in your evaluation suite, so the same problem is caught before the next release.
LLM observability options
Codiste: designs and builds the observability and evaluation layer around your own LLM or agent application, including tracing, cost and quality dashboards, online evaluation and alerts, on the tools that fit your hosting and data rules, then hands it over to your team. See our AI agent development services.
Open-source tracing: Langfuse, Arize Phoenix and OpenLLMetry, for teams that want control and can run the infrastructure.
Framework platforms: LangSmith, for LangChain and LangGraph teams.
Evaluation-first platforms: Braintrust, for teams that compare prompt and model versions constantly.
APM add-ons: Datadog LLM Observability, for teams that want LLM traces next to the rest of their monitoring.
Gateways: Portkey, for teams routing across several model providers.
Codiste builds LLM and agent systems and the monitoring around them, so we have a commercial interest in this topic. Several of the tools above are free and open source, and for many teams they are the right place to start.
Final verdict
LLM observability tools exist because standard monitoring cannot see inside an AI request. It can tell you a call was slow; it cannot tell you the model retrieved the wrong document, looped through a tool, or quietly started citing an old policy. Production teams need that inside view the moment real users arrive.
What you monitor matters more than which tool you buy. Cover all five layers: traces, cost, latency, quality and safety. Teams that stop at cost and latency end up with healthy dashboards and unhappy users, because quality is the layer customers notice first. Run at least one online quality score from the first week, and keep every trace that carries an error, a low score or a complaint.
When choosing among the best LLM observability tools, start with your constraints rather than feature lists. Data that cannot leave your infrastructure points to self-hosted options. An existing Datadog setup or a LangChain codebase points to the tools built around them. Heavy experimentation points to evaluation-first platforms. Then model the cost at ten times today's traffic, because pricing units diverge sharply at scale.
Finally, protect your exit. 2026 has already seen one of the better-known tools move into maintenance mode after an acquisition. Instrument through OpenTelemetry where you can, keep traces exportable, and the tool becomes a choice you can revisit rather than a dependency you are stuck with.
If your LLM feature is live and nobody can say why last month's bill or last week's answers changed, Codiste will review what you trace today and show you what is missing before the next incident.
FAQs
What are LLM observability tools?
LLM observability tools record and analyse what an LLM application does on each request in production: prompts, retrieved context, model and tool calls, tokens, cost, latency and output quality. Teams use them to debug bad answers, control spend, catch regressions after changes, and connect production problems back to specific prompts or model versions.
What should you monitor in an LLM application?
Monitor five layers: full traces of each request, token usage and cost, latency at the 95th percentile and time to first token, output quality through online evaluation and user feedback, and safety signals such as prompt injection, data leaks and drift after provider updates. Quality is the layer teams most often skip.
What are the best LLM observability tools in 2026?
It depends on your constraints. Langfuse and Arize Phoenix lead for open-source and self-hosted setups, LangSmith for LangChain teams, Braintrust for evaluation-heavy teams, Datadog for companies already using its monitoring, and Portkey for teams routing across several model providers. Check data residency and pricing at scale before choosing.
What is the difference between LLM observability and LLM monitoring?
Monitoring watches aggregate signals such as cost, latency and error rate, and alerts when something crosses a threshold. Observability records each request in enough detail to explain why it behaved the way it did. Monitoring tells you something is wrong; observability tells you what happened and where.
Is there an open-source LLM observability tool?
Yes. Langfuse has an MIT-licensed core and can be self-hosted. OpenLLMetry by Traceloop is Apache 2.0 instrumentation built on OpenTelemetry. Arize Phoenix is free to self-host under the source-available Elastic License 2.0. Self-hosting gives you control over data but means running the infrastructure yourself.
Do LLM observability tools support OpenTelemetry?
Many do, either by accepting OpenTelemetry traces directly or through libraries such as OpenLLMetry. OpenTelemetry's GenAI conventions were still marked "Development" as of September 2026, so attribute names can change. Instrumenting through OpenTelemetry still makes it easier to switch tools later.
Nishant builds where ambitious ideas meet production reality. As CTO & Co-Founder at Codiste, he has helped ship 150+ AI systems. Through The CTO Story, he gets technical leaders talking about what rarely makes the slide deck: the trade-offs, failures, and decisions behind what actually ships.
Every great partnership begins with a conversation. Whether you're exploring possibilities or ready to scale, our team of specialists will help you navigate the journey.
