Most LLM observability tools were built for single model calls: a prompt goes in, a completion comes out, and the trace is two spans deep. That model breaks the first time you deploy an agent that calls fifteen tools, decides its own control flow, and re-sends its whole conversation on every turn.
An AI agent observability tool solves a different problem. The trace is 2,000 spans, the failure happens four tool calls deep, and you need to know what the agent said, what each tool returned, and which step went wrong without reading every span in order.
This guide compares seven agent and LLM observability tools in 2026: what each one is, who it suits, where it is strong and weak, and what it costs.
Tracevell publishes this guide and is one of the tools below; we list it first because it is ours, not because of a score. Competitor details come from each vendor's own docs, pricing pages, and license files, checked in October 2026.
Best AI agent observability tools at a glance
| Tool | Type | License | Self-hosting | OpenTelemetry | Pricing model | Best for |
|---|---|---|---|---|---|---|
| Tracevell | Agent observability platform (SDK + OTLP) | Apache 2.0 | Yes, Docker Compose or Helm | Native OTLP (gRPC, HTTP) | GB stored after dedup, no seat fees | Teams debugging long, multi-step agents |
| Langfuse | LLM observability + prompt management | MIT | Yes | OTLP over HTTP | Units (traces + observations + scores) | Prompt-centric teams that want open source |
| LangSmith | LLM and agent observability from LangChain | Closed | Enterprise add-on | OTel endpoint | Per seat + per trace (LangChain Standard Units) | LangChain and LangGraph teams |
| Braintrust | Evaluation platform with tracing | Closed | Enterprise (hybrid data plane) | OTel endpoint | GB processed + scores | Eval-first teams running regression tests |
| Arize Phoenix / AX | OSS tracing and evals / managed platform | Elastic License 2.0 (Phoenix) | Phoenix yes; AX on Enterprise | Native OTLP, OpenInference | Phoenix free; AX spans + GB | Teams using OpenInference or already on Arize |
| Datadog LLM Observability | APM-native LLM module | Closed | No | OTLP over HTTP (GenAI conventions 1.37+ or OpenInference) | Per LLM span | Teams already on Datadog APM |
| Opik | OSS tracing and evaluation from Comet | Apache 2.0 | Yes | OTLP over HTTP | Spans | Teams that want an Apache-licensed alternative to Langfuse |
How we evaluated
We compared the tools on the criteria that decide how useful an agent observability platform is after launch, and that the other widely cited comparisons also use:
- Agent trace reading. Can you read a 1,000+ span run as what the agent did, with sub-agents and tool calls in order, or only as a span tree?
- Instrumentation model. SDK-based, gateway/proxy-based, or APM-native. A gateway sees every request with no code changes but cannot see tool calls; an SDK sees the whole run but needs instrumentation; an APM module sits next to your service traces.
- OpenTelemetry support. Native OTLP ingest, and which transports and semantic conventions.
- Traces to eval datasets. Can you turn a production trace into a dataset row and a regression test?
- Human review. Annotation queues or labeling for traces and outputs.
- Failure detection. Does anything read runs for you and group recurring failures, or do you write every judge yourself?
- Cost attribution. Token cost per span, rolled up per run, per agent, and per user.
- Self-hosting and license. What runs on your own infrastructure, and under which license.
- Pricing model. The billing unit, and how it scales with long agent traces.
We did not run a benchmark across all seven tools; this is a documentation and pricing review. The Tracevell sections also reflect our own use of the product.
What makes agent observability different
A normal LLM observability tool logs prompts, completions, tokens, and latency. That is enough when your app is a single chain. Agents add four problems (more in What is agent observability?):
- Long traces. A research agent can produce thousands of spans across LLM calls, tool calls, sub-agents, and retries, with the full conversation re-sent on every turn.
- Non-deterministic control flow. The agent picks the next tool, so the trace shape changes every run.
- Nested causality. A failure at span 1,800 can come from a bad retrieval at span 42.
- Session continuity. One task can span several process invocations, and the trace model has to stitch them together.
1. Tracevell
What it is: An open-source observability platform built for AI agents, with tracing, evals, failure detection, and SQL over all trace data. Apache 2.0. Cloud or self-hosted (Docker Compose or Helm). Repo: github.com/lmnr-ai/lmnr.
Who it suits: Teams running long, multi-step agents (coding agents, browser agents, research agents) who need to find why a specific run went wrong and which failures recur across thousands of runs.
Strengths
Agent runs read as a conversation. Tracevell's transcript view is the default: what the agent said, what each tool returned, in order. Sub-agents collapse into cards and scope to their own spans when expanded. The span tree and timeline are one click away when you need to see what ran in parallel.
Failure detection that reads the whole run. Signals turn a plain-language instruction plus a JSON schema into a structured event on every matching trace, backfilled across history. Clustering then groups the events, so you see the ten things your agent does wrong ranked by frequency. On Tracevell Cloud the analysis runs on flow-1, our trace model: on our benchmark of 523 difficult traces it scored 0.835 detection F1 against GPT-6-sol's 0.816, at about 1/20th of the cost per trace. Larger frontier models still score higher. See how Tracevell's failure detection works.
SQL over all platform data. Traces, spans, signal events, evals, and metadata are queryable with SQL from the editor, the CLI, the MCP server, or the API. "How many runs called tool X more than five times and then errored" is one query.
A debugger your coding agent drives. Start your agent with LMNR_DEBUG=true and Claude Code, Cursor, or Codex can read the run through the CLI, edit your code, and rerun, with the prefix of the run served from cache so the step under test repeats in seconds.
Code-first evals. Evaluations are a dataset, an executor, and evaluator functions in plain Python or TypeScript. You can turn production traces into dataset rows (how), label them in labeling queues, and compare runs over time.
Storage built for repeated context. Agents re-send their history every turn. Tracevell stores each unique message once per trace and reconstructs the full trace at query time, an average 20x storage reduction (write-up), which is why it can bill on stored data.
Integrations. Python and TypeScript SDKs, native OTLP, and documented integrations for OpenAI, Anthropic, Gemini, LiteLLM, OpenRouter, LangChain and LangGraph, Vercel AI SDK, Mastra, Pydantic AI, Google ADK, OpenAI Agents SDK, Claude Agent SDK, Microsoft Agent Framework, Temporal, Claude Code, Codex, OpenCode, Browser Use, Stagehand, Playwright, and more. Any OpenLLMetry or OpenInference instrumentation also works over OTLP.
Weaknesses
- No prompt management comparable to Langfuse or LangSmith.
- Signals, clustering, and alerts need Tracevell Cloud or an enterprise license; the open-source build covers tracing, search, dashboards, SQL, the debugger, and evals.
- If you only log single LLM calls with no tool use, its agent features add little.
Pricing
Billed by data stored after deduplication, with no seat fees (pricing). Free: 1 GB, 7-day retention, 1 seat. Starter: $30/month for 3 GB, then $2/GB, 30-day retention. Pro: $150/month for 10 GB, then $1.50/GB, 6-month retention. Paid plans include unlimited seats and projects and a one-time $5 Signals credit; Signals are billed by the tokens flow-1 uses to analyze your traces. Self-hosting is free.
2. Langfuse
What it is: Open-source LLM observability with prompt management and evals. MIT licensed except the ee/ folders, which need an enterprise license. Cloud or self-hosted. ClickHouse acquired Langfuse in January 2026; it remains MIT licensed and self-hostable.
Who it suits: Teams whose workflow centers on prompt iteration and dataset management, and who want an open-source tool with a large community.
Strengths
- First-class prompt management with versioning and a playground.
- Explicit data model: traces, typed observations, sessions, scores.
- Datasets, experiments, LLM-as-judge evaluators, and annotation queues for human review.
- OTLP ingestion over HTTP, so it works as a generic OTel backend.
- Agent graph view (in beta) and Sessions for multi-step runs.
Weaknesses
- The trace view is organized around observations; long agent runs are read as a tree, not as a conversation.
- No built-in grouping of recurring failures; judges you write score one observation at a time.
- Queries are filters and the API; there is no SQL editor.
- Unit-based billing counts every observation, so agents with many small tool spans add up.
Pricing
Billed in units: traces + observations + scores. Hobby is free with 50k units a month and 30-day access. Core is $29/month and Pro $199/month, each with 100k units, then $8 per 100k units, falling to $6 at volume. Enterprise starts at $2,499/month. Self-hosting all core features is free; features under ee/ need a license. More detail: Langfuse alternatives.
3. LangSmith
What it is: LangChain's closed-source observability, evaluation, and deployment platform. Cloud, hybrid, or self-hosted on Enterprise.
Who it suits: Teams building on LangChain or LangGraph who want the tightest integration and a purpose-built agent IDE.
Strengths
- One environment variable traces every LangChain and LangGraph run.
- LangSmith Studio (formerly LangGraph Studio): visualize the graph, set breakpoints, edit state, resume from a checkpoint.
- Threads view with trajectory and turn-level reading.
- Online evaluators, annotation queues, dashboards, and alerts; Engine audits traces for issues.
- OTLP ingestion over HTTP, mapping GenAI, OpenInference, and OpenLLMetry attributes.
Weaknesses
- Closed source; self-hosting requires Enterprise.
- Per-seat pricing grows with team size.
- Teams outside LangChain get less of the integration advantage.
- Base retention is 14 days; extended retention (180 days on SaaS) costs extra per trace.
Pricing
Developer is free for one seat with 5k base traces a month. Plus is $39 per seat per month with 10k base traces; usage beyond that is billed in LangChain Standard Units ($1 per LSU). Self-hosting and hybrid deployment are Enterprise only. Five engineers on Plus pay $195 a month before usage. More detail: LangSmith alternatives.
4. Braintrust
What it is: An evaluation platform with tracing, logging, and a prompt playground. Closed source; Enterprise can run the data plane in your cloud.
Who it suits: Teams whose main bottleneck is "did this change break behavior X", with evals in CI.
Strengths
- Mature experiments: scorers, side-by-side comparisons, regression detection.
- Prompt playground tied to datasets and eval runs.
- Online scorers on sampled production logs, and Topics for grouping.
- SQL over logs (btql, bt sql) and an OTel endpoint.
- Unlimited users on paid plans.
Weaknesses
- Tracing serves the eval loop; long agent runs are read as spans and threads rather than a transcript.
- Production failure detection depends on scorers you write.
- Self-hosting is Enterprise-only.
Pricing
Billed on GB processed plus scores. Starter is free with 1 GB, 10k scores, and 14-day retention, then $4/GB. Pro is $249/month with 5 GB, 50k scores, and 30-day retention, then $3/GB. Users are unlimited. Enterprise is custom and adds the hybrid data plane. More detail: Braintrust alternatives.
5. Arize Phoenix and Arize AX
What it is: Phoenix is Arize's source-available tracing and evaluation tool (Elastic License 2.0), built on OpenTelemetry and the OpenInference conventions. Arize AX is the managed platform for production monitoring.
Who it suits: Teams standardizing on OpenInference, notebook-first evaluation workflows, or companies already using Arize for ML monitoring.
Strengths
- OTel-native, OTLP over HTTP and gRPC; OpenInference instrumentations cover many frameworks.
- Phoenix runs locally or in a notebook with pip install.
- Phoenix Evals and AX online evals, with AX grouping issues into signals.
- A path from open source to a managed enterprise product.
Weaknesses
- Elastic License 2.0 is source-available, not OSI open source.
- Trace UX is span-tree-first.
- Phoenix and AX are separate products with different cost curves.
Pricing
Phoenix is free to self-host with no feature limits. Arize AX Free includes 25k spans, 1 GB, and 15-day retention; AX Pro is $50/month for 50k spans, 10 GB, and 30-day retention. Enterprise is custom. More detail: Arize Phoenix alternatives.
6. Datadog LLM Observability
What it is: Datadog's LLM module, now called Agent Observability in its docs. It can run alongside APM or be bought on its own. It ingests LLM and agent spans next to your service traces.
Who it suits: Teams already running Datadog APM who want LLM errors, latency, and cost in the same monitors and dashboards.
Strengths
- Agent traces sit next to infrastructure and service traces, with the same monitors, alerting, and on-call workflow.
- Execution Graph for agent structure; flame graph for timing.
- Nine opt-in judge templates (e.g., Failure to Answer, Hallucination, Tool Selection), custom judges, experiments, datasets, and human annotation.
- Accepts OTLP over HTTP/protobuf using GenAI conventions 1.37+ or OpenInference.
- Estimated cost per LLM span, summed per trace.
Weaknesses
- OTLP spans without gen_ai.operation.name or gen_ai.provider.name are dropped; check attribute mapping before relying on an existing exporter.
- Framework integrations are mostly Python.
- Recurring-failure grouping (Patterns) runs weekly by default; every behavioral check is one you define.
- No self-hosting; billing is separate from APM.
Pricing
Billed per LLM span (calls to a model provider); tool, workflow, and agent spans are not billed. The free tier includes 40k LLM spans a month with 15-day retention. Pro starts at $160/month billed annually for 100k LLM spans, then $3.50 per 10k ($200/month and $4.20 per 10k month-to-month). Longer retention is an add-on.
We already use Datadog. Do we need a separate platform? Not always. If your agent is a few LLM calls and the questions are latency, errors, and spend, Datadog's module is enough and keeps one bill. A dedicated tool earns its place when runs are long, failures are silent (HTTP 200 with a wrong answer), and you need to read whole trajectories, find recurring failures without writing every judge, or turn traces into eval datasets. Many teams run both: APM for services, an agent tool for runs.
7. Opik
What it is: Comet's open-source (Apache 2.0) tracing, evaluation, and prompt optimization platform. Cloud or self-hosted.
Who it suits: Teams that want an Apache-licensed, self-hostable tracing and eval tool and can work within its query and access-control limits.
Strengths
- Apache 2.0 with a full self-host option.
- Threads for conversations; Agent Graph renders automatically for Google ADK and LangGraph (with a graph parameter).
- Evaluation metrics, datasets, experiments, and annotation for human review.
- Automatic cost for listed providers (OpenAI, Anthropic, Google, Bedrock, Groq).
- Agent Optimizer for automated prompt and tool optimization (GEPA, few-shot Bayesian, evolutionary, and others).
Weaknesses
- OTLP over HTTP only, with Opik-specific attributes rather than GenAI conventions.
- Opik Query Language combines conditions with AND only.
- Self-hosted Opik lacks user management.
- Unsupported models have empty cost fields.
Pricing
Self-hosting is free. Free Cloud includes 25k spans a month, 60-day retention, and up to 10 members. Pro Cloud is $19/month for 100k spans, then $5 per 100k. Enterprise is custom.
How to choose
Start from the problem that costs your team the most time today:
| If your main need is… | Start with |
|---|---|
| Finding why long, multi-step agent runs fail, and which failures recur across thousands of runs | Tracevell. See also AI agent failure detection tools. |
| Building on LangGraph with an agent IDE | LangSmith |
| Regression-testing changes before release | Braintrust, or see AI agent evaluation platforms |
| Prompt iteration in an open-source tool | Langfuse or Opik |
| Standardizing on OpenInference, or already on Arize | Phoenix or Arize AX |
| Keeping LLM metrics in your existing Datadog monitors, with short agents | Datadog LLM Observability |
| Logging and cost per request without touching agent code | A gateway; see AI agent cost tracking tools |
Whatever you pick, prefer a tool that accepts OpenTelemetry. Standard instrumentation keeps you from being locked into one vendor's SDK, though each backend maps attributes its own way, so check a sample of real runs before switching.
More on transports and attributes: OpenTelemetry for AI agents and best LLM tracing tools. To compare bills on the same workload, see LLM observability pricing.
FAQ
What is agent observability?
Agent observability is capturing and inspecting the full execution of an AI agent: every LLM call, tool call, retrieval, and sub-agent invocation, in order. It differs from LLM observability because agent runs are long, non-deterministic, nested, and re-send their conversation every turn, so the tool has to make whole runs readable and find failures across many of them.
What are the best AI agent observability tools in 2026?
It depends on your stack and bottleneck. Tracevell fits teams debugging long, multi-step agents. Langfuse and Opik are open-source options for prompt-centric workflows. LangSmith fits LangChain and LangGraph teams. Braintrust is eval-first. Arize Phoenix fits OpenInference users. Datadog LLM Observability fits teams already on Datadog APM.
What are the best open-source LLM observability tools?
Tracevell (Apache 2.0), Langfuse (MIT, except ee/), and Opik (Apache 2.0) can all be self-hosted. Arize Phoenix is source-available under the Elastic License 2.0. Check which features each open-source build includes: Tracevell's Signals and Langfuse's ee/ features need a commercial license.
Do I need a dedicated agent observability tool if I already use Datadog?
Not necessarily. Datadog's LLM module is a good fit if your agent is short and your questions are latency, errors, and cost. A dedicated tool is worth it when runs are long, failures are silent, and you need to read full trajectories, detect recurring failures, or build eval datasets from traces.
Which tools can turn production traces into eval datasets?
Tracevell, Langfuse, LangSmith, Braintrust, Arize, and Opik all let you add traces or spans to datasets for evaluation. The walkthrough in how to turn production traces into an eval dataset uses Tracevell.
Can I send OpenTelemetry traces to these platforms?
Tracevell, Langfuse, LangSmith, Braintrust, Phoenix, Opik, and Datadog all accept OTLP, with different transports and conventions: Tracevell and Phoenix accept gRPC and HTTP, several others HTTP only, and Datadog requires GenAI or OpenInference attributes. Gateways log requests at the proxy instead.
What does agent observability cost?
Billing units differ: Tracevell charges per GB stored, Langfuse per unit, LangSmith per seat and per trace, Braintrust per GB processed and per score, and Datadog per LLM span. Long agent traces with repeated context cost very differently across these models. See LLM observability pricing for the same workload priced on six platforms.
When is Tracevell the right choice?
When you run long, multi-step agents and spend your time working out why specific runs failed and which failures keep recurring. If prompt management is your main workflow, Langfuse or LangSmith fit better; if you are on Datadog with short agents, its module may be enough.
Last updated: October 19, 2026. Verify features and pricing against each vendor's current documentation before committing.