The best LLM observability tool in 2026 depends on what you need to see. For an all-round open-source platform with tracing, evals, cost tracking, and prompt management, pick Langfuse. For LangChain and LangGraph apps, pick LangSmith. For experiment-driven evals, pick Braintrust. If your services already run on Datadog, use Datadog. For the cheapest managed plan at volume or a fully free self-hosted platform, pick Opik. For a free self-hosted workbench with a managed upgrade, pick Arize Phoenix, then Arize AX. To judge whole production conversations and turn failures into pull requests, add Agnost AI alongside one of these.

This page is updated for 2026. A lot changed since 2025: Helicone went into maintenance mode after Mintlify acquired it, Traceloop is joining ServiceNow, Datadog renamed its product to Agent Observability, and most tools added agent and session views. Below we compare eleven tools on tracing, evals, cost tracking, prompts, licence, and price.

If your question is narrower, "how do I know my agent is working for users?", read Best AI agent monitoring tools (2026) instead. That page sorts tools by the monitoring job. This one is the broad LLM observability buyer's guide.

Disclosure: We make Agnost AI. Facts about every other tool come from that vendor's own docs and pricing pages, checked on November 18, 2026. This is a documentation comparison, not a hands-on benchmark. Confirm plan details with each vendor.

What LLM observability covers

An LLM observability tool records what your model and agent did and helps you judge whether it was good. Buyers usually need five things:

  1. Tracing. Every model call, tool call, and retrieval step for a request, with inputs, outputs, latency, and errors.
  2. Evals. Scores on outputs, either offline against a dataset before release or online on production traffic.
  3. Cost and token tracking. Spend by model, user, feature, or agent run.
  4. Prompt management. Versioned prompts you can change without a deploy, plus a playground.
  5. Monitoring and alerts. Dashboards and alerts when quality, cost, or error rates move.

No tool is best at all five. Pick by the one that hurts most today.

Quick comparison

Tool Best for Open source Self-host Cost tracking Prompt mgmt Free tier Entry paid plan Billing unit
Langfuse All-round open source MIT (except ee) Free Yes Yes 50K units Core $29/mo Units
LangSmith LangChain and LangGraph No Enterprise Not stated Not stated 5K traces, 1 seat Plus $39/seat/mo Seats plus traces
Arize Phoenix Free self-hosted workbench ELv2 (source-available) Free Not stated Yes Free — —
Arize AX Managed evals and issue discovery No Enterprise Yes Not stated 25K spans Pro $50/mo Spans and GB
Braintrust Experiments and scorers No Enterprise Yes Prompt versions 1 GB, 10K scores Pro $249/mo GB plus scores
Datadog Teams already on Datadog No No Yes Not stated 40K LLM spans Pro from $160/mo (annual) LLM spans
Opik Cheapest managed, free self-host Apache-2.0 Free, full platform Yes Yes 25K spans Pro $19/mo Spans
W&B Weave Teams on Weights & Biases SDK Apache-2.0 Enterprise Not stated Not stated 1 GB Pro from $60/mo MB ingested
Traceloop / OpenLLMetry OTel instrumentation for any backend OpenLLMetry Apache-2.0 Enterprise on-prem Not stated Yes 50K spans, 24 h Enterprise only Spans
Laminar Open-source tracing, CI evals, Signals Apache-2.0 Yes (Docker Compose) Not stated Not stated 1 GB, 1 seat Starter $30/mo GB plus Signal tokens
Helicone Not for new projects (maintenance mode) Apache-2.0 Yes Not stated Yes 10K requests Pro $79/mo Requests and storage
Agnost AI Whole-conversation failures turned into PRs No Enterprise (VPC) No No 1K events Starter $49/mo Events

"Not stated" means the vendor pages we checked didn't say, not that the feature is missing.

Langfuse

What it is. An open-source LLM engineering platform. It covers tracing of LLM and non-LLM steps, token and cost tracking per user, prompt management with versioned labels, a playground, LLM-as-a-judge and code evaluators, datasets with experiments, and dashboards that alert to Slack, GitHub Actions, or webhooks. It accepts OpenTelemetry on an OTLP endpoint and has typed Python and JS/TS SDKs plus a drop-in OpenAI wrapper.

Best for. Teams that want one open-source tool for the whole LLM observability list, with the option to self-host.

Limits. Self-hosting is free but not small: you run web and worker containers on Postgres, ClickHouse, Redis or Valkey, and S3-compatible storage. A few add-ons, such as the Instance Management API, need an enterprise licence key. LLM-as-a-judge evaluators now target observations or experiments: trace-level evaluators are deprecated and stopped producing results on Langfuse Cloud after November 16, 2026. See what to do about them.

Pricing. Self-hosting is free. Hobby is free for 50,000 units and 2 users. Core is $29 a month and Pro $199 a month, each with 100,000 units, then $8 per 100,000 up to 1 million and $7 per 100,000 from 1 to 10 million. A unit is any trace, observation, or score.

LangSmith

What it is. LangChain's platform for tracing, datasets, online evaluations, dashboards and alerts, annotation queues, and Engine, which detects recurring issues in traces and helps resolve them. It accepts OpenTelemetry through an /otel endpoint.

Best for. Teams building on LangChain and LangGraph.

Limits. Plus is priced per seat. Base traces are kept 14 days, and extended retention (180 days) costs extra. The public pricing page doesn't print a per-trace overage rate; usage is metered in LangChain Standard Units at $1 each through a calculator. Self-hosting is Enterprise only.

Pricing. Developer is free for 1 seat and 5,000 base traces a month. Plus is $39 per seat per month with 10,000 base traces, then pay as you go. Enterprise is custom.

See also LangSmith alternatives.

Arize Phoenix and Arize AX

What they are. Phoenix is Arize's self-hostable tool built on OpenTelemetry and OpenInference. It covers tracing, LLM and code evaluators, a prompt playground with span replay, and datasets with experiments. AX is the managed platform. It records cost on each traced run, runs online evals continuously, and its Signal feature groups recurring failures into ranked issues with a proposed fix.

Best for. Phoenix fits teams that want a free self-hosted workbench with no feature gates. AX fits teams that want managed evals and issue discovery.

Limits. Phoenix is under the Elastic License 2.0, which is source-available, not OSI open source. Arize's pricing page lists no overage rates for AX, so price anything above the Pro allowance with sales.

Pricing. Phoenix is free to self-host. AX Free covers 25,000 spans and 1 GB a month with 15-day retention. AX Pro is $50 a month for 50,000 spans and 10 GB with 30-day retention. Users are unlimited.

Braintrust

What it is. An evaluation and observability platform built around experiments, datasets, and scorers, with tracing, a Discover Patterns feature that finds recurring trace problems, and an AI assistant called Loop. Its SDK wrappers attach token counts and estimated cost to each LLM span. It accepts OpenTelemetry.

Best for. Teams whose releases run on experiments and who want production logs to feed datasets and scorers.

Limits. Each score is metered, so scoring all production traffic gets expensive. Self-hosting is Enterprise only.

Pricing. Starter is free with 1 GB and 10,000 scores, then $4 per GB and $2.50 per 1,000 scores, with 14-day retention. Pro is $249 a month with 5 GB and 50,000 scores, then $3 per GB and $1.50 per 1,000 scores. Users are unlimited.

Datadog LLM Observability (Agent Observability)

What it is. Datadog's LLM tracing product, now sold as Agent Observability. It covers traces, cost and token dashboards, managed and custom evaluations, Patterns topic clustering, and sensitive-data and prompt-injection scanning. You can instrument with its SDK or send OpenTelemetry spans that follow the GenAI conventions (1.37+) or OpenInference, without a Datadog Agent.

Best for. Teams whose services already run on Datadog and want LLM traces linked to APM.

Limits. No self-hosted option is described. OTel traces can take 3 to 5 minutes to appear. Retention is 15 days unless you pay for an add-on.

Pricing. Free covers 40,000 LLM spans a month. Pro starts at $160 a month on an annual commitment ($200 month-to-month, $240 on-demand) for 100,000 LLM spans, then $3.50 per 10,000 on annual billing. Only calls to a model provider count; tool and retrieval spans are not billed.

See also Datadog LLM Observability alternatives.

Opik

What it is. Comet's Apache-2.0 platform for tracing, cost tracking, LLM-as-a-judge and heuristic evals, prompt management with a playground, and automatic prompt optimization. Online rules score spans, traces, or whole threads, with a sampling rate up to 100%. Thread rules wait for a 15-minute cooldown by default. Score alerts go to Slack, PagerDuty, or a webhook. It accepts OpenTelemetry.

Best for. Teams that want the full platform free to self-host, or the lowest managed price at volume.

Limits. Alerts fire when an average score crosses a threshold. Self-hosting runs locally for testing or on Kubernetes for production.

Pricing. Open source is free. Cloud Free covers 25,000 spans a month. Pro is $19 a month for 100,000 spans, then $5 per 100,000.

W&B Weave

What it is. Weights & Biases' toolkit for tracing and evaluating LLM apps and agents, with LLM judges, custom scorers, and an Agents view of sessions, turns, LLM calls, and tool calls. It accepts OpenTelemetry and reads GenAI, OpenInference, Vercel AI SDK, Traceloop, and other conventions.

Best for. ML teams that already use Weights & Biases.

Limits. The GitHub repo is the Apache-2.0 client SDK; the trace server lives in W&B's core platform, and private hosting is an Enterprise option. Billing is by ingested data, so verbose traces cost more. Pricing now lives on CoreWeave's site.

Pricing. Free includes 1 GB of Weave ingestion a month. Pro starts at $60 a month with 1.5 GB, then $0.10 per MB.

Traceloop and OpenLLMetry

What they are. OpenLLMetry is an Apache-2.0 set of OpenTelemetry extensions for LLM apps. You call Traceloop.init() and it exports to Traceloop or to 24 tested backends, including Datadog, Grafana, Honeycomb, Laminar, New Relic, Sentry, and any OpenTelemetry Collector. Traceloop is the hosted platform on top, with monitoring, evaluators, prompt management, and a proxy called Traceloop Hub (beta).

Best for. Teams that want vendor-neutral instrumentation they can point at any backend.

Limits. Traceloop announced in March 2026 that it is joining ServiceNow. It says it will keep selling Traceloop and that OpenLLMetry will remain open source, with roadmap changes announced in advance. The free plan keeps data for 24 hours.

Pricing. Free Forever covers up to 50,000 spans a month with 24-hour retention. Above that is Enterprise, priced by quote, with an on-prem option.

Laminar

What it is. Apache-2.0, OpenTelemetry-native agent observability written in Rust. It has one-line auto-tracing for the Vercel AI SDK, OpenAI, Anthropic, and LangChain, an evals SDK and CLI that run locally or in CI, SQL access to traces, and Signals: plain-English checks an LLM runs on each finished trace, with Slack or email alerts.

Best for. Engineering teams that want open-source tracing with CI evals.

Limits. Signals aren't on the free plan and are billed by tokens. The free plan has one seat and 7-day retention.

Pricing. Free with 1 GB. Starter is $30 a month for 3 GB, then $2 per GB. Pro is $150 a month for 10 GB, then $1.50 per GB, with 6-month retention.

Helicone

What it is. An Apache-2.0 AI gateway with logging. You point an OpenAI-compatible client at Helicone's gateway URL and get routing across 100+ models, sessions, prompts, datasets, and a playground.

Status. Mintlify acquired Helicone on March 3, 2026 and said it "will continue operating in maintenance mode," with security updates, bug fixes, and new models, and that it will help customers migrate. Helicone's site now says it "is now part of Experiential Labs. Details to come soon."

Our take. Don't start a new project on it. If you run it today, keep it patched and plan a move.

Pricing. Hobby is free for 10,000 requests. Pro is $79 a month and Team $799 a month, plus usage-based charges.

Agnost AI

What it is. Agent monitoring for AI agents in production: chat agents, voice agents (through transcripts), copilots, and MCP servers. It is OpenTelemetry-native, with Python and TypeScript SDKs, and works with the OpenAI Agents SDK, Vercel AI SDK, LangChain and LangGraph, Mastra, Pydantic AI, Google ADK, and Agno.

How it differs. Most tools above judge single requests or traces. Agnost groups traffic by user and session and judges whole conversations. It detects failures, user frustration, intents, tool-call errors, loops, and silent wrong answers. Its online evals are plain-language rules: once a conversation has been idle for 15 minutes, an LLM marks each assistant message pass or fail against every rule, in batches, with no sampling. Alerts go to Slack or email. Auto Improve opens a pull request with a fix tested against past runs, with the replay results in the PR, for a person to review.

Best for. Teams whose failures only show up across a conversation, such as a wrong answer three turns in or a user who gives up.

Limits. Agnost is not a full LLM observability platform. It has no cost tracking, no prompt management, no offline datasets or experiments, and no CI gating. Verdicts are pass or fail, not scores, and judging runs in batches, not in real time. It is not open source. Most teams run it next to Langfuse, LangSmith, Braintrust, or Datadog.

Pricing. Free covers 1,000 events a month with 7-day retention. Starter is $49 a month for 10,000 events. Pro is $499 a month for 1,000,000 events. Enterprise is custom and includes self-hosted VPC deployment.

How to choose

If your priority is… Pick
One open-source tool for tracing, evals, cost, and prompts Langfuse
The full platform free to self-host, or the cheapest managed plan at volume Opik
LangChain or LangGraph LangSmith
Experiments, datasets, and scorers gating releases Braintrust
LLM traces next to your existing APM Datadog
A free self-hosted workbench now, managed later Arize Phoenix, then Arize AX
You already train models in Weights & Biases W&B Weave
Vendor-neutral OpenTelemetry instrumentation OpenLLMetry, exporting to any backend above
Open-source tracing plus evals in CI Laminar
Knowing whether whole conversations worked for users, with fixes as PRs Agnost AI, alongside one of the above

Three checks before you commit:

  1. Instrument with OpenTelemetry if you can. Every tool above accepts OTel in some form, so you can switch or send to two backends without re-instrumenting. Our OpenTelemetry guide shows the spans to emit.
  2. Price your own volume. Units, traces, spans, LLM spans, GB, MB, scores, and events don't convert one-to-one. We worked the numbers at 10K, 100K, and 1M traces a month in the easiest and best-value monitoring tools.
  3. Decide on self-hosting early. If data can't leave your network, see free and open-source agent monitoring tools for what each licence allows.

Related reading

Frequently asked questions

What is the best LLM observability tool in 2026?

There is no single best tool; it depends on the job. Langfuse is the best all-round open-source pick, with tracing, evals, cost tracking and prompt management, free to self-host. LangSmith is best for LangChain and LangGraph teams. Braintrust is best when releases run on experiments and scorers. Datadog is best if your services already run on Datadog. Opik is the cheapest managed plan at volume and fully free to self-host. Arize Phoenix is a free self-hosted workbench with Arize AX as the managed upgrade. Agnost AI is best for judging whole production conversations and turning failures into pull requests, alongside one of the others.

What is the best open-source LLM observability tool?

Langfuse (MIT, except its ee folders) and Opik (Apache-2.0, full platform free to self-host) are the most complete. Laminar is Apache-2.0 with tracing, evals and Signals. OpenLLMetry is an Apache-2.0 OpenTelemetry instrumentation library, not a backend. Arize Phoenix is free to self-host with no feature gates, but under the Elastic License 2.0, which is source-available rather than OSI open source. W&B Weave's SDK is Apache-2.0, but its trace server is not in that repo.

What is the difference between LLM observability and AI agent monitoring?

LLM observability records and inspects model calls: traces, token usage, cost, latency, prompts and eval scores on individual requests. AI agent monitoring asks whether the agent did its job for the user across a whole session or conversation: failed tasks, tool-call errors, loops, frustration and wrong answers that never throw an error. Most teams need both, and most tools lean one way.

Which LLM observability tools track cost?

Langfuse, Opik, Datadog, Arize AX and Braintrust all record token usage and estimated cost on traced LLM calls, according to their docs. Agnost AI does not track cost, so pair it with one of these if spend matters.

Is Helicone still maintained?

Mintlify acquired Helicone on March 3, 2026 and said Helicone will continue operating in maintenance mode, with security updates, bug fixes and new models, and that it will help customers migrate to another platform. Helicone's own site now says it is part of Experiential Labs, with details to come. We would not pick it for a new project in November 2026.

Which LLM observability tool is cheapest?

Self-hosting Langfuse, Opik or Arize Phoenix costs no licence fee, only your infrastructure. Of the managed plans, Opik Pro is $19 a month for 100,000 spans plus $5 per 100,000 more, which makes it the cheapest at high volume. Langfuse Core is $29 a month for 100,000 units. See our setup and pricing comparison for worked costs at 10K, 100K and 1M traces a month.

Does OpenTelemetry work with LLM observability tools?

Yes, almost all of them. Langfuse, LangSmith, Braintrust, Arize Phoenix, Opik, W&B Weave, Laminar and Agnost AI accept OTLP traces, and Datadog ingests spans that follow the OpenTelemetry GenAI semantic conventions (1.37+) or OpenInference without a Datadog Agent. OpenLLMetry instruments your app with OpenTelemetry and exports to 24 tested backends.

What changed in LLM observability since 2025?

Consolidation. Mintlify acquired Helicone in March 2026 and put it in maintenance mode. Traceloop announced it is joining ServiceNow in March 2026 and said OpenLLMetry will stay open source. Datadog now sells LLM Observability as Agent Observability. W&B pricing moved to CoreWeave's site. LangSmith meters usage in LangChain Standard Units at $1 each. And most tools added agent features: session views, issue clustering and automatic failure detection.

Originally published on agnost.ai.