The best tool to evaluate AI agents depends on which half of evaluation you need: offline evals before you ship, or online evals on production traffic. For offline evals (datasets, experiments, LLM-as-a-judge scorers, and CI gates), Braintrust and LangSmith are the most complete managed platforms, Langfuse and Opik are the open-source platforms you can self-host, and DeepEval and Promptfoo are free frameworks built to run in CI. For online evals, LangSmith, Arize AX, Braintrust, Opik, and Confident AI score live traces with sampling, and Agnost AI judges every whole production conversation against plain-language rules. Most teams end up with one tool from each half.
This guide is updated for 2026. Since 2025, Promptfoo has become part of OpenAI, Galileo has become Splunk Agent Observability, Langfuse has retired trace-level judges, and whole-conversation evaluators have become standard. Each tool below gets the same structure: what it is, what it evaluates offline and in production, its limits, and its price.
Disclosure: We make Agnost AI, and it only covers the production half of agent evaluation. Facts about every other tool come from that vendor's own docs, repository, and pricing pages, checked on November 18, 2026. This is a documentation comparison, not a hands-on benchmark. Confirm plan details with each vendor.
Offline evals vs online evals: the two halves
Offline evals run your agent on a fixed set of test cases (a dataset) and score each output with code checks or an LLM judge. You run them when you change a prompt, a model, or a tool, and compare the new experiment with the last one. Wired into CI, they block a pull request that makes known cases worse.
Online evals score real production traffic after you ship. They find the failures your dataset didn't contain: new intents, drift, and breakdowns that only appear several turns into a conversation. What are online evals explains the split in more depth, and should you run evals in CI or on production traffic covers how to run both.
The two halves connect. A failure found in production should become a regression case in your offline dataset, so it can't come back. Turning production failures into regression tests shows how.
Quick comparison
| Tool | Offline datasets and experiments | CI gate | Online evals on production | Open source | Entry paid plan |
|---|---|---|---|---|---|
| Braintrust | Yes | eval-action GitHub Action, bt eval CLI | Span, trace, group scope; sampling | No | Pro $249/mo |
| LangSmith | Yes | pytest, Vitest/Jest integrations | Run and thread evaluators; sampling | No | Plus $39/seat/mo |
| Langfuse | Yes | experiment-action GitHub Action, pytest/Vitest | Observation-level judges; sampling | MIT (except ee) | Core $29/mo |
| Arize Phoenix / AX | Yes | GitHub Actions and GitLab CI/CD guides (AX) | AX: span, trace, session; sampling | Phoenix ELv2 (source-available) | AX Pro $50/mo |
| DeepEval / Confident AI | Yes | deepeval test run | Confident AI: traces, spans, threads; sampling | DeepEval Apache-2.0 | Starter $200/mo |
| Ragas | Metrics, test data generation, experiments | Not documented | Not documented | Apache-2.0 | Free library |
| Promptfoo | Declarative test configs | CLI exit codes, GitHub Action | Not documented | MIT | Free (open source) |
| Opik | Yes | PyTest integration | Trace and thread rules; sampling | Apache-2.0 | Pro $19/mo |
| W&B Weave | Yes | No dedicated CI guide found | Signals on agent turns | Apache-2.0 repo | Pro from $60/mo |
| Splunk Agent Observability (Galileo) | Galileo experiments | Not stated | Evaluators and runtime guardrails | No | Not stated |
| Agnost AI | No | No | Every whole conversation, pass/fail per message, no sampling | No | Starter $49/mo |
Braintrust
What it is. An evaluation and observability platform built around experiments, datasets, and scorers. You run your agent on a dataset, score the outputs, and compare experiments to measure improvements and regressions between runs.
Offline and CI. Braintrust's docs list running evals in CI/CD "to catch regressions automatically on every pull request." The braintrustdata/eval-action@v2 GitHub Action posts results as a comment on the pull request. In other CI systems you run bt eval directly; it returns a non-zero exit code if an eval throws.
Production. Online scoring rules run at span, trace, or group scope. Trace and group scopes wait 30 seconds after the last span by default. Braintrust suggests sampling 1-10% for high-volume apps.
Limits. Every score is metered, so scoring all production traffic gets expensive. A 30-second idle window is short for chat, where users pause between turns. Not open source; self-hosting is Enterprise.
Pricing. Starter is free with 1 GB, 10,000 scores, and 14-day retention. Pro is $249 a month with 5 GB and 50,000 scores, then $3 per GB and $1.50 per 1,000 scores. Users are unlimited.
LangSmith
What it is. LangChain's tracing and evaluation platform. Offline evaluation follows a dataset, evaluators, experiment, and comparison loop, with human, code, LLM-as-a-judge, and pairwise evaluators.
Offline and CI. The pytest integration maps each test file to a dataset and each run to an experiment, and raises assertion errors locally, for example in CI. The JS/TS SDK has a Vitest/Jest equivalent.
Production. Online evaluators run on production traces with filters and a sampling rate, and multi-turn evaluators judge whole threads once they go idle. You can cap each evaluator's weekly LLM spend.
Limits. Plus is priced per seat. Base traces are kept 14 days. Self-hosting is Enterprise only.
Pricing. Developer is free for 1 seat and 5,000 base traces a month. Plus is $39 per seat per month with 10,000 base traces. Enterprise is custom.
For a head-to-head, see Braintrust vs LangSmith.
Langfuse
What it is. An open-source platform for tracing, prompt management, datasets, experiments, and evaluations. MIT-licensed except for its ee folders, and free to self-host.
Offline and CI. You run experiments on Langfuse datasets from the SDK. For CI, the langfuse/experiment-action GitHub Action runs experiment scripts, can post results as a pull request comment, and fails the job when a script raises RegressionError, for example when average accuracy drops below a threshold. Outside GitHub Actions, you can run the same check with pytest or Vitest.
Production. LLM-as-a-judge evaluators target observations, the recommended target for live data, with filters and sampling on rules.
Limits. Trace-level evaluators are deprecated and stopped producing results on Langfuse Cloud after November 16, 2026. Sessions are not an evaluation target, so whole-conversation judging is on you. See what to do about Langfuse's trace-level evaluators.
Pricing. Self-hosting is free. Cloud Hobby is free for 50,000 units. Core is $29 a month, Pro $199, Enterprise $2,499, with $8 per 100,000 extra units.
Arize Phoenix and Arize AX
What they are. Phoenix is Arize's self-hostable tool for tracing, evaluation, datasets, and experiments, built on OpenTelemetry. AX is the managed platform.
Offline and CI. Phoenix scores traces and spans with LLM-based evaluators, code checks, or human labels, and reruns the same inputs through different versions to compare results. Arize's AX docs include GitHub Actions and GitLab CI/CD workflows for experiments, with a sample helper that exits with code 1 when the mean score is 0.7 or lower.
Production. AX online evals run LLM judges, code, agent-as-a-judge, or remote evaluators at span, trace, or session level. Arize recommends starting with 10-20% sampling.
Limits. Phoenix uses the Elastic License 2.0, which is source-available rather than OSI open source. Self-hosted AX is Enterprise only.
Pricing. Phoenix is free to self-host. AX Free covers 25,000 spans and 1 GB a month. AX Pro is $50 a month for 50,000 spans and 10 GB. Users and evals are unlimited on every AX tier.
Confident AI and DeepEval
What they are. DeepEval is an Apache-2.0 evaluation framework that works like pytest for LLM apps. It ships G-Eval (an LLM-as-a-judge metric for custom criteria), agent metrics such as task completion, tool correctness, and plan adherence, and multi-turn metrics such as knowledge retention. Confident AI is the hosted platform built on it.
Offline and CI. You write test files and run them with deepeval test run, which slots into any CI system. DeepEval's docs show it in GitHub Actions.
Production. Confident AI runs online evals server-side on traces or spans, with filters and a sample rate. Threads, including voice calls, use a separate workflow.
Limits. Confident AI's pricing page lists online evals from Starter ($200 a month) up, so production scoring isn't free. Live traffic can only use metrics that need no reference answer.
Pricing. DeepEval is free. Confident AI Free is $0 for up to 2 seats. Starter is $200 a month and Team $2,000 a month, both per organization with unlimited seats. Online evals start on Starter.
Ragas
What it is. An Apache-2.0 library for evaluating LLM applications, best known for RAG metrics. It offers LLM-based and traditional metrics, custom metrics, test data generation, and an experiments-first workflow.
Best for. Teams that want metrics and synthetic test sets in Python, inside their own harness.
Limits. It is a library, not a platform. We found no documented CI integration or production evaluation feature, so you bring your own runner, storage, and dashboards.
Pricing. Free and open source.
Promptfoo
What it is. An MIT-licensed CLI and library for testing and red-teaming LLM apps with declarative config files. Its GitHub README says "Promptfoo is now part of OpenAI" and that it "remains open source and MIT licensed."
Offline and CI. Evals run on your machine or CI runner and call the providers you configure. promptfoo eval exits with code 100 when a test fails or when the pass rate falls below PROMPTFOO_PASS_RATE_THRESHOLD, which defaults to 100%, so a failing eval fails the build. The promptfoo/promptfoo-action GitHub Action posts before-and-after comparisons on pull requests.
Limits. Promptfoo's docs describe testing and red teaming, not production monitoring, so pair it with an online eval tool.
Pricing. The open-source CLI is free. Promptfoo also sells an enterprise edition; its docs don't list a price.
Opik
What it is. Comet's open-source LLM evaluation platform. It is Apache-2.0 and the full platform is free to self-host.
Offline and CI. Opik has datasets, experiments, and LLM-as-a-judge metrics for hallucination, moderation, and RAG, plus "a PyTest integration to test LLM pipelines on every commit." An Agent Optimizer SDK tunes prompts and agents.
Production. Online evaluation rules run LLM judges at trace scope or thread scope, which scores a whole conversation after a cooldown (15 minutes by default). Sampling is configurable.
Limits. Pro Cloud allows up to 50 team members; beyond that you need Enterprise or self-hosting.
Pricing. Self-hosting is free. Free Cloud covers 25,000 spans a month. Pro Cloud is $19 a month for 100,000 spans, then $5 per 100,000.
W&B Weave
What it is. Weights & Biases' toolkit for tracing and evaluating agents. The repository is Apache-2.0.
Offline and CI. A Weave Evaluation bundles a dataset and scorers, including an LLMJudge class, and runs your agent over it with optional repeated trials. Results show in the W&B UI. We found no dedicated CI guide, though evaluations are code you can run from any CI job.
Production. Signals score agent turns automatically, for example for user frustration. Only turn scope is supported.
Limits. Billing is by ingested data, so verbose traces cost more. No whole-conversation scope for Signals.
Pricing. Free includes 1 GB a month of Weave ingestion. Pro starts at $60 a month with 1.5 GB, then $0.10 per MB.
Splunk Agent Observability (formerly Galileo)
What it is. Cisco completed its acquisition of Galileo on May 22, 2026, and galileo.ai now redirects to Splunk Agent Observability. Splunk describes out-of-the-box and custom evaluators (safety, compliance, RAG, text-to-SQL), tracing, token and cost tracking, and runtime guardrails that block PII leakage, tool misuse, and prompt injection. Evaluations and guardrails run on small models Splunk calls Eval-G1. Galileo's docs describe LLM-as-a-judge, Luna, and code metrics used in experiments and production log streams.
Best for. Enterprises that want evaluation and runtime guardrails in the Splunk Observability portfolio.
Limits. Splunk's product page doesn't mention datasets, CI, or pricing. Cisco says existing Galileo customers keep the current support process for now and will be told in advance of changes.
Pricing. Not stated; Splunk offers demos and trials.
Agnost AI
What it is. Agent monitoring for AI agents in production: chat agents, voice agents (through transcripts), copilots, and MCP servers. It is OpenTelemetry-native and works with the OpenAI Agents SDK, Vercel AI SDK, LangChain and LangGraph, Mastra, Pydantic AI, Google ADK, and Agno.
Production only. Agnost covers the online half of evaluation and nothing else. You write rules in plain language. Once a conversation has been idle for 15 minutes, an LLM judges the whole conversation in a batch and marks each assistant message pass or fail against every rule. There is no sampling. Agnost also groups traffic by user and session and flags user frustration, intents, tool-call errors, loops, and silent wrong answers. Alerts go to Slack or email. With Auto Improve, a fired alert opens a pull request with a fix tested against past runs: it replays the affected conversations and shows the results in the PR for a person to review.
Limits. No offline datasets, no experiments, and no CI gating, so pair it with one of the tools above. Verdicts are pass or fail, not scores. Judging runs in batches, not real time. Not open source.
Pricing. Free covers 1,000 events a month with 7-day retention. Starter is $49 a month (10,000 events). Pro is $499 a month (1,000,000 events). Enterprise is custom and includes self-hosted VPC deployment.
For a production-only comparison of every tool above, see best online evaluation tools for AI agents.
How to choose
| If you need… | Pick |
|---|---|
| One managed platform for experiments, CI, and production scoring | Braintrust or LangSmith |
| Open source and self-hosted, offline and online | Langfuse or Opik |
| A free CI gate with no platform | Promptfoo or DeepEval |
| RAG metrics and synthetic test data in Python | Ragas |
| Managed session-level evals with a free self-hosted workbench | Arize AX with Phoenix |
| Your ML team already lives in Weights & Biases | W&B Weave |
| Evaluation plus runtime guardrails inside Splunk | Splunk Agent Observability |
| Every production conversation judged against plain-language rules, with failures turned into PRs | Agnost AI, next to an offline tool |
Three checks before you commit:
- Decide which half you're buying for. A CI framework won't tell you what real users hit, and a production monitor won't block a bad prompt before merge. If you need both, check that the tools share a dataset format or can export to each other.
- Check the unit that gets judged. Span, trace, turn, and whole conversation catch different failures. A wrong answer three turns in only shows up at conversation level.
- Price judging at your volume. Scores, spans, GB, and events don't convert one-to-one, and many tools run judges on your own model key on top of the platform fee. How to set up online evals in production walks through the rollout.
Related reading
- What are online evals? Online vs offline evaluation for AI agents
- LLM evals in CI vs production: what each catches and which tools do both
- Best online evaluation tools for AI agents (2026)
- How to set up online evals for AI agents in production
- Turn production agent failures into regression tests
- Best AI agent monitoring tools (2026)
Frequently asked questions
What is the best tool to evaluate AI agents?
There is no single best tool, because agent evaluation has two halves. For offline evals (datasets, experiments, LLM-as-a-judge scorers and CI gates), Braintrust and LangSmith are the most complete managed platforms, Langfuse and Opik are the open-source platforms you can self-host, and DeepEval and Promptfoo are free open-source frameworks that run in CI. For online evals on production traffic, LangSmith, Arize AX, Braintrust, Opik and Confident AI score live traces with sampling, and Agnost AI judges every whole production conversation against plain-language rules. Most teams pair one offline tool with one production tool.
What is the difference between offline and online evals for AI agents?
Offline evals run your agent on a fixed dataset of test cases before you ship, usually in CI, and score the outputs, so you can compare a new prompt or model against the old one. Online evals score real production traffic after you ship, so you find failures your dataset didn't contain: new user intents, drift, and multi-turn breakdowns. Offline evals catch regressions on known cases; online evals find the unknown ones.
Which AI agent evaluation tools are open source?
DeepEval (Apache-2.0), Promptfoo (MIT), Ragas (Apache-2.0), Opik (Apache-2.0, full platform free to self-host), W&B Weave (Apache-2.0 repository) and Langfuse (MIT except its ee folders) are open source. Arize Phoenix is free to self-host under the Elastic License 2.0, which is source-available rather than OSI open source. Braintrust, LangSmith, Arize AX, Splunk Agent Observability and Agnost AI are commercial.
Which agent evaluation tools can block a pull request in CI?
Promptfoo exits with a non-zero code when a test fails or the pass rate falls below PROMPTFOO_PASS_RATE_THRESHOLD. DeepEval runs as a pytest-style suite with deepeval test run. Langfuse's experiment GitHub Action fails the job when a script raises RegressionError. Braintrust's eval-action posts experiment results on the pull request. LangSmith has pytest and Vitest/Jest integrations, Opik has a PyTest integration, and Arize AX documents GitHub Actions and GitLab CI workflows for experiments. Agnost AI does not do CI gating.
What is the best free tool to evaluate AI agents?
For CI and offline testing, DeepEval and Promptfoo are free and open source and run on your machine or CI runner, calling the model provider you configure. For a free platform with datasets, experiments, tracing and production evaluation rules, Opik and Langfuse are free to self-host. Arize Phoenix is free to self-host too. Most managed tools also have free tiers: Braintrust Starter, LangSmith Developer, Arize AX Free, Confident AI Free and Agnost AI Free.
Does Agnost AI run offline evals or CI tests?
No. Agnost AI covers the production half only. It judges every production conversation against plain-language rules once the conversation has been idle for 15 minutes, marks each assistant message pass or fail, and alerts in Slack or email. It has no offline datasets, no experiments and no CI gating, so teams pair it with an offline tool such as Braintrust, LangSmith, Langfuse, DeepEval or Promptfoo, and turn the production failures it finds into regression cases there.
What happened to Galileo?
Cisco completed its acquisition of Galileo Technologies on May 22, 2026, and Galileo is now Splunk Agent Observability. Splunk describes it as out-of-the-box and custom evaluators, tracing, token and cost tracking, and runtime guardrails that run on small models called Eval-G1. Cisco says existing Galileo customers keep using the existing support process for installed products.
What changed in AI agent evaluation tools since 2025?
Promptfoo is now part of OpenAI and says it remains open source and MIT-licensed. Galileo became Splunk Agent Observability after Cisco's acquisition closed in May 2026. Langfuse deprecated trace-level LLM-as-a-judge evaluators, which stopped producing results on Langfuse Cloud after November 16, 2026, in favour of observation-level evaluators. Whole-conversation (thread or session) evaluators also became common in LangSmith, Arize AX, Braintrust, Opik and Agnost AI.
Originally published on agnost.ai.



