Evidently AI

Editor’s note: This Evidently AI review is based on my hands-on test of the open-source Evidently library 0.7.23 on October 27, 2026: a fresh install, an eight-case LLM evaluation scored by two judge models, a 100,000-row drift report, and the self-hosted UI. I weighed cost, evaluation quality, setup effort, platform features, and alternatives.

Quick verdict: I recommend Evidently AI for Python teams that want one free, Apache-2.0 library for LLM output checks and classic ML drift reports, and are happy to host the dashboard themselves. The trade-off: there is no hosted SaaS anymore, no-code evals, alerts and RBAC sit in an unpriced commercial edition, and version 0.7.23 needs pandas<3 to import.

Among LLM eval tools, Evidently is unusual: 100+ metrics, more than 40 built-in descriptors (11 LLM-judge types among them) and statistical drift presets ship in one Python package. In this review, I’ll take a closer look at Evidently’s pricing, LLM evals, and drift monitoring, so you can see exactly whether it belongs in your stack.

Key Takeaways

• The library is free under Apache 2.0, with 8,015 GitHub stars and about 823,000 PyPI downloads last month.
• Its LLM judges caught both planted hallucinations in my test, each with a readable reasoning column.
• Evidently Cloud is no longer sold as a SaaS product, and the pricing page now redirects to the homepage.
• A clean pip install evidently pulled pandas 3.1.0, and the import crashed until I pinned pandas below 3.
• The open-source UI has no login, alerts or no-code evals.

Pros and Cons

How Much Does Evidently AI Cost?

The open-source library is free under Apache 2.0, and it is now the only Evidently you can start without a sales call. The docs say Evidently Cloud “is no longer available as a SaaS product,” and the pricing page now redirects to the homepage.

• Open source (Free): for Python teams that run the library, Tracely tracing and the self-hosted UI themselves. It includes unlimited projects and users, but no built-in login.
• Enterprise self-hosted (Custom): for private-cloud or on-premises deployments that need no-code evals, alerts, scheduled tasks, authentication and RBAC. It has no published price; the contact form is the only route.
• Former Cloud plans (Discontinued): until at least March 2026, Developer was free (10,000 rows/month, 3 projects, 2 seats) and Pro cost $80/month (100,000 rows, 10 projects, 5 seats, $10 per extra 10,000 rows). Neither plan can be bought now.

Option	Price	What you get	Who runs it
Open-source library	Free	Code-first evals, reports and tests	You
Self-hosted Evidently UI	Free	Dashboard, datasets, trace viewer	You
Enterprise self-hosted	Not published	No-code evals, alerts, auth and RBAC	Your organization
Cloud Developer (discontinued)	Formerly free	10,000 rows/month	No longer available
Cloud Pro (discontinued)	Formerly $80/month	100,000 rows/month	No longer available

Is Evidently AI Good Value for Money?

• Good fit: Python teams that already own CI pipelines and need LLM evals plus tabular drift checks. The software costs nothing until a judge model runs.
• Less convincing fit: teams that want a shared dashboard with logins. The free UI has none, so hosting and access control become your job.
• Budget priority: judge tokens. My eight-row run fired 24 judge calls (3 LLM descriptors x 8 rows) in 10.2 seconds with gpt-4o-mini, and the whole test cost a few cents. Calls scale with rows times LLM descriptors, so 10,000 rows with three judges means 30,000 calls.

Hosted rivals charge for the dashboard: Langfuse Core costs $29/month, LangSmith Plus $39 per seat/month, Confident AI Starter $200/month per organization and Braintrust Pro $249/month. Langfuse Hobby, Arize AX Free and Opik’s Free Cloud tier cost $0, though, so Evidently’s edge is control over your data and stack rather than a lower bill.

Reviewer’s Notes: I recommend starting with the library in notebooks or CI and paying nothing. Contact Evidently’s sales team only if you need RBAC and alerts in a private deployment. If what you actually want is a hosted dashboard with logins, pick Langfuse or Arize AX Free instead of standing up Evidently’s UI.

My Experience With Evidently AI

My first import evidently crashed before I had scored a single answer, and one dependency pin fixed it. After the pin, I ran an LLM eval, a 100,000-row drift report and the self-hosted UI on the same 0.7.23 install.

Installing Evidently 0.7.23

Evidently 0.7.23 requires Python 3.10 or later, so my Mac’s system Python 3.9 could not install it at all. I used a Python 3.12.16 virtual environment on Apple Silicon instead, which adds a step for anyone on an older system Python.

pip install evidently==0.7.23 resolved pandas 3.1.0, and the import then failed with ImportError: cannot import name 'infer_dtype_from_object' from 'pandas.core.dtypes.common'. The error matches open GitHub issue #1974. Running pip install "pandas<3" installed pandas 2.3.3, and the import worked.

Once installed, the package is broad: more than 40 descriptors, presets from TextEvals to DataDriftPreset, and CLI commands for ui, legacy_ui, demo_project, migrate and migrate-status. The 0.7 rewrite of April 2025 replaced column mapping with DataDefinition, so pre-2025 notebooks need a rewrite or a pin at 0.6.7.

Reviewer’s Notes: Pin pandas below 3 in your requirements before you add Evidently 0.7.23 to CI. The crash happens at import, not inside an optional feature, so a one-line import evidently smoke test catches it before a pipeline does.

Building My First LLM Eval

I built an eight-row support-bot dataset with question, context, answer and target columns:

• 4 correct answers.
• 2 planted hallucinations.
• 1 correct refusal to reveal another customer’s address.
• 1 wrong refusal to a pricing question the context answered.

Everything hangs off Dataset.from_pandas, a DataDefinition and a list of descriptors. This trimmed version shows the faithfulness and refusal judges:

dataset = Dataset.from_pandas(
    df,
    data_definition=DataDefinition(text_columns=["question", "context", "answer", "target"]),
    descriptors=[
        FaithfulnessLLMEval("answer", context="context"),
        DeclineLLMEval("answer"),
    ],
)

My full run added CorrectnessLLMEval against the target column, plus Sentiment, IncludesWords and TextLength with a 200-character test. The default gpt-4o-mini judge took 10.2 seconds for eight rows and 24 judge calls.

Reading the Report

Report([TextEvals()], include_tests=True) produced 26 metrics and 8 tests, with 0 failed, even though two answers were hallucinations. The only test I attached was the 200-character length limit, and every answer passed it.

Each LLM descriptor added a category column (FAITHFUL, UNFAITHFUL, DECLINE) plus a plain-English reasoning column, and the reasoning explained the wrong refusal better than any pass count. I saved the report as a single HTML file. Zero failed tests is not zero bad answers: attach tests to the judge categories that should block a release.

Opening the Self-Hosted UI

Workspace.create("workspace"), then create_project and add_run, stored my drift snapshot locally. evidently ui --workspace ./workspace --port 8765 started the service; its API listed my project and reported version 0.7.23.

There was no login screen. The console also printed “fatal: not a git repository” on every request, The API calls still succeeded, but it is the same Git dependency behind GitHub issue #1035, open since 2024, which reports the dashboard failing to update when the project has no Git repository.

LLM Evals: What Evidently Caught and What It Missed

Faithfulness and correctness both flagged both planted hallucinations: a 60-day refund window where the context said 30 days, and an invented 1-800 phone line where support was email-only. For the refund, gpt-4o-mini’s reasoning column said the answer “contradicts the SOURCE” and named the 30-day limit.

Case	Faithfulness (gpt-4o-mini)	Correctness (gpt-4o-mini)	Decline (gpt-4o-mini)	Correctness (gpt-5-mini)
Invented 60-day refund	UNFAITHFUL	INCORRECT	OK	INCORRECT
Invented phone line	UNFAITHFUL	INCORRECT	OK	INCORRECT
Correct privacy refusal	FAITHFUL	CORRECT	DECLINE	CORRECT
Wrong refusal to pricing question	FAITHFUL	UNKNOWN	DECLINE	INCORRECT
Correct Canada shipping answer	FAITHFUL	CORRECT	OK	INCORRECT

The wrong refusal, “I’m not able to help with that.” to a question the context answered, contradicts nothing, so faithfulness passed it as FAITHFUL and gpt-4o-mini’s correctness judge returned UNKNOWN. Only DeclineLLMEval caught it.

Decline also flagged the justified privacy refusal, so that label alone cannot tell a good refusal from a bad one. I recommend stacking Decline with correctness against a full reference answer.

The cheap checks did worse. IncludesWords with “sorry”, “unable” and “can’t” missed the wrong refusal, because “not able” is not in the list. Sentiment scored the invented phone line and the correct 30-day answer at 0.0 each, so it is no quality signal here.

Judge Choice Changed the Verdict

Swapping the judge to gpt-5-mini, with no other code change, took 21.8 seconds, about 2x the default run. It marked the wrong refusal INCORRECT, an improvement over UNKNOWN.

It also failed a correct answer. My reference read “Yes, 5-8 business days,” the answer also named Canada, and gpt-5-mini called that a new claim not present in the reference. A stricter judge produces false negatives when references are terse, so write full reference answers and read the reasoning column before you trust an aggregate score.

Checks Beyond LLM Judges

Evidently’s built-in descriptors span five families:

• Deterministic: IncludesWords, RegExp and ExactMatch for vocabulary and pattern checks.
• Syntax: IsValidJSON, IsValidSQL and IsValidPython for structured output.
• Text stats: TextLength and SentenceCount for length limits.
• LLM judges: FaithfulnessLLMEval, PIILLMEval and ToxicityLLMEval, among others.
• ML models: BERTScore, SemanticSimilarity and HuggingFace classifiers.

Custom judges use LLMEval with BinaryClassificationPromptTemplate, and litellm routes them to 50+ providers, including Anthropic, Gemini, Mistral and Ollama. The docs’ examples still cite older model ids such as gpt-3.5-turbo, so expect to update them. Prompt optimization (0.7.10, July 2025) tunes a judge prompt against labeled examples. The docs also ship a GitHub Actions example, where tests with eq, gte or lte conditions fail the CI job when an output misses its threshold.

For RAG-only metrics, Ragas is the narrower open-source library. For pytest-style unit tests on LLM outputs, DeepEval is the closer fit.

Data Drift and Classic ML Monitoring

DataDriftPreset plus DataSummaryPreset compared two 50,000-row datasets in 1.3 seconds and flagged exactly the two columns I shifted. Report.run takes the current dataset first and the reference second, then picks the statistic per column type on its own: normed Wasserstein distance for the numeric basket_value, Jensen-Shannon for the categorical channel. That speed and precision is what eval-only tools cannot offer teams that run tabular models next to LLM features.

Column	Planted change	Result
basket_value	Mean +20%	Normed Wasserstein 0.280 vs 0.1 threshold: drifted
channel	Web 50% to 35%, app 30% to 50%	Jensen-Shannon 0.147: drifted
items	Unchanged	0.0: not drifted
latency_ms	4% missing values	0.001: not drifted; missing values caught by summary tests

With half the columns drifted (share 0.5), the dataset-level drift test failed with “Less 0.500”. The summary tests caught the latency gaps separately, which is the right split: missing data is a quality problem, not drift.

The noise came from the auto tests. include_tests=True generated 43 tests from the reference with zero configuration, and 13 failed. The +20% basket_value shift alone turned its min (0.286 +/- 10%), max and every quantile red, about 8 failures from one change.

I recommend the auto tests for a first look only. For this dataset, the conditions worth keeping before anything reaches an alert are:

• Drift share: the dataset-level test that failed at 0.5, plus the per-column drift on basket_value and channel.
• Missing values: an explicit tolerance for latency_ms instead of the auto threshold.
• Range and quantile tests: cut or widen them, since one shift fails about eight of them at once.

Beyond drift, ClassificationPreset, RegressionPreset and ranking/RecSys metrics cover model quality; I ran only the drift and summary presets. Open GitHub issue #1929 reports chi_stat_test and g_test returning NaN p-values on single-category columns, so check categorical features that hold one value.

This is where Evidently beats DeepEval, Ragas, Promptfoo and Braintrust, none of which do tabular drift. Arize is the closer rival if you want ML and LLM monitoring as a hosted platform.

The Self-Hosted Platform, Tracing and What’s Left Out

Evidently’s own edition table puts tracing, a trace viewer, a monitoring dashboard and dataset management in the free platform, but authentication, alerts and no-code evals are commercial-only.

• In the free edition: Tracely tracing and trace viewer, reports and test suites, monitoring dashboard, custom metrics, dataset management, prompt CMS, plus synthetic data generation and prompt optimization through the API.
• Commercial edition only: no-code evals, no-code data generation, no-code dashboards, side-by-side comparison, alerts, scheduled tasks, authentication and RBAC.

Tracely is OpenTelemetry-based and Apache 2.0, with timeline, dataset and dialogue views. I tested report snapshots in the UI, not tracing. The tracing quickstart still walks through Evidently Cloud signup and CloudWorkspace, a dead path now, so follow the self-hosting guide instead.

Workspaces live on the local filesystem, in SQL (SQLite or Postgres) or in S3-compatible storage, and evidently ui serves on port 8000 by default (--demo-projects all loads sample projects). A RemoteWorkspace with an optional secret lets scripts on other machines push runs. Because my UI had no login, put any shared deployment behind your own authenticated proxy or VPN.

Synthetic data generation arrived as an API in 0.7.11 (July 2025). Adversarial test generation (harmful content, forbidden topics, prompt leakage, hijacking) and alerts to Email, Slack or Discord sit in the commercial UI.

Long-open GitHub requests include a configurable base path for the self-hosted UI (#1363), and #1910 reports the UI service breaking with litestar 3.0. Open-source support is Discord and docs, and the docs warn that production users “must be comfortable troubleshooting and resolving issues on your own.” If you want logins without running a server, Langfuse Hobby and Comet’s Opik Free Cloud give you hosted dashboards with team access at $0.

What Happened to Evidently Cloud?

Evidently’s docs now state that Cloud “is no longer available as a SaaS product” and point users to the self-hosted platform or the library. The pricing page redirects to the homepage, app.evidently.cloud no longer resolves, and the homepage has dropped its Pricing and Log in links.

No shutdown date is published. The Developer and Pro plans were on sale until at least March 2026, nearly a year after Cloud v2 (April 2025) forced users onto evidently 0.7 or later.

The library still ships, but less often:

• 2025: about every two weeks, from 0.7.1 in April to 0.7.18 in December.
• January 2026: 0.7.19 and 0.7.20.
• March 2026: 0.7.21.
• September 11, 2026: 0.7.23, whose release notes only change the PyPI publishing workflow.

The repository was last pushed on September 29, 2026, and has about 97 contributors and 350 open issues and pull requests. Those include the pandas 3 break (#1974), which I reproduced, and 0.7 memory growth on Kubernetes (#1851), which my short local test could not.

Evidently AI was founded by Elena Samuylova (CEO) and Emeli Dral (CTO) and went through Y Combinator in Summer 2021. My read: the library is alive on a slower cadence, so plan to own your upgrades and version pins.

How Does Evidently AI Compare to Competitors?

Evidently wins when LLM evals and tabular drift belong in one Python workflow. Each rival below wins a narrower job, usually by hosting the dashboard for you.

• Langfuse: my pick for hosted tracing and evals with logins. Hobby is free with 50,000 units/month, 30-day retention and 2 users; Core costs $29/month for 100,000 units and unlimited users.
• Confident AI / DeepEval: the fit for pytest-style LLM unit tests in CI. The hosted Free plan gives 2 seats, 1 project and 5 test runs/week; Starter costs $200/month per organization.
• Arize AX / Phoenix: the closest rival for ML plus LLM observability. Phoenix is open source and local-first; AX Free includes 25,000 spans, and AX Pro costs $50/month for 50,000 spans.
• Braintrust: built for eval-driven product teams that live in experiments. Starter is free with 10,000 scores and 14-day retention; Pro costs $249/month for 50,000 scores and 30-day retention.
• Promptfoo: the CLI for red teaming and prompt regression in CI. Community is free with up to 10,000 red-team probes/month; Enterprise and On-Premise are custom-priced.
• Opik (Comet): open-source tracing and evals with a hosted option. Free Cloud covers 25,000 spans/month for up to 10 members; Pro costs $19/month for 100,000 spans, plus $5 per extra 100,000.
• Ragas: a free, open-source library for teams that only evaluate RAG. Compare its metrics with Evidently’s FaithfulnessLLMEval and ContextRelevance before adopting the broader library.

Tool	Open source	Hosted option	Starting paid price	Tabular drift
Evidently	Yes	No (SaaS discontinued)	Enterprise, unpublished	Yes
Langfuse	Yes	Yes	$29/month	No
DeepEval / Confident AI	DeepEval library	Yes	$200/month per org	No
Arize AX / Phoenix	Phoenix	AX	$50/month	Yes
Braintrust	No	Yes	$249/month	No
Opik (Comet)	Yes	Yes	$19/month	No
Promptfoo	Community CLI	Enterprise	Custom	No
Ragas	Yes	No	Free library	No

These allowances measure different things: Langfuse units are not Braintrust scores or Arize spans. Estimate your own trace and eval volume before ranking the hosted platforms by headline price.

How I Reviewed Evidently AI

I installed Evidently 0.7.23 in a Python 3.12.16 virtual environment on Apple Silicon macOS. I ran an eight-case support-bot evaluation judged by gpt-4o-mini and gpt-5-mini, generated a drift and summary report on two 50,000-row synthetic datasets with planted shifts, and loaded the result into the self-hosted UI.

I weighed cost, eval quality, setup effort, platform features and alternatives. Eight cases expose specific failure modes, not a judge-accuracy rate, and 1.3 seconds is a local timing, not a throughput guarantee.

I tested only Evidently, not the Enterprise edition or any rival. Evidently Cloud could not be tested because it is no longer sold.

Prices current as of October 2026.

Should You Use Evidently AI to Evaluate Your LLM App?

Yes, if you are a Python team running both tabular models and LLM features. Its judges caught both planted hallucinations and its drift report flagged exactly the two shifted columns in 1.3 seconds, all for $0 plus a few cents of judge tokens.

• Best for: ML and AI engineers who want code-first evals, drift reports in CI, and a dashboard they can host themselves.
• Skip if: you need a managed UI with logins and alerts. Start with Langfuse ($29/month Core) or Arize AX ($50/month Pro) instead.
• Choose DeepEval instead if: pytest-style LLM unit tests are the whole job and tabular drift is not.

If you adopt it, pin pandas<3 for 0.7.23, stack Faithfulness with Decline and Correctness judges, and write full reference answers. Then run a small labeled dataset with known failures and confirm your tests actually reject them before Evidently gates a release.

FAQs

Is Evidently AI free?

Yes. The Evidently library and its self-hosted platform are free under Apache 2.0, with unlimited projects and users. You pay for your own hosting and for judge-model tokens; my eight-row test cost a few cents. Evidently Cloud is discontinued as SaaS, and the commercial Enterprise self-hosted edition has no published price.

Is Evidently Cloud still available?

No. Evidently’s docs say Cloud is no longer available as a SaaS product, the pricing page redirects to the homepage, and app.evidently.cloud no longer resolves. The former Developer (free) and Pro ($80/month) plans existed until at least March 2026. Today you run the library locally or self-host the open-source platform.

Which LLMs can Evidently use as a judge?

OpenAI’s gpt-4o-mini is the default, and litellm adds other providers, including Anthropic, Gemini, Mistral and Ollama. I ran the same eval with gpt-4o-mini and gpt-5-mini: gpt-5-mini took 21.8 seconds instead of 10.2 and failed a correct answer, so re-check labeled examples whenever you switch judges.

Does Evidently work with pandas 3?

Not in version 0.7.23. My fresh install pulled pandas 3.1.0, and import evidently failed because infer_dtype_from_object no longer exists in pandas. Running pip install "pandas<3" installed pandas 2.3.3 and fixed it. GitHub issue #1974 tracks the break, so pin pandas until a release closes it.

Is Evidently good for RAG evaluation?

Yes, for teams that also monitor other models. Evidently ships FaithfulnessLLMEval, ContextRelevance, ContextQualityLLMEval and CompletenessLLMEval, and faithfulness caught both hallucinations in my test. Ragas is the narrower open-source alternative if RAG is all you evaluate. Either way, write full references: terse ones made gpt-5-mini reject a correct answer.