Confident AI (DeepEval)

Editor’s note: This Confident AI review is based on my hands-on test of Confident AI’s Free plan and the open-source DeepEval framework (version 4.2.8) in October 2026. It weighs pricing and plan limits, setup, metric coverage, production monitoring, team workflows and rivals.

Quick verdict: I recommend Confident AI for teams that already write DeepEval tests and want the same metrics scoring production traces, with alerts and human review. Starter is a flat $200/month for unlimited seats, but SOC 2, SSO and custom RBAC sit on Team at $2,000/month.

Confident AI is the hosted platform behind DeepEval, the Apache-2.0 LLM evaluation framework with 18,800+ GitHub stars and 50+ metrics, and it runs those metrics on CI test runs and live production traces.

In this review, I’ll take a closer look at Confident AI’s pricing, features, and production monitoring, so you can see exactly whether the cloud is worth paying for on top of free DeepEval.

Key Takeaways

• DeepEval is free and runs locally with no account.
• Free plan: 2 seats, 1 project, 5 test runs a week, 1 GB-month of traces and 1-week retention.
• Starter is $200/month flat with unlimited seats, online evals and alerts; tracing overage is $1 per GB-month.
• Compliance costs 10x more: SOC 2, SSO and custom RBAC need Team at $2,000/month; on-prem and HIPAA need Enterprise.
• Judge costs and variance remain: LLM-as-judge metrics bill tokens on your own key, and scores near a threshold can flip between runs.

Pros and Cons

How Much Does Confident AI Cost?

Confident AI sells four plans and bills Starter and Team per organization, not per seat: Free $0, Starter $200/month, Team $2,000/month and Enterprise at a custom price. A 20-person team pays the same $200 for Starter as a 2-person team.

• Free ($0): for a developer wiring DeepEval to cloud reports. 2 seats, 1 project, evals in dev and CI/CD, tracing, prompt versioning and cloud datasets. After 5 test runs a week, further runs lock; spans beyond 1 GB-month are dropped, and data is kept for 1 week.
• Starter ($200/month): for teams scoring production traffic. Unlimited seats, 5 projects and 5 GB-months of trace data, then $1 per GB-month ingested or retained. Adds no-code eval workflows, custom metrics, online evals and classifications on live traffic, annotation queues with custom annotation criteria, auto-curating datasets from traces, 15+ multi-turn metrics, chat simulations, real-time alerting and full project API access.
• Team ($2,000/month): for companies facing a security review. Unlimited projects and 75 GB-months included, then $1 per GB-month. Adds custom annotation forms, metric and dataset versioning, scheduled dataset runs, Git-based prompt branching and approval workflows, custom RBAC, SOC 2, SSO, a dedicated support channel and NET-30 invoice billing.
• Enterprise (custom): for on-prem or regulated deployments. Adds on-prem deployment, custom data residency (Canada, Australia, Japan), HIPAA, an organization management API, infosec review, 24×7 support and unlimited online eval metric runs; the red teaming and governance modules are Enterprise ++ add-ons.

Plan	Price	Seats	Projects	Trace data included	Retention	Key additions
Free	$0	2	1	1 GB-month	1 week	CI/CD evals, tracing, cloud datasets
Starter	$200/month	Unlimited	5	5 GB-months, then $1/GB-month	Unlimited	Online evals, custom metrics, alerts, annotation queues
Team	$2,000/month	Unlimited	Unlimited	75 GB-months, then $1/GB-month	Unlimited	SOC 2, SSO, RBAC, versioning, custom annotation forms
Enterprise	Custom	Unlimited	Unlimited	Unlimited	Unlimited	On-prem, HIPAA, custom data residency

Free needs no credit card. Annual plans carry a discount Confident AI does not publish, and startups that raised under $5M can apply for a grant. For scale, Braintrust Pro is $249/month with 5 GB and 30-day retention, and LangSmith Plus is $39 per seat.

Is Confident AI Good Value for Money?

• Flat pricing wins from 6 seats: LangSmith Plus at $39 per seat costs $234/month for 6 people, already above Starter’s $200 before usage.
• Tracing overage is cheap: $1 per GB-month against Braintrust Pro’s $3/GB. Langfuse bills $8 per 100k units, a different meter, so compare on your real trace volume.
• The real price is $2,000: any company whose procurement asks for SOC 2 or SSO lands on Team, a 10x jump from Starter.
• Judge tokens are extra: OpenAI, Anthropic or Gemini bill every judged metric on your own key, on every plan.

Reviewer’s Notes: Start on free DeepEval plus the Free plan, and move to Starter only when you need online evals on production traffic. Budget for Team only if procurement requires SOC 2, SSO or custom RBAC; for everything else, including annotation queues for domain experts, Starter covers the job.

My Experience With Confident AI

Confident AI’s setup is four steps for anyone who writes pytest: install, log in, write a test file, run one CLI command. I started on the Free plan with DeepEval 4.2.8.

Creating the Account and Picking a Region

I created a free account at app.confident-ai.com without entering a credit card. Confident AI bills at the organization level and isolates test runs, metrics, datasets and traces per project; it recommends one project per LLM use case, and Free allows 1 project.

Data lands in the United States region by default. The EU region is available on every plan, including Free, and I pointed both SDKs at it with two variables:

• Evaluation API: CONFIDENT_BASE_URL=https://eu.api.confident-ai.com
• Tracing endpoint: CONFIDENT_OTEL_ENDPOINT=https://eu.otel.confident-ai.com/v1/traces

Writing My First DeepEval Test

I installed the framework with pip install -U deepeval and connected it with deepeval login; in CI, the CONFIDENT_API_KEY variable does the same job. My first test used an LLMTestCase with an input and actual output, an AnswerRelevancyMetric with a threshold, and assert_test:

from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric

def test_refund_answer():
    test_case = LLMTestCase(
        input="What is your refund window?",
        actual_output="You can request a refund within 30 days of purchase.",
    )
    metric = AnswerRelevancyMetric(threshold=0.7)
    assert_test(test_case, [metric])

RAG tests add expected_output and retrieval_context to the same test case. I ran the file with deepeval test run test_file.py, and the results synced to Confident AI as a shareable test report. Three flags matter on larger suites:

• -n: runs tests in parallel processes.
• -c: reads results from the local cache.
• -r: repeats each test case, the quickest way to expose judge variance before a threshold goes into CI.

The judge runs on my own key: the OpenAI default is gpt-5.4, and Ollama, Azure OpenAI, Gemini, Anthropic or a custom DeepEvalBaseLLM wrapper also work. Two variables belong in any CI setup: DEEPEVAL_TELEMETRY_OPT_OUT=1 is the telemetry opt-out, and DEEPEVAL_DISABLE_DOTENV=1 stops DeepEval from auto-loading .env.local and .env at import.

Adding Tracing to an App

Tracing ships as a separate package. I installed confident-trace (Python 3.10+) and wrapped functions with the @span decorator, which reads the same CONFIDENT_API_KEY.

Integrations for OpenAI, LangChain, LangGraph, LlamaIndex, CrewAI, Pydantic AI and the Vercel AI SDK, or plain OpenTelemetry, replace the decorator where a framework already emits spans. CONFIDENT_SAMPLE_RATE (0 to 1, default 1.0) decides per trace, so each trace is exported whole or not at all.

Reviewer’s Notes: Installation is the easy part; the friction is choosing metrics and a judge model. Promptfoo is YAML and CLI-first, while DeepEval is Python test-first, which suits teams whose app code already lives in Python. On Free, the 5-runs-a-week cap bites the moment tests go into CI.

DeepEval Metrics: RAG, Agents and Multi-Turn Conversations

DeepEval ships 50+ metrics, and the platform lists 30+ single-turn and 15+ multi-turn of them, with agentic and conversational families on top of the standard RAG set:

• RAG: answer relevancy, faithfulness, contextual precision, contextual recall and contextual relevancy.
• Agentic: task completion, tool correctness, argument correctness, step efficiency, plan adherence and plan quality.
• Multi-turn: turn relevancy, role adherence, knowledge retention, conversation completeness and goal accuracy.
• Safety: bias, toxicity, PII leakage, misuse, non-advice and role violation.
• Multimodal and voice: 5 image metrics and 8 voice metrics, including speech intelligibility and turn-taking naturalness.
• MCP: MCP-Use, Multi-Turn MCP-Use and MCP Task Completion for agents that call MCP servers.
• Custom: G-Eval for plain-English criteria, DAG decision-tree metrics and Arena G-Eval for side-by-side comparisons.

The Synthesizer generates goldens from documents, and the Conversation Simulator drives multi-turn conversations with personas, including a voice mode. Ragas covers RAG metrics as an Apache-2.0 library too, but it has no hosted platform, so it cannot score live traffic.

Every judged metric is LLM calls on your own key. DeepEval 4.2.2 added Jev, TypeSafe AI’s model for closed-ended decisions; Confident AI says it costs $0.042 per million input tokens with no output charge, while language tasks stay with the evaluation LLM.

Users report unchanged test cases landing on both sides of a 0.6 threshold across re-runs, and DeepEval 4.2.0 standardized scores so higher is always better, which flipped the direction of some metrics between versions.

Reviewer’s Notes: Re-run borderline cases with -r before a metric becomes a blocking CI gate. A 0.6 threshold with scores hovering around it will fail builds that nothing changed, so set thresholds with a margin or gate on repeated runs.

Tracing, Online Evals and Alerts in Production

Online evals (Starter and up) run metric collections server-side on live traces or spans, with optional filters and a sample rate. That is the clearest reason to pay for Confident AI instead of staying on local DeepEval tests.

1. Build a metric collection of single-turn metrics.
2. Create an evaluation task for Traces or Spans in Workflows, with filters and a sample rate.
3. Set test-case parameters (input, output, retrieval context) on the span or trace in code.

Threads, the multi-turn case, use a separate flow. The task’s sample rate is separate from the trace-export CONFIDENT_SAMPLE_RATE, so you can export every trace and score only a fraction of them. Scores then feed these monitoring tools:

• Drift: anomaly and regression detection.
• Alerts: threshold rules on traces, spans and threads, sent through notification integrations set in Project Settings.
• Cost: token usage and LLM cost tracking.
• Data controls: masking, plus multimodal tracing for images, PDFs and audio.

Tracing plugs into 25+ frameworks and gateways beyond the ones above:

• Gateways: LiteLLM, OpenRouter and Portkey.
• Agent SDKs: OpenAI Agents, Claude Agent SDK and Google ADK.
• Other frameworks: Mastra, LiveKit and AWS Bedrock AgentCore.
• OpenTelemetry: Python, TypeScript, Go, Ruby and C#.

Tracing costs $1 per GB-month past the included allowance, and Confident AI puts roughly 100K traces in 1 GB, so Starter’s 5 GB-months hold about 500K traces before overage. It has also announced Confident Monitor, which it says classifies and grades traces as they arrive and ranks failures as issues. For tracing without DeepEval metrics, Langfuse self-hosts for free and Arize AX Pro includes 10 GB for $50/month.

Datasets, Annotation Queues and Red Teaming

Every plan includes cloud datasets of goldens, synthetic data generation through data-source connectors, and experiments that compare test runs for regressions; metric and dataset versioning start on Team. Starter adds auto-curating datasets from traces, so production failures become regression cases.

Human Review and Annotation

Annotation queues start on Starter at $200/month: they group traces, spans, threads or test cases for reviewers, who fill in a form with these fields (custom annotation forms need Team):

• Ratings: thumbs, 1 to 5 stars, choices, yes/no or a number.
• Feedback: free text, an explanation and an optional expected output.

Annotations feed eval alignment and error analysis, and the expected-output field lets a domain expert correct an answer instead of only flagging it.

Red Teaming With DeepTeam

DeepTeam, Confident AI’s open-source red teaming framework (Apache-2.0, 3,000+ GitHub stars), claims 50+ vulnerabilities, from bias and PII leakage to BOLA, BFLA and RBAC flaws. Its attacks come in two sets:

• 14 single-turn enhancements: including Base64, ROT-13, roleplay, prompt injection and multilingual.
• 5 multi-turn progressions: including Crescendo and Tree Jailbreaking.

The no-code red teaming module, with OWASP Top 10 for LLMs and NIST AI RMF frameworks, is an Enterprise add-on. Promptfoo’s free Community edition includes 10k red-team probes a month, the cheaper route for CLI-driven red teaming.

Is Open-Source DeepEval Enough on Its Own?

DeepEval runs fully without an account: metrics, pytest integration, the Synthesizer, the Conversation Simulator, local caching and SQLite storage through DEEPEVAL_LOCAL_STORE=sqlite. Version 4.2.8 shipped on October 2, 2026, the last of six releases since September 14, under Apache-2.0 with 18,800+ GitHub stars.

The cloud adds shareable reports, test-run history with regression comparison, online evals, alerts, annotation queues and prompt management. Free keeps that history for 1 week; Starter and up keep test runs and traces with no time limit. None of it touches DeepEval’s core metrics, which stay free on every plan.

Users report three recurring problems in the framework itself:

• Async timeouts: evaluate() batches are cut off after about 207 seconds by default.
• Long runs: request timeouts when several metrics run together, and async test runs that never complete.
• Upgrades: one release broke Azure deployment lookup, and 4.2.0 changed score direction.

Pin the DeepEval version in CI and re-run a sample of cases before each upgrade, since a change like 4.2.0’s score direction can invert thresholds set on older versions.

Best for: Python teams running offline regression suites with no one outside engineering reviewing outputs. Skip the cloud if: nothing needs scoring in production and nobody needs shared reports.

How Does Confident AI Compare to Competitors?

Confident AI wins on metric depth and flat seat pricing; rivals win on self-hosting, LangChain fit or cheaper hosted observability.

• Braintrust: best for a hosted eval platform with a generous free tier for big teams. Starter is free with unlimited users, 1 GB, 10K scores a month and 14-day retention; Pro is $249/month with 5 GB and 30-day retention, and Pro overage runs $3/GB against Confident AI’s $1. Starter has no RBAC or SSO.
• LangSmith: best inside LangChain and LangGraph. Developer is free for 1 seat and 5k base traces a month; Plus is $39 per seat a month with 10k base traces, and base traces are kept 14 days, with 180-day retention at extra cost. At 6 seats, Plus already costs more than Starter.
• Langfuse: best when self-hosting is mandatory, because the self-hosted version is free. Its Hobby plan is free for 2 users and 50k units a month; hosted Core is $29/month (100k units, 90 days) and Pro is $199/month, with overage at $8 per 100k units.
• Arize AX and Phoenix: best for cheap hosted observability. AX Free covers 25k spans; AX Pro is $50/month with 10 GB and 30-day retention. Phoenix is open source and local-first, while self-hosted AX is Enterprise only and overage rates are not published.
• Galileo, now Splunk Agent Observability: best for buyers who want Luna evaluator models. It starts at $100/month per 1.2 million spans, billed annually, with 500 million Luna tokens and 5 Luna fine-tuning runs a year, in the cloud or self-hosted.
• Promptfoo: best for CLI and YAML-first evals and red teaming on a zero budget. The MIT-licensed Community edition includes all eval features and 10k red-team probes a month; team features, SSO and continuous monitoring are Enterprise only, with no self-serve paid tier.

Tool	Free option	Entry paid price	Billing basis	Best for
Confident AI	Free plan (2 seats) plus Apache-2.0 DeepEval	$200/month	Per organization plus GB-months	DeepEval teams scoring production
Braintrust	Starter, unlimited users	$249/month	Plan plus data and scores	Hosted evals for big teams
LangSmith	Developer, 1 seat	$39/seat/month	Per seat plus traces	LangChain and LangGraph apps
Langfuse	Hobby plan plus free self-hosting	$29/month	Plan plus units	Mandatory self-hosting
Arize AX / Phoenix	AX Free plus open-source Phoenix	$50/month (AX Pro)	Plan plus spans and GB	Cheap hosted observability
Galileo / Splunk	Annual paid plans	$100/month, billed annually	Per 1.2 million spans	Luna evaluator models
Promptfoo	Community edition, MIT	Custom (Enterprise)	Enterprise contract	CLI evals and red teaming

Quick comparison: pick Confident AI for DeepEval-to-production continuity, Langfuse for free self-hosting, LangSmith for a LangChain stack, Arize AX for observability under $200 a month, and Promptfoo for free CLI red teaming.

How I Reviewed Confident AI

I set up Confident AI’s Free plan and DeepEval 4.2.8 myself, wrote pytest-style tests, connected the confident-trace SDK to an app, and compared every plan limit against Braintrust, LangSmith, Langfuse, Arize, Galileo and Promptfoo. I weighed six areas:

• Pricing: organization-level fees, trace allowances, retention and upgrade gates.
• Setup: account, region, Python tests and tracing configuration.
• Metrics: RAG, agentic, conversational and custom coverage, plus judge-model choice.
• Production: online evals, sampling, drift, alerts and tracing integrations.
• Team and security: annotation, versioning, RBAC, SOC 2 and SSO.
• Rivals: price, free tier and best-fit buyer for each alternative above.

User-reported timeouts and judge variance count as risks in this review, not results from my own runs. I assigned no numerical score; the verdict below names who should buy.

Prices current as of October 2026.

Should You Use Confident AI for LLM Evaluation?

I recommend Confident AI for teams already on DeepEval that run customer-facing agents or RAG apps and need the same metrics scoring production traffic. Starter at $200/month is the right paid tier for them: online evals, alerts and unlimited seats for less than six LangSmith Plus seats cost. That fits product teams shipping RAG chatbots or tool-calling agents where a bad answer reaches a customer, and where domain experts need to review outputs next to engineers.

Skip it if you need self-hosting below Enterprise (Langfuse self-hosts for free), if your stack is LangChain end to end (LangSmith fits better), or if a security review demands SOC 2 on a small budget, since SOC 2 starts at $2,000/month.

Start with free DeepEval and the Free plan, wire one CI suite with -r on borderline cases, and upgrade to Starter once you need online evals on live traffic.

FAQs

Is Confident AI free?

Yes, in part. The Free plan costs $0 with 2 seats, 1 project, 5 test runs a week, 1 GB-month of traces and 1-week retention, and it needs no credit card. DeepEval itself is free under Apache-2.0 and runs without an account. Your judge model provider still bills every LLM-as-judge metric.

What is the difference between DeepEval and Confident AI?

DeepEval is the open-source framework that runs metrics and pytest-style tests on your machine. Confident AI is the optional hosted platform from the same company: shareable reports, test-run history, online evals on production traces, alerts, annotation and prompt management. DeepEval’s metrics stay free either way.

How much does Confident AI cost?

Free is $0, Starter is $200/month, Team is $2,000/month and Enterprise is custom. Starter and Team are flat per organization with unlimited seats. Tracing beyond the included 5 GB-months (Starter) or 75 GB-months (Team) costs $1 per GB-month, plus judge-model tokens from your LLM provider.

Can I use DeepEval without OpenAI?

Yes. DeepEval accepts Anthropic, Gemini, Azure OpenAI and Ollama as judge models, plus any custom model wrapped in DeepEvalBaseLLM. Ollama and Azure have no default model, so you must set a model name. A different judge produces different scores, so pick it before you set CI thresholds.

Is Confident AI SOC 2 compliant?

Yes. Confident AI is SOC 2 Type II certified, and SOC 2 is included from the Team plan at $2,000/month. HIPAA is listed on Enterprise, and the EU data region is available on every plan, including Free. Data is encrypted at rest and protected by TLS in transit.