Ragas

Editor’s note: This Ragas review is based on my hands-on test of Ragas 0.4.3 on October 14, 2026: a fresh pip install, a five-case RAG evaluation scored with two judge models, a synthetic testset run, and the experiments API. My verdict weighs cost, metric quality, setup, maintenance, and alternatives.

Quick verdict: I recommend Ragas for Python teams that want free, Apache-2.0 RAG metrics and synthetic test questions inside their own code. The trade-offs are no hosted dashboard, no release since January 13, 2026, and a separate bill for the judge LLM.

In my test, Ragas was good at telling a retrieval failure apart from an unsupported answer, but no single score worked as a pass/fail gate. In this review, I’ll take a closer look at Ragas’s pricing, metrics, and test data generation, and at where Langfuse or Braintrust fits better.

Key Takeaways

• Faithfulness scored my hallucinated answer 0.0, while answer relevancy rated it above the correct response.
• The free library pairs RAG metrics with a knowledge-graph testset generator that wrote six questions from three docs pages in 95.6 seconds.
• My fresh install crashed on import because Ragas leaves langchain-community unbounded and version 0.4.2 removed a module it imports.
• An honest refusal, where my test bot said the answer was not in its documentation, scored 0.0 on all five LLM-judged metrics, so refusals need separate grading.
• Maintenance has stalled: 0.4.3 is still the latest release, nothing has merged to main since February 24, 2026, and 253 pull requests are open.

Pros and Cons

How Much Does Ragas Cost?

Ragas costs $0 for the library, but every LLM-judged score spends tokens with your model provider, so the judge is the real line item.

• Open source ($0): For Python teams running metrics, experiments, and synthetic test generation themselves.
• Enterprise (email the founders): For enterprise features and collaborations. There is no published price or plan allowance.

Option or expense	Price	What it covers	Main limitation
Ragas open source	$0	Apache-2.0 library	You operate the workflow
Enterprise inquiry	Not published	Scope agreed with founders	No public plan or service commitment
Judge LLM	Provider token rates	LLM-based scoring and generation	Billed separately by your provider
Embeddings	Provider usage rates	Embedding-based metrics and generation	Additional usage cost
Local execution	Your infrastructure costs	Scripts, storage, and caching	Your operational responsibility

The Ragas docs’ cost example runs context recall over 20 samples and uses 25,097 input and 3,757 output tokens. The docs print $1.17 for that run, but at the stated GPT-4o rates of $5 and $15 per million tokens those counts work out to about $0.18.

At gpt-4.1-mini rates of $0.40 input and $1.60 output per million tokens, the same counts cost about $0.016. That is a rate comparison only: a different judge produces different token counts and, as my test showed, different scores.

My synthetic testset run reported 13,527 input and 1,711 output tokens, or $0.0081 at gpt-4.1-mini rates. That counter skips the knowledge-graph build, so it understates the end-to-end bill.

Faithfulness makes several LLM calls per sample (claim extraction, then verification), so seven metrics do not mean seven requests. Longer answers produce more claims and more tokens.

DiskCacheBackend stores exact-match LLM and embedding responses on local disk. It removes the provider charge when you rerun an unchanged evaluation, but the first run is billed in full.

Is Ragas Good Value for Money?

• Strong value for code-owned evaluation: the full metric library and testset generator cost nothing.
• Weaker for team review: Braintrust Pro costs $249/month and gives you the experiments UI that Ragas leaves to CSV files.
• Not a tracing tool: Langfuse Core ($29/month) and LangSmith Plus ($39/seat/month) cover traces and shared dashboards, which Ragas does not.
• Compare for CI: DeepEval is also a free library, and its hosted Confident AI Starter plan costs $200/month.

Reviewer’s Notes: I recommend staying on the free library and running a mini judge like gpt-4.1-mini while you iterate. Before a score becomes a release gate, check a stronger judge against a few human labels, cache repeated requests, and keep a separate baseline per judge.

My Experience With Ragas

My first Ragas run stopped at import ragas, before I could score a single answer. Once I fixed it, the metrics told me useful things, but that crash is why I would never ship Ragas unpinned.

Installing Ragas

I installed Ragas in a fresh virtual environment on macOS with Python 3.14.3. pip install ragas pulled version 0.4.3 and resolved langchain-community to 0.4.2.

Importing the package failed with this error:

ModuleNotFoundError: No module named 'langchain_community.chat_models.vertexai'

Even ragas --help failed, because the CLI imports the same package. langchain-community 0.4.2, released May 22, 2026, no longer has the module Ragas imports, and GitHub issue #2753 tracks the problem.

I fixed it with pip install "langchain-community<0.4.2", which resolved to 0.4.1. The import and the CLI both worked after that, and ragas --help listed three commands: evals, quickstart, and hello-world.

Ragas sets no upper bounds on langchain, langchain-core, langchain-community, or langchain_openai. The base install also pulled the whole LangChain stack (langchain 1.4.6, langchain-core 1.6.10, langchain-openai 1.1.9) plus datasets 5.1.0 and networkx, even though I only wanted to score answers.

Two optional dependencies surfaced only at call time: BLEU stopped with ImportError: sacrebleu is required, and testset generation asked for rapidfuzz.

The Quickstart Templates

ragas quickstart listed eight templates:

• rag_eval: RAG evaluation.
• improve_rag: Naive versus agentic RAG, with BM25 and Hugging Face docs.
• agent_evals: Math-problem agent evaluation.
• llamaIndex_agent_evals: Tool-call accuracy.
• text2sql: SQL execution accuracy.
• workflow_eval: Email classification and routing.
• prompt_evals: Sentiment prompt variations.
• judge_alignment: Judge alignment workflows.

For my metric test I wrote my own cases with the v0.4 API instead of a template: llm_factory("gpt-4.1-mini", client=AsyncOpenAI()) returned an Instructor-backed judge, and each metric’s ascore() returned a MetricResult.

My First RAG Evaluation

I built five cases around a mock SaaS help center covering refund rules, SSO, and office hours: a correct answer, a hallucinated answer (a 60-day refund window and prorated monthly refunds, both false), noisy retrieval, missing context, and an off-topic response.

Each case ran through seven metrics, including BLEU and exact match. With gpt-4.1-mini as judge and text-embedding-3-small for embeddings, the first run took 50.4 seconds for 35 metric calls, run concurrently per sample. The second took 50.1 seconds.

Case	Faithfulness	Context precision	Context recall	Factual correctness	Answer relevancy
Correct answer	1.0	1.0	1.0	1.0	0.752
Hallucinated answer	0.0	1.0	1.0	0.0	0.807
Noisy retrieval	1.0	0.5	1.0	1.0	1.0
Missing context with refusal	0.0	0.0	0.0	0.0	0.0
Off-topic answer	0.0	1.0	1.0	0.0	0.154

Answer relevancy rated the fabricated refund answer 0.807, above the correct answer’s 0.752. Faithfulness and factual correctness both rejected it at 0.0, so I would not use relevancy as a correctness gate.

Context precision dropped to 0.5 when an irrelevant chunk ranked first, while recall stayed at 1.0. That pins the fault on ranking, not on missing information, and faithfulness still gave the correct answer 1.0.

The honest refusal, where the bot said the answer was not in the documentation, scored 0.0 across all five. Context recall at 0.0 correctly blamed the retriever, but Ragas has no notion of a correct refusal, so the bot got no credit for declining to invent a policy.

Reviewer’s Notes: I recommend starting with faithfulness, context precision, and context recall. In my five-case test, that trio separated unsupported answers from retrieval problems, which answer relevancy on its own could not do.

RAG Metrics: What Ragas Catches and What It Misses

My hallucinated answer got a faithfulness score of 0.0 from gpt-4.1-mini and 0.333 from gpt-5-mini. Both judges spotted the unsupported claims, but a release threshold of 0.3 would pass one and fail the other.

The docs catalog lists 30 metrics in seven groups. My installed ragas.metrics.collections exported 38 metric classes, counting variants such as ContextPrecision with and without a reference.

Metric family	Examples	What it helps diagnose
Retrieval	ContextPrecision, ContextRecall	Ranking and evidence coverage
Answer grounding	Faithfulness, ResponseGroundedness	Support in retrieved material
Reference comparison	FactualCorrectness, SemanticSimilarity	Agreement with expected answers
Agent behavior	ToolCallAccuracy, AgentGoalAccuracy	Tool use and task completion
SQL	SQLSemanticEquivalence, DataCompyScore	Query meaning and execution results
String comparison	BleuScore, ExactMatch	Literal or lexical agreement

Faithfulness divides the response claims supported by the retrieved context by the total claims. The judge extracts the claims first, so claim splitting changes the denominator.

That is what happened in my gpt-5-mini run. It split the fabricated refund answer into three claims and counted “Yes, annual plans can be refunded” as supported, which produced 0.333 instead of 0.0.

The gpt-5-mini run took 87.3 seconds, 1.7 times as long as gpt-4.1-mini, and returned the same verdicts on every other case. Its answer relevancy scores (0.745, 0.767, 0.977, 0.0, 0.167) also differed from gpt-4.1-mini’s, so scores are not comparable across judges.

My two gpt-4.1-mini runs matched except on answer relevancy, which moved from 0.807 to 0.829 on the hallucinated case. That metric generates questions from the answer and compares embeddings, which adds run-to-run variation.

For regression testing, I recommend fixing these between runs:

• Judge: the same model and provider settings.
• Metrics: the same prompts and Ragas version.
• Dataset: the same questions and references.
• Refusal labels: a separate grade for correct abstentions.

BLEU and exact match both scored my correct paraphrase 0.0. They suit literal-output checks, not help-center answers, and BLEU needs sacrebleu installed separately.

Judge reliability is the other weak point. GitHub issues #1309 and #2044 report NaN scores, the latter with a local Ollama judge, and issues #1914 and #2457 cover timeouts and HTTP 429 rate limits under parallel scoring.

Non-English teams face a further gap: GitHub issue #2671 reports that BLEU, ROUGE, and quoted-spans scoring hardcode English assumptions and return wrong or zero scores in other languages.

TruLens covers a narrower RAG triad of groundedness, context relevance, and answer relevance. I favor Ragas when synthetic test generation matters alongside scoring, but neither tool spares you from validating the judge.

Synthetic Test Data Generation

Ragas generated six questions from three documentation pages in 95.6 seconds in my test. I would use the output as a seed dataset, not as a release benchmark.

The generator builds a knowledge graph before it writes any questions:

1. Documents become nodes with extracted summaries, themes, named entities, and embeddings.
2. Relationship builders connect nodes by cosine similarity and overlap scores.
3. Personas and scenarios shape each question through query style, length, and the evidence selected. My run generated three personas.
4. Synthesizers write the samples with reference answers and supporting contexts.

The default mix is 50% single-hop specific, 25% multi-hop abstract, and 25% multi-hop specific. My six questions came out two of each type, so a small batch will not mirror the configured split.

My first attempt split about 3,000 words of Ragas docs into paragraphs and failed with ValueError: Documents appears to be too short (ie 100 tokens or less). I switched to whole pages as documents and installed the missing rapidfuzz.

The successful run used three pages (about 3,075 words), gpt-4.1-mini, and text-embedding-3-small. About 80 of the 96 seconds went into building the knowledge graph, not into writing questions.

Output column	What it contains
user_input	Generated question
reference_contexts	Supporting source material
reference	Expected answer
persona_name	Associated persona
query_style	Question style
query_length	Length category
synthesizer_name	Generation type

Several questions carried deliberate misspellings, such as “scorng”, “evalute”, and “enviroment”, from the query-style setting that mimics real users. That helps if your users type that way.

The weak spots needed a human. One multi-hop question drifted to “financial documents”, a phrase from an example on a docs page rather than a real theme, and another read “How AsyncOpenAI used for evaluate…”.

Before these questions become benchmark items, I recommend checking each reference answer against its contexts and deleting questions built on incidental examples. Because the $0.0081 counter excludes graph building, measure the full pipeline once before you scale generation.

Agents, Text-to-SQL and the Experiments API

My two-row @experiment() run finished in 3.9 seconds and wrote experiments/judge-4.1-mini.csv. That suits experiments kept in code, while Braintrust and LangSmith give you the UI that Ragas leaves to files.

Ragas covers more than question answering:

• Agents: ToolCallAccuracy, ToolCallF1, AgentGoalAccuracy, and TopicAdherence score tool use and task behavior.
• Text-to-SQL: SQLSemanticEquivalence judges query meaning, and DataCompyScore compares execution results.
• Custom grading: DomainSpecificRubrics and InstanceSpecificRubrics support domain-level and per-sample criteria.

AspectCritic is not in collections in 0.4.3. The v0.4 migration guide points to @discrete_metric() as its replacement, so older AspectCritic examples will not run against the new imports.

The v0.4 workflow replaces evaluate() with @experiment(), although the legacy API still works for older metrics. Collections scorers return a MetricResult, and the number lives in result.value.

I created Dataset(name="helpdesk", backend="local/csv", root_dir="."), appended two rows, and saved it to datasets/helpdesk.csv. My async experiment wrapped a Faithfulness call and wrote the input columns plus a score column.

The monthly prorated-refund answer scored 0.0 and the annual-refund answer scored 1.0. The rows came back in completion order, not input order, so join runs on an ID column rather than row position.

Versioning experiments means naming runs and diffing CSVs, or committing them to git; there is no run browser.

For anything beyond files, the documented integrations include:

• RAG frameworks: LangChain, LlamaIndex, Haystack, and R2R.
• Agent and cloud workflows: LangGraph, Griptape, and Amazon Bedrock.
• Tracing: Arize Phoenix and LangSmith.

I did not test the tracing integrations, so my experience covers the local CSV workflow only.

Who Maintains Ragas Now?

Ragas has had no release since January 13, 2026, after 14 releases between July 2025 and January 2026. That gap matters because my fresh install already needed a compatibility workaround.

The last merge to main was February 24, 2026, a CI workflow fix. As of October 14, 2026, the repository had 253 open pull requests within 637 open issues and pull requests, and community pull requests were still arriving as recently as October 13.

The team announced Vibrant Labs on November 19, 2025, with a $2.5 million pre-seed round to build simulation environments for long-horizon agents. The repository moved to vibrantlabsai/ragas, and the company’s site now pitches reinforcement-learning data for agents rather than Ragas.

None of that proves Ragas is abandoned. It does mean I would not adopt it expecting prompt dependency fixes.

Ragas still logged about 1.05 million PyPI downloads last month and has 16,000 GitHub stars. Neither number replaces merge activity.

ragas.io still lists “Online Monitoring” and an email route for enterprise features, but Ragas has no publicly documented hosted app or published plan. Confirm any enterprise scope in writing before relying on support.

I recommend treating the Apache-2.0 code as fork-ready infrastructure: lock dependencies and budget time for your own patches. DeepEval, at about 2.58 million downloads last month and backed by the paid Confident AI platform, is the alternative if active vendor-led development matters more than Ragas’s metric workflow.

How Does Ragas Compare to Competitors?

Pick a Ragas alternative by the workflow you are missing, not by metric count.

• DeepEval / Confident AI: My pick for pytest-style CI tests with a hosted platform. The free hosted tier allows five test runs per week; Starter costs $200/month.
• TruLens: An MIT library for the RAG triad and OpenTelemetry tracing, without Ragas’s knowledge-graph testset generator.
• promptfoo: Better for YAML prompt/model test matrices and security red teaming. The Community edition is free.
• Arize Phoenix / AX: A stronger fit for tracing alongside evaluation. Phoenix uses ELv2, which is not an OSI-approved open-source license.
• Langfuse: My choice for prompt management and tracing in a shared UI, with Core at $29/month.
• LangSmith: A direct alternative for managed datasets and experiments, especially in LangChain stacks. Plus costs $39/seat/month.
• Braintrust: Better for UI-led experiment comparison. Starter includes 10,000 scores before usage charges; Pro costs $249/month.
• Opik: An Apache-2.0 option for agent tracing and test suites, with Pro Cloud at $19/month.

Tool	License or model	Hosted starting price	Best for
Ragas	Apache-2.0	No published hosted plan	Python metrics and synthetic testsets
DeepEval / Confident AI	Apache-2.0 library; commercial platform	Free; Starter $200/month	CI-oriented LLM tests
TruLens	MIT	No hosted price published	RAG triad and tracing
promptfoo	MIT	Free community; Enterprise custom	YAML matrices and red teaming
Phoenix / Arize AX	ELv2 Phoenix; commercial AX	AX Free; Pro $50/month	Tracing and evaluation
Langfuse	MIT core; separate enterprise license	Hobby free; Core $29/month	Tracing and prompt management
LangSmith	Proprietary platform	Developer free; Plus $39/seat/month	Managed experiments
Braintrust	Proprietary platform	Starter free; Pro $249/month	Collaborative evaluation
Opik	Apache-2.0	Free Cloud; Pro Cloud $19/month	Agent tracing and test suites

For Ragas vs DeepEval, I favor Ragas when ready-made retrieval metrics and graph-based test generation come first, and DeepEval when pytest-style checks and a commercial dashboard matter more.

Ragas can also sit next to a tracing platform instead of replacing it. Its documented Phoenix and LangSmith integrations support that setup, although I did not test either connection.

How I Reviewed Ragas

I installed Ragas 0.4.3 in a clean Python 3.14.3 environment on macOS and tested it against a mock SaaS help-center dataset.

My hands-on scope included:

• RAG scoring: Five cases across seven metrics, twice with gpt-4.1-mini and once with gpt-5-mini.
• Synthetic generation: Six questions from three documentation pages.
• Experiments: A two-row @experiment() run on the local CSV backend.
• Setup failures: The LangChain import crash and two missing optional dependencies.

I also checked the docs, release history, and open GitHub issues, and weighed the results against judge costs, maintenance activity, and named alternatives. I did not test enterprise services, tracing integrations, or non-English prompt adaptation, and five constructed cases are not a production accuracy benchmark.

Prices current as of October 2026.

Should You Use Ragas to Evaluate Your RAG App?

Yes, if your Python team owns its evaluation code and wants free, judge-agnostic scoring plus synthetic test generation. In my test, faithfulness caught fabricated refund terms, and context precision told noisy ranking apart from missing evidence.

Use Ragas as a diagnostic layer, not a single quality score. The zero-scored refusal and the 0.0 versus 0.333 faithfulness split mean you need human-labeled checks before any score becomes a release gate.

Skip it as your main evaluation platform if you need a shared dashboard or vendor support. Choose Langfuse for tracing and prompt management, Braintrust for experiment collaboration, or DeepEval with Confident AI for CI tests backed by a hosted product.

The import crash and the release pause make dependency ownership a prerequisite. A free license loses its appeal if nobody on your team can maintain the environment.

Start with 20 to 50 real questions scored for faithfulness, context precision, and context recall. Pin langchain-community below 0.4.2 for Ragas 0.4.3, then lock the full working environment before you record a baseline.

FAQ

Is Ragas free?

Yes. Ragas is free under the Apache-2.0 license, and you pay separately for judge-model calls, embeddings, and your own infrastructure. There is no published hosted subscription. Enterprise features require emailing the founders rather than picking a standard paid tier.

Does Ragas have a hosted app or dashboard?

No hosted app is publicly documented. My local experiments wrote CSV files, and the docs also support Google Drive-backed datasets. ragas.io still mentions Online Monitoring, so confirm any enterprise offering directly. See “Agents, Text-to-SQL and the Experiments API” above.

Which LLM can Ragas use as a judge?

Ragas supports OpenAI, Anthropic, and other providers through Instructor, plus 100+ providers through LiteLLM, including Gemini, Bedrock, and local models via Ollama. I tested gpt-4.1-mini and gpt-5-mini, and their faithfulness scores differed, so fix the judge before you set thresholds.

How much does a Ragas evaluation cost to run?

It depends on the judge’s rates, token volume, and metrics. The docs’ 20-sample context-recall example used 25,097 input and 3,757 output tokens, about $0.016 at gpt-4.1-mini rates of $0.40/$1.60 per million tokens. Testset generation adds graph-building costs its counter does not report. See “How Much Does Ragas Cost?” above.

Is Ragas still maintained?

Only partly. The latest release is 0.4.3 from January 13, 2026, and the last merge to main was February 24, 2026, while community pull requests keep arriving. I treat that as a maintenance risk, not abandonment: pin dependencies and plan to make compatibility fixes yourself.