Galileo

Galileo is an eval engineering platform for LLM apps and agents, with one of the deepest eval libraries in the LLM observability and evals category: 31 built-in metrics, feedback-tuned custom judges, Luna-2 small judge models, and Signals for production failures. Since Cisco closed the acquisition on May 22, 2026, it has been sold as Splunk Agent Observability, and galileo.ai/pricing points at Splunk’s price list.

In this Galileo review, I’ll take a closer look at Galileo’s pricing, features, and how it compares to LangSmith, Arize, Braintrust and Langfuse, so you can see exactly whether it’s worth buying under Splunk.

Editor’s note: This Galileo review evaluates the pricing model, setup requirements, and documented capabilities of Galileo (now Splunk Agent Observability) as of October 2026, covering both Splunk’s current SaaS edition and the original Galileo platform. It is not a hands-on performance test. I weigh pricing, eval quality, setup and integrations, production monitoring and guardrails, and vendor direction.

Quick verdict: I recommend Galileo for teams shipping agents to production who want offline evals to become cheap production checks, especially Splunk or Cisco shops. Go in knowing that Splunk’s SaaS edition runs only 5 preset Luna evaluators and no Luna fine-tuning, Protect is deprecated in favor of Agent Control, and Splunk publishes new edition pricing in early November 2026.

Key Takeaways:

• 31 built-in metrics, 9 of them agentic, including Action Completion and Tool Selection Quality
• Agent Observability costs from $100 per month per 1.2 million spans a year, billed annually
• Luna-2 judges average 152 ms against 3,200 ms for GPT-4o, on Galileo’s own benchmark
• The SaaS edition caps Luna at 5 preset evaluators with no fine-tuning, so the full Luna stack needs the on-prem deployment
• Pricing is a moving target: Luna token and retention add-ons have no public price, and new editions land in November 2026

Pros and Cons

How Much Does Galileo Cost?

Galileo’s pricing link now resolves to Splunk’s price list, which carries one line: Splunk Agent Observability starts at $100 per month per 1.2 million spans annually, billed annually. That works out to roughly 100,000 spans a month for $100, with 500 million Luna tokens and 5 Luna fine-tuning runs per year included. Trace retention defaults to 30 days and extends to 60, 90 or 180 days, experiment data is kept for 2 years, and extra Luna tokens and longer retention are add-ons with no public price. The plan runs in Splunk Observability Cloud or self-hosted.

• Splunk Agent Observability ($100/month per 1.2M spans a year): for production agent teams ready to sign an annual contract
• Splunk Observability Cloud Free Edition ($0): for instrumenting a first agent, capped at 15 hosts, no credit card, with agent tracing, token tracking and the Agent Observability sample projects
• Enterprise (custom): for regulated or high-volume teams that need VPC or on-prem deployment, SSO, enterprise RBAC and Luna-2 inference

For accounts opened before the deal, Galileo’s pre-acquisition plans looked different: Free cost $0 with 5,000 traces a month and unlimited users, Pro started at $100 a month billed yearly with 50,000 traces, and Enterprise was custom. Pro scaled with trace volume, so $100 was a floor, not a flat rate.

Plan	Price	Usage basis	What’s included	Who it fits
Splunk Agent Observability	From $100/month, billed annually	1.2M spans per year	500M Luna tokens, 5 Luna fine-tuning runs a year, 30-day trace retention (up to 180), cloud or self-hosted	Production agent teams
Splunk Observability Cloud Free Edition	$0	Up to 15 hosts	Agent tracing, token tracking, no time limit	First instrumentation
Galileo Free (pre-acquisition)	$0	5,000 traces/month	Unlimited users, unlimited custom evals	Legacy accounts
Galileo Pro (pre-acquisition)	From $100/month, billed yearly	50,000 traces/month	Standard RBAC, advanced analytics, Slack support	Legacy small teams
Enterprise	Custom	Unlimited traces, custom rate limits	Hosted, VPC or on-prem, SSO, enterprise RBAC, real-time guardrails, dedicated inference servers, 24/7 support	Regulated and high-volume teams

Splunk announced new Observability editions at its Conf26 event in September 2026, each bundling Agent Observability with APM, Infrastructure Monitoring and Observability Logs. Detailed entitlements and pricing are due in early November 2026, so the single plan above may change within weeks of this review.

Splunk quotes Luna-2 evaluations at about $0.12 per million tokens against roughly $5 per million for a frontier LLM judge, while Galileo’s own benchmark table lists Luna-2 at $0.02 per million. The two vendor figures don’t reconcile, so I’d price Luna on a sample of my own traffic before trusting either.

Is Galileo Good Value for Money?

• Good fit: teams that would otherwise run GPT-class judges on every production trace, and Splunk customers who can fold Agent Observability into an existing contract
• Less convincing fit: small teams that want transparent self-serve pricing, since Langfuse Core costs $29/month for 100k units, Comet Opik Pro $19/month for 100k spans, and Arize AX Pro $50/month for 50k spans
• Budget priority: Datadog LLM Observability is free up to 40K spans a month and $160/month for 100K, so Splunk’s roughly 100,000 spans for $100 is competitive on paper, but only on an annual commitment
• Hidden cost: every LLM call, retriever hit and tool call is its own span, so an agent that calls five tools per request uses at least six spans where the legacy plans counted one trace

Reviewer’s Notes: I’d start on the free path to instrument one agent and run a few experiments, then hold off on an annual commitment until Splunk publishes the November edition entitlements. The exception is a team that needs Luna-2 guardrails on live traffic right now: that means an Enterprise negotiation, and I’d get the extra Luna token and retention rates in writing, since neither has a public price.

Getting Started With Galileo

Galileo’s setup is one package install and a few environment variables, but which package depends on when the account was opened: accounts onboarded since August 7, 2026 use Splunk’s splunk-ao SDK and docs, while earlier accounts keep the galileo SDK and docs.galileo.ai.

Creating a Project and API Key

On a new account, I’d open Agent Observability from the Splunk Observability Cloud main menu, then put two values in a .env file next to my model provider key: SPLUNK_AO_REALM and SPLUNK_AO_O11Y_TOKEN on the SaaS edition, or SPLUNK_AO_API_KEY and SPLUNK_AO_CONSOLE_URL on a standalone deployment. Accounts from before the rebrand use GALILEO_API_KEY, GALILEO_PROJECT and GALILEO_LOG_STREAM instead.

What I like about the setup: calling splunk_ao_context.init() (galileo_context.init() on legacy accounts) with a project and stream name creates both if they don’t exist yet, so there’s no console clicking before the first trace. Splunk now calls log streams Agent Streams.

Logging My First Traces

New accounts install pip install "splunk-ao[openai]"; legacy accounts use pip install galileo (Python 3.10 or later) or npm install galileo for TypeScript. From there, I’d pick the lightest hook for my stack:

• @log decorator: wraps any function as an agent, llm, retriever, tool or workflow span, and flushes the trace when the function exits in Python
• LangChain and LangGraph: a GalileoCallback handler passed into the run
• OpenAI SDK: a drop-in wrapper client from the SDK’s openai module
• OpenAI Agents SDK and CrewAI: a trace processor and an event listener
• OpenTelemetry: an OTLP traces endpoint plus a span processor for distributed tracing, with OpenInference instrumentors available on Python only

Running the script prints a Project link and a stream link in the terminal, and each trace opens in the console with a Messages tab showing the full exchange. If my agents already emit OpenTelemetry from Vercel AI SDK, Strands Agents or Google ADK, I’d point the exporter at Galileo rather than re-instrument, which keeps the option of switching to Arize Phoenix or Langfuse later.

Running an Experiment Over a Dataset

Datasets upload as CSV, JSON or Feather, with standard columns such as input, ground_truth and metadata, and Galileo versions every change automatically. The Python SDK handles datasets up to 100,000 rows.

run_experiment takes the dataset, a prompt template or my own function, and a list of metrics, and each row becomes a trace I can open and inspect. Prompt-driven experiments and synthetic data generation both need an LLM integration configured first. Galileo also supports running experiments inside unit tests, which is how I’d wire evals into CI.

Dataset version history and local code-based metrics are Python-only for now, so TypeScript teams lose both.

Reviewer’s Notes: Every page on docs.galileo.ai now carries a banner saying it applies only to customers onboarded before August 7, 2026, with newer accounts sent to agent-observability-docs.splunk.com. I’d check which doc set matches my account before copying any snippet, since the package names, import paths and environment variables differ between the two.

Galileo Metrics, Custom Judges, and Autotune

Galileo ships 31 active built-in metrics, and the 9 agentic ones are why I’d shortlist it over tools built around single-turn chat.

🧪 31 Built-In Metrics

• Agentic (9): Action Advancement, Action Completion, Agent Efficiency, Agent Flow, Conversation Quality, Reasoning Coherence, Tool Errors, Tool Selection Quality, User Intent Change
• RAG (6): Chunk Relevance, Context Precision, Context Relevance, Precision @ K, Completeness, Context Adherence
• Response quality (3): Correctness, Ground Truth Adherence, Instruction Adherence
• Safety (5): PII/CPNI/PHI, Prompt Injection, Sexism/Bias, Toxicity, Unsafe Output
• Other (8): 3 multimodal metrics, Tone, and 4 text-to-SQL checks (SQL Correctness, SQL Adherence, SQL Injection, SQL Efficiency)

Older metrics such as BLEU/ROUGE, Prompt Perplexity, Uncertainty and Chunk Attribution Utilization are deprecated, so dashboards built on them need rebuilding.

Action Completion, Agent Flow and Agent Efficiency score a whole session, Tool Selection Quality scores each LLM span, and Tool Errors scores each tool span, so a multi-turn agent gets judged at the level where its failures actually happen.

⚖️ Custom LLM-as-a-Judge Metrics

When the built-ins miss my use case, a custom judge lets me set the level (session, trace, retriever, LLM or tool span), the judge model from my integrations, step-by-step reasoning on or off, and the number of judges whose scores get averaged. Outputs can be boolean, categorical, count, discrete or percentage. Galileo researchers published the same multi-judge idea in 2023 as ChainPoll, a hallucination detection method.

Every extra judge adds API cost and time; Braintrust at least prices its scoring openly, with 10K scores free on Starter and $2.50 per 1,000 after.

🔁 Autotune

Autotune turns reviewer corrections into prompt changes in six steps: review the spans where a metric underperforms, flag the wrong scores, enter the expected value and the reasoning, retune, review the prompt diff, and publish. Feedback has to state the correct output and why; a bare “this score is wrong” gives Autotune nothing to learn from. Each publish creates a new metric version, works across span, trace and session levels, and can recompute historical results. It shipped in March 2026.

Luna-2 Evaluation Models and Signals

Luna-2 is Galileo’s answer to the frontier-judge bill: small 3B and 8B models fine-tuned from Llama to do nothing but score, which Splunk pitches as the way to check 100% of traffic at a fraction of a frontier judge’s cost.

🌙 Luna-2 Small Judges

On Galileo’s own benchmark, Luna-2 reaches an F1 of 0.95 with 152 ms average latency and a 128k-token context, against GPT-4o at 0.94 and 3,200 ms and GPT-4o mini at 0.90 and 2,600 ms. On H100 GPUs, Galileo lists the 3B model at 15 ms for a 500-token input and 141 ms at 15K tokens. Splunk claims 10 to 20 parallel checks in under 200 ms on L4 GPUs.

• Fine-tuning: roughly 4,000 labeled samples per customer-specific metric, built as lightweight adapters on a shared base model
• Luna Studio: no-code fine-tuning deployed into your own Kubernetes, Vertex AI, AzureML or SageMaker pipelines, with GPU jobs running in your environment
• Hardware: L4 or higher, Galileo-hosted or self-hosted, though L4 cards cover log streams and experiments only, not runtime protection
• Access: Splunk’s SaaS edition supports only 5 preset Luna evaluators (Prompt Injection, Toxicity, Sexism, PII and Context Adherence) and no fine-tuning; the full Luna-2 set and Luna Studio need the on-prem deployment

Splunk’s plan includes 5 Luna fine-tuning runs a year, yet the SaaS edition can’t fine-tune Luna at all, so I’d confirm in writing which deployment those runs assume before signing.

📡 Signals

Signals runs a reasoning LLM over 100% of a log stream’s production traces, distills each session into notes, then looks for patterns across sessions. Each finding gets a priority from 1 to 10: Error (8 to 10), Warning (4 to 7) or Info (1 to 3). Each detail view shows an explanation, a suggested action, example traces and a timeline of affected spans, and any signal can become a metric.

Signals runs on request from a Log Stream or the AI Assistant rather than continuously, and it needs my own LLM integration with a reasoning model, so the analysis bills against my provider account. I’d use Signals to decide which failures deserve an eval, and I’d test it first against the tracing views in Datadog LLM Observability or Arize AX.

Guardrails, Agent Control, and the Splunk Integration

Galileo’s original runtime guardrail product, Protect, has been deprecated since June 2026, so any team that built on it is already facing a migration.

🛡️ From Protect to Agent Control

Protect chained rules (metric, operator, target) into rulesets with AND logic, then ordered rulesets into prioritized stages. Every Protect metric except custom code-based ones required the Enterprise tier with Luna-2.

Agent Control replaces it as an Apache-2.0 open-source control plane released on March 11, 2026:

• @control() decorator marks the agent functions to govern
• Remote policies update without redeploying the agent
• Five decisions: deny, steer, warn, log or allow
• Pluggable evaluators: Galileo Luna, NVIDIA NeMo, AWS Bedrock, regex or custom code
• Launch partners: Cisco AI Defense, CrewAI, AWS Strands, Glean, Rubrik and ServiceNow

An enterprise edition followed on June 5, 2026, and the GitHub repo sits at 326 stars with commits as recent as October 2026. Open sourcing the guardrail layer is the strongest hedge Galileo offers against lock-in.

🔗 Splunk Agent Observability

Under Splunk, the product splits into modules: Evaluation, Visibility (tool, model and retrieval tracing correlated with infrastructure down to the GPU), Tokenomics (cost by request, model, agent and workflow, now including Claude Code and Codex spend by team), Guardrails for PII, PHI and PCI leakage, tool misuse and prompt injection, and Luna. The SaaS edition leaves out custom code-based evaluators, Luna fine-tuning and the AI Assistant, which stay on the on-prem deployment, and its results don’t show up in Observability Cloud’s global search. Former CEO Vikram Chatterji now runs agent resilience as a Splunk senior director. Splunk has not described specific changes for existing Galileo customers, which is the roadmap risk I’d weigh most.

🔒 Security and Access

Galileo holds SOC 2 Type II and offers HIPAA BAAs, with full documents on request through a Vanta-hosted Trust Center. Roles are Admin, Manager, User and Read-only, but Manager and user groups are Enterprise-only. Data residency and regions are not publicly documented, so regulated EU buyers should ask before signing.

How Does Galileo Compare to Competitors?

Galileo’s edge over most rivals is agent-specific metrics plus a cheap in-house judge; its weakness is buying friction, since the only listed plan is an annual Splunk contract with unpriced add-ons.

• LangSmith: best for teams building on LangChain and LangGraph. The Developer plan is $0 per seat with 5k base traces and Plus is $39 per seat with 10k, though base retention is 14 days and the 180-day option costs extra. Self-hosted and hybrid deployment sit on Enterprise.
• Arize AX and Phoenix: best for teams that want a self-hostable path. AX Free covers 25k spans with 15-day retention and unlimited users, AX Pro is $50/month for 50k spans, and Phoenix installs via pip, Docker or Helm with 11.8k GitHub stars, though under the Elastic License 2.0 rather than an OSI license.
• Braintrust: best for eval-first teams that live in experiments. Every tier includes unlimited users, projects and experiments; Starter is $0 with 1 GB and 10K scores, and Pro is $249/month with 5 GB and 50K scores. Free retention is 14 days, and Starter overage runs $2.50 per 1,000 scores.
• Langfuse: best for open source on a budget. It is MIT-licensed, free to self-host and part of ClickHouse since January 2026; Hobby is free for 50k units and 2 users, Core is $29/month for 100k, but SSO and fine-grained RBAC need the Pro plan plus a $300/month Teams add-on, or the $2,499/month Enterprise plan.
• Datadog LLM Observability: best for teams already running Datadog APM. It is free up to 40K LLM spans a month, Pro costs $160/month for 100K, and extra spans run $3.50 per 10K, with traces sitting next to the APM data Datadog customers already pay for.
• Comet Opik: the cheapest hosted entry point. It is open source, Free Cloud includes 25k spans with 60-day retention, and Pro is $19/month for 100k spans. Extra spans cost $5 per 100k, and 400-day retention is a $29 add-on per 100k spans.

Tool	Free tier	Entry paid plan	Open source	Self-host
Galileo (Splunk Agent Observability)	Splunk Free Edition, 15 hosts	From $100/month per 1.2M spans a year	SDKs and Agent Control only	Yes
LangSmith	5k base traces	$39/seat, 10k traces	No	Enterprise
Arize AX / Phoenix	25k spans, 15 days	$50/month, 50k spans	Phoenix (ELv2)	Phoenix
Braintrust	1 GB, 10K scores	$249/month, 5 GB, 50K scores	No	Not listed
Langfuse	50k units, 2 users	$29/month, 100k units	Yes (MIT)	Yes
Datadog LLM Observability	40K LLM spans	$160/month, 100K spans	No	No
Comet Opik	25k spans, 60 days	$19/month, 100k spans	Yes	Yes

Pick Galileo if you need agentic metrics and cheap production judges and can buy through Splunk; pick Langfuse or Phoenix if open source and self-hosting come first; pick LangSmith if your agents already run on LangGraph.

How I Reviewed Galileo

I judged Galileo on five areas: pricing and feature gating, eval quality and metric coverage, setup and integrations, production monitoring and guardrails, and vendor direction after the Cisco deal.

For pricing, I compared Splunk’s current plan with Galileo’s pre-acquisition plans and with published rates from LangSmith, Arize, Braintrust, Langfuse, Datadog and Comet Opik. For eval quality, I went through all 31 built-in metrics, the custom judge options, Autotune, Signals and Luna-2’s cost and latency claims, treating every benchmark as Galileo’s or Splunk’s own figure. For setup, I mapped the path from sign-up through the SDK, framework hooks, OpenTelemetry and experiments. For direction, I weighed the Protect deprecation, the Agent Control release, the split between Galileo’s legacy docs and Splunk’s new ones, and the November edition change.

This is not a hands-on performance test, so I make no claims about speed or accuracy on my own workloads.

Prices current as of October 2026.

Should You Choose Galileo for LLM Evals?

Yes, if you run agents in production and want evals that carry over into always-on checks. Galileo’s 9 agentic metrics, Autotune and Signals cover the full loop from experiment to production failure, and teams already on Splunk or Cisco observability get the cleanest buying path.

Skip it if you’re a small team that wants open source and transparent self-serve pricing: Langfuse at $29/month or Arize Phoenix self-hosted will cost less and lock you in less. LangChain-first teams will get more out of LangSmith’s native tracing.

The SaaS edition limits Luna to 5 preset evaluators with no fine-tuning, and Splunk publishes detailed edition pricing in early November 2026, so an annual commitment signed in October may look different a month later.

My next step would be to instrument one agent with the splunk-ao SDK on the Free Edition, run an experiment scored on Action Completion and Tool Selection Quality, and only then talk to Splunk sales with real span volumes in hand.

FAQs

Is Galileo AI free?

Yes, with limits. Splunk Observability Cloud’s Free Edition covers up to 15 hosts with agent tracing and token tracking, needs no credit card, and has no time limit. Galileo’s pre-acquisition Free plan allowed 5,000 traces a month with unlimited users and unlimited custom evals. Luna fine-tuning and the full Luna evaluator set need the on-prem deployment, so neither free option shows the full production feature set.

Who owns Galileo AI now?

Cisco owns Galileo. Cisco announced the deal on April 9, 2026, completed it on May 22, 2026, and placed the product inside Splunk, where it was renamed Splunk Agent Observability on August 7, 2026. New accounts use Splunk’s splunk-ao SDK and docs, while the Galileo-named SDK, docs and app serve accounts opened before August 7, 2026.

How much does Galileo cost?

Splunk Agent Observability starts at $100 per month per 1.2 million spans a year, billed annually, including 500 million Luna tokens and 5 Luna fine-tuning runs a year. Enterprise pricing is custom. Extra Luna tokens and longer retention cost extra, with no public rate. See How Much Does Galileo Cost? above for the full table, including the pre-acquisition plans.

What are Galileo Luna models?

Luna-2 models are 3B and 8B small language models fine-tuned from Llama to score LLM outputs and power guardrails. On Galileo’s own benchmark they average 152 ms per check with an F1 of 0.95, and Splunk quotes evaluations at about $0.12 per million tokens (Galileo’s benchmark table says $0.02). Splunk’s SaaS edition runs 5 preset Luna evaluators; the full set and fine-tuning need the on-prem deployment.

Is Galileo open source?

Partly. The original galileo Python and TypeScript SDKs and the Agent Control guardrail plane are Apache-2.0, but the platform, metrics engine and Luna models are proprietary. If a fully open-source, self-hostable stack is a requirement, Langfuse (MIT) and Comet Opik are the closer fits.