Datadog LLM Observability

Editor’s note: This Datadog LLM Observability review is based on my hands-on test of the product on the Free plan with the Python SDK, ddtrace 4.16.0, in October 2026. Datadog now sells it as Agent Observability. I weighed pricing, setup, agent tracing, evaluations, experiments, data controls and rivals.

Quick verdict: I recommend Datadog LLM Observability for teams already running Datadog APM or infrastructure monitoring that want agent traces on the same trace ID as their services. The trade-off: closed SaaS, 15-day default trace retention, and $6.50 to $9 per 10K LLM spans a month for longer retention through the account team.

Datadog LLM Observability traces model calls and agent steps, scores them with evaluators and runs offline experiments. Free covers 40K LLM spans a month; Pro starts at $160 a month on annual.

In this review, I’ll take a closer look at Datadog LLM Observability’s pricing, features, and evaluation tooling, so you can see exactly whether it belongs next to your production monitoring.

Key Takeaways

• Free includes 40K LLM spans a month, 15-day retention, full features and no credit card.
• Pro includes 100K LLM spans for $160 a month on an annual plan, $200 month-to-month or $240 on-demand. Extra spans cost $3.50 to $5.00 per 10K.
• Only LLM spans bill; tool, retrieval and workflow spans are free.
• No self-hosting and no US government sites, and traces keep 15 days even on Pro.
• LLM judges run on your own provider account, and evaluator tracing adds billed spans.

Pros and Cons

How Much Does Datadog LLM Observability Cost?

Datadog LLM Observability has two plans, Free and Pro, metered on LLM spans, where one LLM span is one call to an LLM provider. I budget on model calls rather than user requests, since one agent request can call a model several times.

• Free ($0/month): for individuals and small teams sending up to 40K LLM spans a month, with 15-day trace retention, unlimited evaluations, full feature access and no credit card.
• Pro ($160/month annually): for teams past the Free allowance, with the first 100K LLM spans included and $3.50 per extra 10K on an annual commitment.

Plan	Monthly price	LLM spans included	Extra spans per 10K	Trace retention
Free	$0	40K	Not offered	15 days
Pro annual	$160	100K	$3.50	15 days
Pro month-to-month	$200	100K	$4.20	15 days
Pro on-demand	$240	100K	$5.00	15 days

Longer trace retention is not available on Free. On Pro, the Datadog account team sets it up rather than a self-service toggle, and it applies to spans that have not expired yet; expired traces cannot be recovered.

Trace retention add-on	Rate per 10K LLM spans/month	Experiment retention
30 days	$6.50	6 months
60 days	$8	9 months
90 days	$9	12 months

Datadog does not state whether the add-on rate stacks on top of the $3.50 to $5.00 span rate or replaces it, so get that answer in writing before you commit.

Datadog does not charge for these span kinds:

• Workflow and Agent: the orchestration around your model calls.
• Tool and Task: function calls and task steps.
• Embedding and Retrieval: vector generation and retrieved context.

Sensitive Data Scanner is included at 1 GB of processing per 10K LLM spans, and no APM purchase is required. Evaluations carry no Datadog fee, but LLM judges bill tokens to the provider account you connect (OpenAI, Azure OpenAI, Anthropic, Amazon Bedrock, Vertex AI or a custom provider). Turning on Trace Evaluations for a custom evaluator also adds the judge’s own billed spans.

Is Datadog LLM Observability Good Value for Money?

• At 200K spans, annual wins: the meter comes to $195 a month on annual, $242 month-to-month and $290 on-demand, so annual is about a third cheaper than on-demand before retention and judge tokens.
• At 1M spans, annual comes to $475 a month: $160 plus 90 extra blocks of 10K at $3.50 each.
• Tool-heavy agents benefit: an agent that makes 3 model calls per request bills 3 LLM spans, while its tool and retrieval steps cost nothing.
• Paid entry is steep at low volume: Langfuse Core costs $29 a month and Arize AX Pro $50 a month, so the $160 floor only pays off when you already use Datadog APM and want the cross-service traces.

Reviewer’s Notes: I recommend staying on Free while usage fits within 40K LLM spans, then choosing annual Pro over on-demand for steady production traffic. Ask the account team about retention before you need traces older than 15 days, because expired spans cannot be restored.

My Experience With Datadog LLM Observability

I ran the whole setup in agentless mode, so the Python SDK sent spans straight to Datadog with an API key and no Datadog Agent installed. That path covers tracing and evaluations but not Prompt Experimentation, which needs a reachable Agent.

Getting an API Key and Installing the SDK

I created an API key under Organization Settings > API Keys, then installed the Python SDK with pip install ddtrace. My test used ddtrace 4.16.0, which supports Python 3.9 and later.

I configured the app through environment variables and started it with ddtrace-run:

pip install ddtrace
export DD_LLMOBS_ENABLED=1
export DD_LLMOBS_ML_APP=support-agent
export DD_SITE=datadoghq.com
export DD_API_KEY="$DD_API_KEY"
export DD_LLMOBS_AGENTLESS_ENABLED=1
ddtrace-run python app.py

The key came from the existing DD_API_KEY environment variable, so no credential sat in the command. DD_LLMOBS_ML_APP named the application support-agent, and DD_SITE matched my account’s site, datadoghq.com.

DD_LLMOBS_AGENTLESS_ENABLED=1 switched on direct delivery. Because ddtrace-run turns on all LLM integrations, I did not add an initialization call inside app.py.

Tracing My First Agent

Auto-instrumentation picked up my OpenAI, Anthropic and LangGraph calls as LLM spans, and the framework integration supplied the surrounding agent and tool context. For my own functions, I imported decorators from ddtrace.llmobs.decorators:

• @workflow: wrapped the overall application flow.
• @agent: marked the agent function and captured its inputs and outputs.
• @tool: marked a tool function inside the enclosing workflow.

I kept the trace root as a Workflow or Agent span, since Datadog accepts only LLM, Workflow or Agent at the root. Task, Embedding and Retrieval cover the remaining work below that root.

I then opened AI Observability > Traces and used the trace view and Execution Graph to follow tool decisions next to model-call latency, tokens and estimated cost. The graph had more nodes than the bill: decorating a tool did not make it a billable span, only the model calls counted.

Adding an Evaluator

I opened AI Observability > Evaluators > Create Evaluator and chose the Hallucination template. I picked a connected provider account and model, then set the application under Runs On.

Hallucination runs on LLM spans, so I kept span scope and set a sampling rate and filters; custom judges can also score a whole trace or a session. I left Trace Evaluations off to keep the judge’s own calls out of the span meter, then saved and published the evaluator.

Reviewer’s Notes: I recommend checking language coverage before copying this Python setup into Node.js or Java, where auto-instrumentation is much narrower. Prompt Experimentation requires a reachable Datadog Agent and cannot use this agentless path, and offline experiments also need DD_APP_KEY alongside the API key.

Tracing AI Agents: Auto-Instrumentation, OpenTelemetry and APM Correlation

Datadog’s 18 Python auto-instrumentations make it a far stronger fit for Python agents than for Java ones:

• Model providers: OpenAI/Azure OpenAI, Anthropic, Amazon Bedrock, Vertex AI, Google GenAI, MistralAI, vLLM and LiteLLM.
• Agent frameworks: OpenAI Agents, LangChain, LangGraph, CrewAI, Pydantic AI, Google ADK, Bedrock Agents, Claude Agent SDK, and Strands Agents through OpenTelemetry.
• Tool connectivity: MCP.

Each integration has its own minimum tracer version: Anthropic needs ddtrace 2.10.0+, LangGraph 3.10.1+ and the Claude Agent SDK 4.5.0+.

Outside Python, coverage narrows:

• Node.js: Bedrock, Anthropic, LangChain, MCP, OpenAI, Vercel AI SDK, Vertex AI and Google GenAI.
• Java: OpenAI/Azure OpenAI only.
• Go: experimental, manual instrumentation through dd-trace-go 2.3.0 or later.

Check the specific integration, not just the language: the Bedrock integration does not trace embedding calls, even though Embedding is a supported span kind.

OpenTelemetry GenAI 1.37+ and OpenInference traces ingest without the Datadog SDK, and OpenTelemetry span links render as edges in the Execution Graph, which keeps agent handoffs visible.

APM correlation is my strongest reason to pick Datadog over LangSmith. When both use ddtrace, an APM span shows View Span and the LLM span links back through APM span, with no extra setup. LangSmith is the better fit for a LangChain-centered stack; Datadog wins when the agent sits inside services Datadog APM already traces.

For local development, Lapdog runs on localhost:8126 with a browser dashboard and no Datadog account. It captures prompts and tool calls locally and can forward telemetry to Datadog.

Framework patches can lag releases. Users report that the ADK patch fails on google-adk 2.9.0 and later because an internal method moved, a problem still open since September 2026, so pin framework versions until the tracer catches up.

Evaluations: Templates, Custom LLM Judges and Human Review

Datadog has only 2 managed evaluators, Language Mismatch and Sensitive Data Scanning. “Unlimited evaluations” is not free inference: every template below runs on the custom LLM-as-judge engine against your own provider account.

• Failure to Answer
• Hallucination (LLM spans)
• Sentiment
• Toxicity
• Prompt Injection
• Topic Relevancy
• Tool Selection
• Tool Argument Correctness
• Goal Completeness

Custom judges return boolean, score, categorical or JSON output at span, trace or session scope. That split let me score a single answer separately from whether the whole agent run reached its goal.

Human review and external scoring sit next to the judges:

• End-user feedback: thumbs and comments tied to spans, traces or sessions.
• External evaluations: results submitted through the API, plus a NeMo integration.
• Annotation queues: human review of production interactions, with annotated interactions kept 90 days at no extra charge.
• Automation rules: route production traces into queues or datasets.

Automation rules are forward-only, not retroactive, so set up routing before the production traffic you want reviewed arrives.

Bits Evals, introduced at DASH in June 2026, adds an agentic evaluation workflow; Datadog lists no separate price for it.

Splunk Agent Observability, formerly Galileo, scores traffic with its own small Luna models, and Braintrust bills scores directly ($1.50 to $2.50 per 1,000 beyond the plan allowance). With Datadog, the judge cost lands on your OpenAI or Anthropic invoice instead.

Reviewer’s Notes: I recommend sampling LLM judges deliberately and leaving Trace Evaluations off unless you are debugging the judge. That toggle sends evaluator traces into the datadog-evaluations application and increases billed spans, on top of the provider’s token charges.

Experiments, Prompt Management and Patterns

Offline experiments compare prompt, model and agent variants against versioned datasets, which turns a bad production trace into a repeatable release check. Datasets come from three places:

• CSV imports: an existing evaluation set.
• Production traces: captured application interactions.
• Code: datasets built programmatically.

Experiments require ddtrace 4.3.0+ or dd-trace 6.13.0+, plus DD_APP_KEY alongside the API key. I compared runs side by side in AI Observability > Experiments.

Experiment retention follows its own schedule:

• Free and on-demand: results last 15 days.
• Committed plans: results last 90 days.
• Retention add-ons: results last 6, 9 or 12 months.
• Datasets: the current version lasts 3 years; previous versions last 90 days, reset on use.

Prompt Management is a registry for versioned prompts pulled at runtime, in Python with ddtrace 4.13.0+. Prompts stay 3 years, extended on each pull.

Prompt Experimentation is still Preview. Its A/B tests and Guarded Rollouts need an access request, ddtrace[openfeature] 4.15.0+ and a reachable Agent.

Patterns, introduced in Preview at DASH 2026, clusters production traffic into topics with volume, latency, cost, error rate and evaluation scores per cluster. Its summaries run through your connected LLM account.

Insights flags recurring cost and reliability problems with a recommended fix, with no extra configuration. Langfuse and Braintrust remain the stronger picks when dataset and experiment management is the main buying criterion; Datadog’s edge is that experiments sit in the same UI as production traces.

Cost Tracking, Sampling and Data Controls

Datadog estimates model cost per LLM and embedding span from provider public prices and token counts, including cache-read, cache-write and reasoning tokens. That estimate is your provider spend, separate from the Datadog span bill. For custom or self-hosted models, I can enter cost values manually.

DD_LLMOBS_SAMPLE_RATE defaults to 1.0, keeping every trace, and needs ddtrace 4.12.0+. The decision is made at the root span and applies to children, including downstream services connected through distributed tracing.

Token and cost metrics still cover 100% of instrumented traffic. Datadog does not spell out how much a lower sample rate cuts the span bill, so confirm that with them before you count on it.

Tag-based rules use DD_LLMOBS_SAMPLING_RULES and require ddtrace 4.16.0 or later. The Go SDK does not support sampling.

For sensitive data, these are the controls that matter:

• Span processors: redact content or drop spans before they leave the app.
• Sensitive Data Scanner: scans prompts and responses.
• RBAC and Data Access Control: restrict access, including by the ml_app tag.
• HIPAA eligibility: Datadog lists it as a HIPAA-eligible service.

Trace metrics stay for 15 months. The product is unavailable on app.ddog-gov.com, us2.ddog-gov.com and US-FED, which rules it out for those deployments.

How Does Datadog LLM Observability Compare to Competitors?

I recommend Datadog LLM Observability when LLM traces need to sit next to existing Datadog APM data; for standalone evaluation or self-hosting, the specialists below fit better.

• Langfuse: my pick for open-source deployment and a lower paid entry. Self-hosting is free; Hobby includes 50K units a month, Core costs $29 a month and Pro $199 a month. Core keeps data 90 days, against Datadog’s 15-day default.
• LangSmith: the fit for LangChain and LangGraph teams. Developer costs $0 per seat with 5K base traces; Plus costs $39 per seat a month. Self-hosting requires Enterprise.
• Arize AX and Phoenix: open-source Phoenix runs locally for evaluation. AX Free includes 25K spans; AX Pro costs $50 a month with 50K spans and 30-day retention.
• Braintrust: the evaluation-first alternative. Starter is free and Pro costs $249 a month with 50K scores and 30-day retention, a score meter that does not map onto Datadog’s LLM spans.
• Helicone: proxy-based request logging. Hobby includes 10K requests free and Pro costs $79 a month, but Hobby keeps data 7 days, shorter than Datadog Free’s 15.
• Comet Opik: cheap cloud tracing or free open-source self-hosting. Pro Cloud costs $19 a month for 100K spans; Free Cloud includes 25K spans and 60-day retention.
• Splunk Agent Observability (formerly Galileo): Luna evaluation models that score all traffic, sold on annual plans. Confirm Luna is included in the tier you are quoted before comparing evaluation costs.
• New Relic AI Monitoring: the alternative for existing New Relic teams. Ingest includes 100 GB a month free, then $0.40/GB for Original data, and AI features sit in an optional Advanced Compute add-on, so the ingest rate alone is not a full quote.

Compare allowances on your own workload, not headline volume: Langfuse units, LangSmith traces, Braintrust scores and Datadog LLM spans are different billing units, so price each rival on your real monthly call count.

Tool	Free tier	Paid entry	Self-host	Best for
Datadog LLM Observability	40K LLM spans	$160/month annual	No	Teams already on Datadog APM
Langfuse	50K units	$29/month	Yes	Open-source deployment
LangSmith	5K base traces	$39/seat/month	Enterprise	LangChain and LangGraph teams
Arize AX	25K spans	$50/month	Enterprise; Phoenix is open source	Evaluation and local Phoenix workflows
Braintrust	1 GB processed, 10K scores	$249/month	Not listed	Evaluation-first teams
Opik	25K spans	$19/month	Yes	Budget cloud or self-hosted tracing

How I Reviewed Datadog LLM Observability

I installed ddtrace 4.16.0 on the Free plan and instrumented a Python agent app with auto-instrumentation and decorators. I followed it from setup to trace inspection and configured a Hallucination template evaluator on a connected provider account.

I checked what offline experiments and Prompt Experimentation need on top of that, including the application key and the Agent. I also worked through the sampling controls and the separate retention rules for traces, experiments and datasets.

For pricing, I ran the meter at 100K, 200K and 1M LLM spans on Datadog’s published rates, excluding retention add-ons and provider tokens, and set the results against the rival plans above.

I weighed pricing, setup, agent tracing, evaluations, experiments, data controls and rivals, with the most weight on APM correlation and language coverage.

Prices current as of October 2026.

Should You Use Datadog LLM Observability for Your AI Agents?

Yes, if your services already run on Datadog and you want agent traces on the same trace ID, dashboards and monitors, with the APM span one click away from the LLM span. The standalone Free plan is also worth a look if your agents do heavy tool and retrieval work between model calls, since only LLM spans bill.

Skip it if you need self-hosting, a US government site or long default retention. Langfuse is free to self-host and Arize Phoenix runs locally; for hosted retention on a small budget, Langfuse Core keeps 90 days for $29 a month and Arize AX Pro 30 days for $50, before Datadog’s $6.50 to $9 add-ons.

Start on Free with ddtrace-run and one sampled Hallucination evaluator. Count the model calls your workload makes, keep evaluator tracing off, and price annual Pro against that number before you cross 40K spans; at 200K spans, annual Pro comes to $195 a month.

FAQs

Is Datadog LLM Observability free?

Yes. Datadog LLM Observability has a $0 plan with up to 40K LLM spans per month, full feature access and 15-day trace retention, with no credit card. LLM-judge tokens still bill to your connected model provider, and the 30, 60 and 90-day retention add-ons are not available on Free.

How is Datadog LLM Observability priced?

Datadog bills only LLM spans, with Pro starting at $160 a month on annual for 100K spans. Month-to-month costs $200 and on-demand $240; extra spans cost $3.50 to $5.00 per 10K. Budget on model calls, not user requests, because one agent request can generate several billable spans.

Do I need Datadog APM to use LLM Observability?

No. Datadog LLM Observability is a standalone purchase and supports agentless delivery with an API key. APM, infrastructure and RUM add trace correlation when you have them, but none of them is required to trace LLM applications or run evaluations.

Does Datadog LLM Observability support OpenTelemetry?

Yes. It ingests traces that follow the OpenTelemetry GenAI semantic conventions 1.37+ or the supported OpenInference conventions, without the Datadog SDK. Span links render as edges in the Execution Graph, so an app that already emits compatible telemetry needs no second instrumentation library.

Can I self-host Datadog LLM Observability?

No. Datadog LLM Observability has no self-hosted edition, and production trace data lives in Datadog. Lapdog runs a local development agent on localhost:8126 with a browser dashboard and no account, but it is not a production replacement. Langfuse or Arize Phoenix fit when the observability stack must stay under your control.