MLflow Tracing

Editor’s note: This MLflow Tracing review is based on my hands-on test of MLflow 3.17.2 on October 27, 2026: a self-hosted tracking server on SQLite, an OpenAI-backed toy support agent, LLM-judge evaluations, OpenTelemetry ingest and trace sampling. I weighed cost, setup effort, trace detail, evaluation and production readiness.

Quick verdict: I recommend MLflow Tracing for Python teams that want free, OpenTelemetry-compatible tracing and LLM-judge evaluation they can host themselves, especially if they already run MLflow for classic ML. The trade-off is that you run the server, the SQL database and the upgrades, and live judge scoring in open source only works through MLflow’s AI Gateway.

MLflow Tracing is Apache 2.0 software under Linux Foundation governance, with one-line autolog for 40+ libraries and about 28.6k GitHub stars on the main MLflow repo. In this review, I’ll take a closer look at MLflow Tracing’s pricing, trace detail, and evaluation tools, so you can see exactly whether it belongs in your LLM stack. Experiment tracking and the model registry are covered in Panoply’s full MLflow review.

Key Takeaways

• Self-hosting is free under Apache 2.0, with no per-trace or per-seat metering.
• In my test, one autolog line captured every OpenAI call with token counts and a computed cost of $0.0000516 for one gpt-4.1-mini call.
• Plain OpenTelemetry spans sent to /v1/traces landed with token counts read from gen_ai attributes, no MLflow SDK required.
• Open-source automatic scoring refused my judge pointed straight at OpenAI: “only supported for scorers that use gateway models.”
• Databricks publishes no standalone price for managed tracing, and SageMaker managed MLflow bills from $0.60 an hour.

Pros and Cons

How Much Does MLflow Tracing Cost?

MLflow Tracing ships inside open-source MLflow under Apache 2.0, so self-hosting costs $0 in license fees with no trace, span or seat limits. Your bill is the machine, the SQL database, object storage and the tokens your LLM judges burn.

• Self-hosted open source ($0): for teams that will operate the tracking server, SQL database and artifact storage. Tracing, evaluation, the prompt registry and the AI Gateway all run on the same server.
• Amazon SageMaker managed MLflow ($0.60/hour Small, $1.40/hour Medium): for AWS teams that want AWS to run the tracking server. AWS’s pricing example puts 160 hours of a Small server at $96.00 and metadata storage at $0.10 per GB; the Large rate is not listed.
• Databricks managed MLflow (consumption-based): for teams whose data already sits in Databricks. Traces can live in Unity Catalog Delta tables with no trace cap, and production monitoring schedules judges on live traffic, but no standalone tracing rate is published.

Option	Price	Who runs it	Limits worth knowing
Self-hosted OSS	$0 license fee	You	Hosting and judge tokens extra
SageMaker Small	$0.60/hour	AWS	Metadata storage $0.10/GB
SageMaker Medium	$1.40/hour	AWS	Large size rate not listed
Databricks managed	Consumption-based	Databricks	No standalone tracing rate; Unity Catalog has no trace cap
Langfuse Core	$29/month	Langfuse	100,000 units; 90-day retention
LangSmith Plus	$39/seat/month	LangChain	10,000 base traces; 14-day base retention

mlflow.org routes new users to a Databricks free trial with trial credits, but the credit amount is not stated, so budget Databricks tracing as part of your existing Databricks consumption rather than as a fixed subscription.

Is MLflow Tracing Good Value for Money?

• Good fit: teams already running MLflow, or with an engineer who can own a SQL-backed service. With no metering, retention is limited by your disk, not by a $29 or $199 plan tier.
• Less convincing fit: teams that only need hosted LLM observability. Opik Pro costs $19/month for 100,000 spans, and Arize AX Pro costs $50/month for 50,000 spans with 30-day retention.
• Hidden cost: judge tokens and your own ops time. My 3-row evaluation with three gpt-4.1-mini judges ran in 6.8 seconds, and the whole test cost well under one cent in OpenAI usage.

Reviewer’s Notes: Start self-hosted on a small VM with PostgreSQL if someone on the team can own patching and backups. Move to SageMaker Small only if you already live on AWS and do not want to maintain the tracking server, and pick Databricks only if your data is already there.

My Experience With MLflow Tracing

Installing MLflow and Starting the Server

pip install mlflow openai in a Python 3.12.16 environment on an Apple silicon Mac pulled MLflow 3.17.2 and openai 3.40.0. On first import, MLflow printed a hint pointing coding agents to its bundled instrumenting-with-mlflow-tracing skill, one of about 15 agent skills the package ships alongside an mlflow skills CLI. I silenced it with MLFLOW_DISABLE_AGENT_HINT=1.

I started the server with mlflow server --host 127.0.0.1 --port 5055 --backend-store-uri sqlite:///mlflow.db, and its health endpoint answered after 9 seconds. The 3.17.2 SQLite schema already carries tables for review queues, guardrails, online scoring configs, budget policies, issues, MCP servers and daily trace and cost rollups, so the server is built for far more than storing spans. The CLI matches: mlflow mcp runs an MCP server, mlflow assistant analyzes traces with AI, and mlflow demo loads pre-populated demo data. That running server is a requirement Langfuse Cloud or LangSmith users never see, and a local SQLite file is a test setup, not a production template.

Tracing My First Agent

mlflow.openai.autolog() took 0.34 seconds to set up and instrumented every OpenAI call with one line. I then added @mlflow.trace decorators to a toy support agent: an AGENT root, a RETRIEVER span and a TOOL span. Two runs took 1.2 seconds and 0.93 seconds end to end, and one trace tree held four spans:

Span	Duration	What it showed
AGENT: support_agent	921.7 ms	Root span
RETRIEVER	0.5 ms	Retrieval step
TOOL: lookup_price	40.7 ms	ERROR (KeyError caught by my code)
CHAT_MODEL: Completions	833.2 ms	Autologged OpenAI call

Token usage rolled up to the trace: 37 input, 23 output, 60 total. MLflow priced the call itself at $0.0000516 ($0.0000148 input, $0.0000368 output) and resolved the model to the dated snapshot gpt-4.1-mini-2025-04-14, so I never had to rebuild cost from application logs.

The trace metadata also captured the git branch, commit and repo URL automatically, plus the session and user ids I set through mlflow.trace.session and mlflow.trace.user. Those ids became my search filters later, and because metadata is immutable once logged, they give a dependable per-user history.

What Happened When Something Failed

My price lookup raised a KeyError for an unknown plan, which my code caught. The TOOL span went ERROR, but the trace state stayed OK, so a dashboard that filters on trace state alone would miss that failed tool call.

An unhandled ValueError behaved differently: trace state ERROR, span status “ValueError: upstream 500”, and an exception span event.

Reading a trace immediately after creating it, before the async queue flushed, logged the warning “span data is corrupted”. After mlflow.flush_trace_async_logging(), the same trace loaded fine (state ERROR, 1 span). It was a race, not corruption, and the wording sent me looking for a data problem that did not exist.

Reviewer’s Notes: The handled-error behavior is the one I’d teach a team on day one. My agent logged an OK trace with a failed TOOL span inside it, so alerting and search should target span status, not just trace state.

Trace Detail and Integrations

MLflow’s integrations page claims 40+ libraries and lists 77 entries, which is the broadest list among the tools I compare below:

• Python agent frameworks (24): LangChain, LangGraph, OpenAI Agent, DSPy, PydanticAI, CrewAI, LlamaIndex and more.
• TypeScript (6): LangChain, LangGraph, Vercel AI SDK, Mastra, VoltAgent, Claude Agent SDK.
• Java (2): Spring AI and Quarkus LangChain4j.
• Coding agents (8): Claude Code, Codex CLI, Gemini CLI and five others.
• Infrastructure: 19 model providers and 10 gateways.

Each trace splits into two layers:

• TraceInfo: state (OK, ERROR or IN_PROGRESS), request and response previews, immutable metadata and mutable tags.
• Span tree: typed spans such as AGENT, TOOL, RETRIEVER and CHAT_MODEL, each with inputs, outputs and latency.

That split is why my handled KeyError showed up on the TOOL span but not on the trace state.

For OpenTelemetry, I used the plain OTel SDK with no MLflow import and pointed its OTLP/HTTP exporter at http://127.0.0.1:5055/v1/traces with an x-mlflow-experiment-id header. Both spans landed. MLflow typed the root as LLM from its gen_ai.request.model attribute and read token usage (12 in, 7 out, 19 total) from gen_ai.usage.* attributes. Any service in Go, Java or Rust can feed the same endpoint, and MLflow can also export over OTLP to Datadog or Grafana, including dual export.

The 2026 releases added:

• MLflow 3.12: image, audio and PDF attachments on traces.
• MLflow 3.13: archival of old traces from SQL to object storage such as S3.
• MLflow 3.16: a redesigned trace explorer with span links and session grouping.

MLflow accepted a standard OTLP producer straight into the same server that holds my evaluation runs and prompt registry, which Langfuse, as an LLM-only platform, does not combine with experiment tracking. I’d pick MLflow when that consolidation matters, and Langfuse when a dedicated hosted LLM UI matters more.

Searching Traces and Collecting Feedback

Both of my filters returned the expected 2 traces: metadata.mlflow.trace.user = 'user-42' and this combined tag-and-latency query:

mlflow.search_traces(
    filter_string="tags.env = 'test' AND trace.execution_time_ms > 900"
)

The filter DSL covers trace status, timestamps, execution time, span name and type, tags, metadata, feedback and expectation values, and prompts, so a query like “every trace a reviewer marked False this week” is one string. mlflow.search_traces returns a pandas DataFrame by default, with columns for trace_id, state, request, response, spans and assessments, or a plain list. The query language has limits worth knowing before you build saved searches:

• AND only: there is no OR operator.
• Full-text search: trace.text matching works only on the open-source SQL store.
• Archival: archived traces stay readable but drop out of content search, and archival cannot be undone.
• API change: experiment_ids= raised a FutureWarning in 3.17.2; use locations=.
• Scale: mlflow.search_traces() loads results into memory, so large result sets need MlflowClient.search_traces() with page_token pagination.

mlflow.log_feedback (HUMAN source, value False, with a rationale) and mlflow.log_expectation both attached to the trace and came back on trace.info.assessments. The mlflow traces CLI covers the same ground from a shell with search, get, evaluate, log-feedback, log-expectation, set-tag and delete subcommands, which makes trace triage scriptable from CI or a coding agent.

Best for: teams comfortable writing SQL-style filters in Python. Skip it if: you need OR queries or full-text search across archived history.

Evaluation and LLM Judges

My first mlflow.genai.evaluate run scored 3 questions about MLflow in 6.8 seconds, with all three judges answering “yes” on all three rows. The judges, all on model="openai:/gpt-4.1-mini":

• Correctness: checks the answer against the expected one.
• RelevanceToQuery: checks that the answer addresses the question.
• Guidelines(“one sentence”): enforces my own rule.

Offline evaluation needed no gateway, and MLflow logged the results to an evaluation run with a UI link. I then fed a static dataset with two deliberately wrong outputs, claiming MLflow uses an MIT license and is governed by the Apache Software Foundation. Correctness returned “no” on both, with rationales. Five rows prove the wiring works, not that the judge is reliable on your workload. Results land as runs in the UI, and traces carrying human feedback and expectations can be turned into evaluation datasets, so production failures become regression tests. Since 3.15, judges can also read image attachments from spans through a get_span_image tool.

MLflow 3.17.2 ships 27 built-in scorer classes, including:

• Conversation-level: ConversationCompleteness, UserFrustration, KnowledgeRetention.
• Tool calls: ToolCallCorrectness, ToolCallEfficiency.
• Retrieval: RetrievalGroundedness, RetrievalRelevance, RetrievalSufficiency.
• Safety: Safety, PIIDetection.

The Catch With Automatic Scoring

scorer.register() worked, but starting automatic scoring with a 0.5 sample rate failed with “Automatic evaluation is only supported for scorers that use gateway models.” The judge that ran fine offline against OpenAI cannot score live traffic until you route it through MLflow’s AI Gateway, which I did not set up for this test.

MLflow’s own docs set three more limits on open-source automatic evaluation: LLM judges only (code scorers built with @scorer are not supported), an AI Gateway endpoint for judge execution, and a newly enabled judge scores only traces and sessions at most one hour old. Databricks lists production monitoring that schedules scorers on live agent traffic as a managed feature. LangChain and LangGraph teams that want the most mature hosted evaluation should look at LangSmith instead.

Reviewer’s Notes: If live scoring is on your requirements list, route judges through MLflow’s AI Gateway from day one. My direct-to-OpenAI judge worked for offline runs and then had to be reconfigured before it could touch live traces.

Running It in Production: Overhead, Sampling and Storage

My 300 traced no-op calls added 0.14 ms per call on the request path. Flushing those 300 traces to the local server then took 6.42 seconds in the background, about 21 ms per trace. Request latency barely moves; the export backlog is where the work lands. The Databricks tracing FAQ gives larger payloads a higher price: roughly 1 ms at about 10 KB per trace, 50 to 100 ms at about 1 MB, and 150+ ms at about 10 MB. Custom span processors can filter or mask data before export, which is also where PII redaction belongs.

Open-source MLflow’s documented async defaults:

• On by default: trace logging runs off the request path.
• 10 worker threads handle background export.
• 1,000-trace queue: new traces are discarded once it is full.
• 500-second retry window before failed logging gives up.

That discard rule means durability is your job: size the queue for peak traffic and test shutdown flushing.

With MLFLOW_TRACE_SAMPLING_RATIO=0.1, my 200 calls kept 24 traces (about 12%). Sampling is per trace, so all spans in a trace are kept or dropped together, and @mlflow.trace accepts a per-function override for root spans.

For services, the mlflow-tracing package carries only the minimum dependencies (the docs claim a 95% smaller footprint than full mlflow). Do not install it in the same environment as full mlflow.

Spans live in a SQL spans table. My roughly 330 traces took 3.0 MB in SQLite, plus 288 KB in the artifacts folder. Use PostgreSQL or MySQL in production, and archival to object storage (since 3.13) for old traces.

On Databricks, experiments not backed by Unity Catalog cap out at 100,000 traces, and the trace list shows only the latest 1,000 by default. Long-running operations can sit IN_PROGRESS until MLFLOW_TRACE_TIMEOUT_SECONDS closes them. My recommendation: install mlflow-tracing in the service image, keep async logging on, set a sampling ratio below 1.0, and set that timeout explicitly.

How Does MLflow Tracing Compare to Competitors?

MLflow is the only tool here that also runs experiment tracking and a model registry in the same server, which pays off if you already use it and adds weight if you only need LLM tracing.

Tool	License/hosting	Free tier	Entry paid plan	Best for
MLflow Tracing	Apache 2.0; self-host or managed	Unlimited self-hosted	SageMaker from $0.60/hour	Existing MLflow teams
Langfuse	MIT core; cloud or self-host	Hobby: 50,000 units/month, 30 days	Core $29/month	Dedicated LLM observability
LangSmith	Closed source; self-host on Enterprise	5,000 base traces/month	Plus $39/seat/month	LangChain/LangGraph evaluation
Arize Phoenix / AX	Phoenix open source; AX cloud	AX Free: 25,000 spans, 15 days	AX Pro $50/month	OpenInference local tracing
Opik by Comet	Open source and cloud	Cloud: 25,000 spans/month	Pro $19/month, 100,000 spans	Cheapest hosted span allowance

Langfuse bills in units, the others in traces or spans, and a single trace can hold several spans, so headline quotas do not compare one to one.

• Langfuse: a polished LLM-native UI, MIT-licensed self-hosting and a $29/month cloud entry plan with 90-day retention. Pro runs $199/month with 3 years of data, overage is $8 per 100k units, and SSO and RBAC need a $300/month Teams add-on. Pick it over MLflow when not running a server matters more than one platform for ML and LLM work.
• LangSmith: the most mature hosted evals for LangChain and LangGraph teams, with a free Developer seat at 5k base traces and extended retention of 180 days. It is closed source with self-hosting only on Enterprise, so MLflow wins when you need the code on your own servers.
• Arize Phoenix: OpenInference-first local tracing. The hosted AX Free tier holds 25k spans and 1 GB for 15 days, AX Pro costs $50/month for 50k spans with only 30-day retention, and AX self-hosting is Enterprise-only.
• Opik by Comet: the cheapest hosted span allowance here at $19/month for 100,000 spans and $5 per extra 100k, with up to 50 members. It is a better buy than MLflow when nobody has time to run a server.
• W&B Weave: tracing and evals for teams already working in Weights & Biases, where runs, models and LLM traces share one workspace.

How I Reviewed MLflow Tracing

I installed MLflow 3.17.2 in a fresh Python 3.12.16 environment on an Apple silicon Mac on October 27, 2026, and ran everything against a local tracking server on SQLite. I:

• Traced a toy support agent with OpenAI autolog and manual spans.
• Forced handled and unhandled errors.
• Ran search filters and attached feedback and expectations.
• Ran three LLM judges plus a deliberate-failure dataset.
• Sent plain OpenTelemetry spans to the OTLP endpoint.
• Measured request-path overhead and trace sampling.
• Tried open-source automatic scoring.

I did not test Databricks-managed features, SageMaker hosting, the TypeScript SDK, the AI Gateway, or load at production scale, and I only tested MLflow Tracing, not its rivals.

Prices current as of October 2026.

Should You Use MLflow Tracing for LLM Observability?

Yes, if you are a Python team that wants free, self-hosted, OpenTelemetry-compatible tracing plus offline LLM judges, and a strong yes if MLflow already runs your classic ML. In my test it caught a failed tool call inside an OK trace, priced every OpenAI call, accepted a plain OTel producer and flagged two wrong answers.

Not yet, if you are a small team that wants a hosted UI tomorrow without running a server: Langfuse Core at $29/month or Opik Pro at $19/month gets you there faster. LangChain shops that want the most mature hosted evals should pick LangSmith. If your data already lives on Databricks, managed MLflow is the natural path, billed through your Databricks consumption.

Next step: run pip install mlflow, start mlflow server, add mlflow.openai.autolog(), and open one real request’s trace tree, failed child spans included.

FAQ

Is MLflow Tracing free?

Yes. MLflow Tracing is free to self-host under Apache 2.0, with no per-trace, per-span or per-seat charges. You pay for hosting and judge tokens. SageMaker managed MLflow starts at $0.60 an hour, and Databricks bills through platform consumption with no standalone tracing rate.

Does MLflow Tracing work without Databricks?

Yes. My whole test ran on a local SQLite-backed server, from autolog to offline judges. Databricks adds Unity Catalog trace storage with no trace cap and scheduled production monitoring. In open source, automatic scoring of live traffic needs judges routed through MLflow’s AI Gateway.

Does MLflow Tracing support OpenTelemetry?

Yes. The MLflow server accepts OTLP traces at /v1/traces and can export to OTel backends such as Datadog and Grafana, including dual export. In my test, a plain OpenTelemetry SDK script with no MLflow import sent two spans, and MLflow read the model and token counts (12 in, 7 out) from the gen_ai attributes.

Can I use MLflow Tracing from TypeScript or Java?

Yes. MLflow has a TypeScript SDK with integrations for LangChain, LangGraph, Vercel AI SDK, Mastra, VoltAgent and the Claude Agent SDK, and lists Spring AI and Quarkus LangChain4j for Java. Other languages can send spans to /v1/traces. I tested the OTLP route, not the TypeScript or Java SDKs.

Is MLflow Tracing production-ready?

Mostly, if you own the operations. Async logging is on by default (10 workers, a 1,000-trace queue that discards new traces when full), sampling works per trace, and the lightweight mlflow-tracing package keeps service images small. My 0.14 ms per-call overhead came from a local test, so load-test your own export path before relying on it.

How is MLflow Tracing different from MLflow experiment tracking?

MLflow Tracing records LLM and agent executions as traces and spans, with feedback and judge evaluation on top. Experiment tracking logs parameters and metrics for training runs. Both run on the same server; Panoply’s separate MLflow review, linked in the introduction, covers that side.