Lunary

Editor’s note: This Lunary review is based on documented workflows and current product specifications. I have not tested the product firsthand.

Quick verdict: I recommend Lunary for chatbot teams that want to investigate conversations and connect problems to the underlying model or agent activity. That context is what makes it worth considering. The trade-off is a paid cloud entry point and instrumentation work beyond wrapping an LLM client.

Among LLM observability and evaluation tools, Lunary makes my shortlist when engineers and product colleagues need to understand the same failed interaction. A response can look technically successful while still being useless to the person receiving it. I value a workflow that makes both perspectives available.

For casual experimentation, I prefer a free starting point. For a project built around evaluation, I give the available assessment methods more weight than the chat workflow.

Key Takeaways

• Conversation-linked investigation is the strongest reason to consider Lunary.
• Versioned prompts and dataset evaluation support a workflow beyond passive logging.
• New cloud access requires payment; grandfathered Free workspaces are not a new free plan.
• Full agent and chat visibility requires deliberate instrumentation.
• The current Docker deployment guides describe an Enterprise route, not a simple free-hosting shortcut.

Below, I weigh the commercial commitment against the setup work, investigation workflow, prompt tools, and deployment requirements.

Lunary Pros and Cons

My main attraction is the surrounding context. My main reservation is that collecting useful context takes more implementation effort than basic request logging.

How Much Does Lunary Cost?

Lunary pricing combines seat charges and usage allowances. I recommend budgeting for everyone who needs to investigate problems, then assessing the workload you intend to record.

• Team: $20 per user monthly, billed monthly. My starting choice for a small chatbot engineering team.
• Enterprise: custom pricing. My route for organizations with deployment or governance requirements.

Pricing detail	Team	Enterprise
Seats	Maximum 10	Custom arrangement
Monthly events	50,000 included; $10 per additional 50,000	Custom arrangement
Playground queries	300 monthly; then $0.05 each	Custom arrangement
History	One year	Custom arrangement
Projects and prompts	Unlimited	Custom arrangement
Enterprise features	See Enterprise	Self-hosting, SSO, granular access control, PII masking, white labeling, warehouse connectors, 99.9% SLA

Five Team seats produce a $100 monthly seat subtotal, before usage. This covers seats only; it is not an estimate of your complete bill.

Customer count alone is a poor basis for estimating event consumption. Lunary describes runs covering different operations, including inference calls and chat messages. A conversation involving an agent should therefore not be equated with a single recorded operation.

My budgeting preference is to start with an application diagram. Mark where the application calls a model and where it creates additional tracked activity. That is a more useful commercial starting point than assuming that a chatbot serving a modest audience necessarily produces modest telemetry.

Is Lunary Good Value for Money?

I see the best value when investigation is recurring work. A specific debugging problem gives you a clearer reason to pay than a long feature list does.

• For chatbot debugging: I see a good fit when engineers regularly need the surrounding interaction to explain a bad answer.
• For occasional prompt experiments: I recommend starting with a free alternative if the wider workflow is likely to remain unused.
• For cross-functional review: Include the intended reviewers in the budget. The people assessing bad answers may sit outside engineering.

I also recommend assessing experimentation and production monitoring separately before buying. A team may care deeply about one and barely use the other. Paying for a combined workflow makes sense only if the combination removes a real handoff.

Reviewer’s Notes: My pick is Team for a small support-chatbot team with a defined debugging problem. Infrastructure-controlled deployments belong in Enterprise discussions, while Langfuse’s free offering is my starting point for zero-budget exploration.

Getting Started With Lunary

Start with one complete application path. My priority is a representative request that you can follow from beginning to end; disconnected records make that harder.

The documented Python workflow begins with a project key and a monitored client. Agent and conversation context comes afterward, through additional integration.

1. Configure the Python SDK. Lunary requires Python 3.10 or newer. Install it with pip install lunary and set the project public key through LUNARY_PUBLIC_KEY. My recommendation is to keep the initial integration narrowly scoped so configuration errors are easier to isolate.

2. Monitor the OpenAI client. Create the client, apply lunary.monitor(client), and make normal completion calls. The integration accepts tags and user context alongside requests. This is the starting point for model-call visibility, not the whole application trace.

3. Add agent structure. Instrument agents, chains, and tools with the documented wrappers or decorators. Nested calls can then retain their parent relationships. Lunary’s concepts documentation places tracked tools within an agent or chain rather than treating them as standalone tool records.

4. Connect the conversation. Open or resume a thread using an ID, record messages, and connect message IDs to backend activity. Design those identifiers alongside the application’s conversation model so both sides refer to the same interaction.

5. Inspect the recorded path. Logs and analytics expose model activity for investigation. My first review would focus on whether the records explain the request from entry to response, rather than whether a dashboard merely contains data.

Be deliberate about which user data you send. Lunary’s user tracking takes a user ID and optional properties; context can pass that identity to child runs. My preference is stable application identifiers and purposeful organization metadata. Available fields are not a reason to send names or email addresses.

JavaScript serverless deployments have an additional requirement: await lunary.flush() at the end of the function to send queued data. That belongs in the deployment implementation, not in a later dashboard cleanup task.

My recommended starting scope is one model call inside one meaningful user interaction. Once that path is understandable, expanding the same conventions is more sensible than instrumenting every component independently. The useful milestone is an explainable interaction, not an impressive volume of logs.

Tracing Agents and Understanding Conversations

The selling point is connecting chats to execution. Lunary is most interesting to me when a technically successful call can still disappoint the person using the application.

Its observability documentation covers cost and token usage alongside latency and errors. Request and response logs provide the content behind those measurements, while nested instrumentation adds workflow structure. My suggested approach is to identify a problem in aggregate analytics, then inspect individual runs to understand it.

The underlying distinctions matter:

• Runs represent individual operations, such as model requests or agent activity.
• Traces group related runs into an execution path.
• Threads collect the chat activity belonging to a conversation.

Those are related views, not interchangeable names for the same thing. The useful division is an engineer following the execution path while a product reviewer assesses what the person actually asked.

Consider a hypothetical support chatbot that gives the wrong refund answer. My investigation would begin with the complaint, follow the relevant interaction, and inspect whether retrieval or a tool supplied an incorrect result. The prompt is not necessarily responsible simply because the final wording was wrong.

Conversation tracking also supports tags and custom events, such as a document being opened. I see value in that when an application action explains why a later response looks strange. It is less compelling if the team records events without a clear question they are intended to answer.

Feedback can arrive from the frontend or backend as a JSON object attached to a message or run ID. Thumbs and comments are recommended fields, and feedback can filter calls. Feedback is a triage signal, not proof that an answer is correct or incorrect.

Classification adds another layer: local models assign topics, languages, and sentiment. It remains beta and may not be available on every plan. For me, that makes it a supporting organizational aid rather than the reason to buy.

Lunary also documents OpenTelemetry ingestion through OTLP-compatible instrumentation paths. That is useful to consider if you already have a telemetry strategy, although it does not establish complete provider parity or a frictionless migration.

Rich context earns its keep when someone uses it to make a decision. I recommend choosing a small set of investigation questions before expanding the tagging scheme: which failures matter, who reviews them, and what can that person change?

Managing Prompts and Evaluating Responses

Versioned prompt management gives Lunary a role beyond collecting logs. I like the possibility of moving from an observed problem to a controlled candidate change, provided the team distinguishes experimentation from evidence.

Prompt templates store versions and render through SDK calls using a slug and variables. The documented OpenAI integration records template usage. After the application adopts templates, prompt edits can be used without redeploying application code.

That flexibility deserves a release habit. A prompt adjustment is a behavioral change, even when it does not travel through the same deployment process as application code. A shorter path to production is useful only if ownership remains clear.

The latest-template API returns version information, including a draft flag and notes. Those distinctions belong in the team’s review process. In my view, a saved edit and an approved production decision should be separate milestones.

Playground Experiments Are Not Regression Tests

The Playground supports interactive experimentation. It supports provider and model comparisons, generation parameters, and dynamic variables. Draft versions can carry notes and be shared.

Custom endpoints extend that workflow to an application API, with configurable authentication, headers, and payload defaults. For a retrieval-based product, my preference is an experiment that includes application logic. An isolated prompt leaves that part of the behavior outside the comparison.

My preferred distinction is:

• Playground experiments: Explore candidate behavior and decide which changes deserve further attention.
• Human feedback: Identify interactions that people found useful or disappointing.
• Dataset evaluations: Apply an explicit assessment process to a saved collection of cases.

A few attractive replies are not enough for me to recommend promoting a change. Use exploration to narrow the candidates, then evaluate them against a deliberate set of expectations.

Dataset APIs Provide a Repeatable Structure

Lunary’s dataset API imports CSV or JSONL items. It supports attaching evaluators by ID and running attached evaluators across every item, returning updated results.

Dataset version snapshots include a version number, timestamp, and item count. A fixed test set matters because changing the examples while changing the prompt makes the comparison harder to interpret.

The evaluation API separately supports creating a named evaluation tied to a dataset. Criteria accept a metric and can include thresholds, weighting, and parameters; another endpoint runs an evaluation by ID. Evaluation creation uses a project private API key, which belongs in backend automation rather than frontend code.

I see enough structure here to support a disciplined workflow, but the discipline remains the team’s job. My recommendation is to include ordinary successful requests alongside failure cases, then define what an acceptable answer means before reviewing scores.

If evaluator breadth is the deciding factor, LangSmith deserves a close comparison. Its documented offline and online approaches include code-based and LLM-as-judge evaluation, with pairwise and summary options. My preference would depend on the assessment method the application actually needs, not the largest feature inventory.

Security and Self-Hosting

Lunary self hosting involves an infrastructure commitment and licensing for the current documented Docker route. I would budget for both before treating it as a cheaper alternative to cloud access.

The Docker Compose guide describes an Enterprise Edition deployment requiring:

• PostgreSQL 15 or newer for the database.
• Private registry access to obtain the deployment images.
• A license key for the licensed installation.

The Compose services include the frontend and backend alongside ML and enrichment components. The separate Docker guide also describes an Enterprise-only route; optional model-provider keys enable Playground and evaluation features.

The FAQ still refers to a free Community Edition. That wording does not make the current licensed Docker instructions a free deployment guide. It also leaves the Community route distinct from the Enterprise installation described here.

My buying judgment turns on operational ownership. Budget for database administration and backup recovery, with explicit responsibility for secrets and updates. Moving the application into infrastructure under the team’s control should not mean leaving those tasks unassigned.

For cloud security, Lunary documents TLS in transit and encryption at rest. It identifies Hetzner hosting, while the FAQ says data is hosted in Europe.

Lunary also reports SOC 2 Type 2 and ISO 27001:2022 external audits. I treat those as vendor-reported audit status, not an independent assurance from this review. Its security documentation explicitly states that Lunary Cloud is unsuitable for HIPAA-compliant data collection.

For me, deployment suitability comes before workflow convenience. A strong tracing interface cannot compensate for an unsuitable data-handling arrangement, and self-hosting alone is not a security or compliance guarantee.

How Does Lunary Compare to Competitors?

Your main workflow should determine the shortlist. The most relevant Lunary alternatives have different commercial units and different reasons to make the shortlist.

• Langfuse: My first comparison for free exploration and repeatable pre-release experiments. Its documentation covers dataset-driven tasks and evaluators, plus SDK agent experiments and UI prompt experiments. Its open-source self-hosting route also deserves attention when deployment cost is decisive.

• LangSmith: My closer comparison when evaluation design is the central requirement. Its range of evaluation approaches is most relevant when the team already knows which assessment methods it needs.

• Helicone: Relevant when unified model access, routing, and fallbacks belong in the purchase. However, its founders’ March 2026 announcement placed it in maintenance mode. The service remains live with security, model, bug, and performance updates, but that lifecycle position weighs heavily against it as a new dependency for me.

Alternative	Entry plan	Paid reference	Usage unit	My buyer-fit judgment	Main trade-off
Langfuse	Hobby: free, 50,000 units/month, two users, 30-day data access	Core: $29/month, 100,000 units, unlimited users, 90-day access	Units	Free exploration and structured experiments	Compare history requirements carefully
LangSmith	Developer: $0/seat, one seat, 5,000 base traces/month	Plus: $39/seat/month, 10,000 base traces; usage separate	Traces	Evaluation-centered teams	Base retention is 14 days; extended retention is 180 days
Helicone	Hobby: free, 10,000 requests/month, 1 GB, one seat	Pro: $79/month plus usage, unlimited seats	Requests and storage	Gateway-centered requirements	Maintenance mode; Hobby retention is seven days

These quotas do not support a cheapest-per-interaction ranking. A trace can contain multiple operations, and each vendor’s billing vocabulary needs its own workload model.

Lunary remains my more compelling shortlist candidate when reviewing a conversation is the starting point for debugging. If the conversation itself is incidental, the competing workflow deserves more weight.

How I Reviewed Lunary

I reviewed Lunary’s pricing and technical documentation to assess fit, implementation effort, and limitations. That included the SDK integrations, feature guides, evaluation APIs, and deployment requirements an engineering buyer would need to work with.

I focused on the path from instrumentation to investigation and prompt evaluation, alongside commercial and deployment boundaries. I did not create a workspace, execute instrumentation, or benchmark performance. My recommendations are buyer judgments based on documented behavior.

Prices are current as of October 2026.

Should You Use Lunary?

Lunary makes my shortlist for chatbot investigation. Its strongest case is a team that needs to understand an unsatisfactory answer in the context of the interaction and the execution behind it.

I recommend it when the engineering team is willing to implement the relationships that make that workflow useful. Simply collecting model requests would leave much of the appeal unused. The buying question is whether connected investigation is valuable enough to justify that implementation effort.

My preference is Langfuse for free exploration or a free-self-hosting shortlist. LangSmith gets priority when evaluator design dominates the decision. Helicone belongs in a gateway comparison, subject to its maintenance-mode caveat.

My final recommendation is to define the investigation workflow before committing: identify who reviews failures and what decision the review should produce. If Lunary’s conversation-to-execution model matches that work, it has a clear purpose. If the requirement is only a place to store requests, I would keep comparing.

Lunary FAQ

Is Lunary free?

New cloud workspaces require payment before access. Existing Free workspaces retain their plans, but that does not create a free signup option for a new team. For a no-budget starting point, my recommendation is a rival’s free tier.

Can I self-host Lunary?

Yes. The current Docker and Docker Compose guides describe an Enterprise deployment with licensed images. The FAQ’s Community Edition wording should not be treated as instructions for that route. Base your deployment planning on the specific edition you intend to use.

Does Lunary require LangChain?

No. Lunary documents direct OpenAI client monitoring in Python, so LangChain is not a prerequisite. Prompt templates also have documented manual use with other LLM APIs. I recommend choosing the integration that matches your application rather than adding a framework solely for observability.

What is the difference between tracing and evaluations?

Tracing records what happened during execution. Evaluations assess outputs against a defined process. Lunary supports both, but they answer different questions: tracing to investigate a particular failure, and dataset evaluation to assess a proposed change across selected cases.

Does Lunary replace an LLM provider?

No. Lunary instruments and records model and application activity; the application still uses its model provider. I assess it as part of the AI data stack around your model service, with its own integration work and budget.