Promptfoo

Editor’s note: This Promptfoo review is based on my hands-on evaluation of Promptfoo 0.124.1 (the October 8, 2026 release) in October 2026. I weighed pricing and the probe cap, setup, eval assertions and CI, red teaming, privacy defaults, the OpenAI ownership change, and rivals.

Quick verdict: I recommend Promptfoo for engineering teams that want prompt and model regression tests in CI plus pre-launch red teaming against the OWASP LLM Top 10 for free. The trade-offs: no production monitoring, free red teaming stops at 10,000 probes a month, and OpenAI ownership (announced March 9, 2026) matters if you test rival models.

Promptfoo puts prompts, providers and tests in one YAML file, runs from npx, and shows results in a local web matrix, which suits engineers who review prompt changes next to application code.

In this review, I’ll take a closer look at Promptfoo’s pricing, features, and red-teaming limits, so you can see exactly whether it fits your development workflow.

Key Takeaways:

• Community is free forever, with every eval feature and 60+ model providers.
• 157 red-team plugins ship with OWASP, NIST and MITRE ATLAS presets.
• Free red teaming stops at 10,000 probes a month; Enterprise is quote-only.
• No production observability: tracing serves evals, not live apps.
• OpenAI owns Promptfoo now, but the MIT license stays.

Pros and Cons

How Much Does Promptfoo Cost?

Promptfoo has three tiers, and only one has a price: free. Enterprise and On-Premise are custom quotes, with no self-serve paid tier between the free toolkit and Enterprise.

• Community (free): for engineers running evals locally or on a self-hosted server. It includes every LLM eval feature, all providers and integrations, custom integration with your own app, vulnerability scanning, community support and 10,000 red-team probes a month.
• Enterprise (custom quote): for teams that outgrow the probe cap or need managed cloud deployment. It adds custom red-team limits, team sharing, continuous monitoring, a security and compliance dashboard, customizable attack profiles, SSO, granular permissions, API access, professional services, priority support and an SLA.
• On-Premise (custom quote): for organizations that must keep everything on their own infrastructure. It adds complete data isolation, a dedicated runner and an assigned deployment engineer to the Enterprise feature set.

Plan	Price	Red-team probes	Deployment	Key extras
Community	Free	10,000/month	Local or self-hosted	All eval features and providers, vulnerability scanning, community support
Enterprise	Custom quote	Custom limits	Managed cloud	Team sharing, continuous security monitoring, compliance dashboard, SSO, permissions, API access, priority support, SLA
On-Premise	Custom quote	Custom limits	Your infrastructure	Enterprise features, data isolation, dedicated runner, deployment engineer

A probe is one request to your target system during red teaming. The cap exists because some plugins need inference for dynamic test generation and grading, and Enterprise customers can buy additional probes.

Free software does not mean free inference: every eval and every llm-rubric grader call bills on your own provider key. Hosted eval platforms charge from the first paid tier instead: Braintrust Pro costs $249/month and Confident AI Starter costs $200/month.

Is Promptfoo Good Value for Money?

• Good fit: Community is excellent value for prompt regression suites, because no assertion type or provider integration sits behind an upgrade. Engineers can compare competing models without buying a hosted workspace first.
• Less convincing fit: the free self-hosted server is no substitute for Enterprise collaboration. It has no authentication or horizontal scaling, which matters more than the subscription savings once several teams need access.
• Budget priority: budget provider inference separately from red-team capacity. Repeated evals and model-graded assertions run up API charges while the Promptfoo bill stays at zero.

Reviewer’s Notes: Start on Community for engineering-led evals. Ask for an Enterprise quote only when the 10,000-probe cap, SSO or shared dashboards become real constraints, not simply because the team plans to ship an LLM feature.

My Experience With Promptfoo

Installing Promptfoo and Running the Starter Eval

Promptfoo needs Node.js 22.22.0 or newer (Node 24 LTS is recommended), and that applies even to the pip package. I ran it through npx promptfoo@latest, which skips a global install on the first run. The other install routes are:

• npm: npm install -g promptfoo
• Homebrew: brew install promptfoo
• pip: pip install promptfoo, which still requires Node.js.

With an OpenAI key exported, init --example getting-started created a getting-started directory holding promptfooconfig.yaml and a README.md. Inside it, eval ran the example tests and view opened the local results viewer.

export OPENAI_API_KEY="your-provider-key"
npx promptfoo@latest init --example getting-started
cd getting-started
npx promptfoo@latest eval
npx promptfoo@latest view

For guided setup, promptfoo init walks through an interactive CLI flow and promptfoo eval setup opens a web-based setup instead.

Reading the promptfooconfig.yaml

The config file splits into prompts, providers and tests. Prompts take {{variable}} placeholders, the providers list picks the models to call, and each test supplies variables plus optional assertions.

That layout is Promptfoo’s real strength for regression work: prompt wording and expected behavior sit in one reviewable file, and adding a provider runs the same cases against another model. I rebuilt the example into the config below, which compares two prompt styles across two OpenAI models with a case-insensitive string check and an llm-rubric grader pinned to a named model.

prompts:
  - "Explain {{topic}} in one sentence."
  - "Explain {{topic}} to a beginner in one sentence."

providers:
  - openai:gpt-6-luna
  - openai:gpt-6-sol

tests:
  - vars:
      topic: encryption
    assert:
      - type: icontains
        value: encryption
      - type: llm-rubric
        value: "Accurately explains encryption in accessible language."
        provider: openai:gpt-6-sol

Assertions are optional, so a first pass can be manual review. The web viewer laid out a prompt-by-provider matrix with outputs, pass/fail, cost and latency, and I added assertions once I could describe an acceptable answer precisely.

Running My First Red Team Scan

npx promptfoo@latest redteam setup opened a web UI that walks through four steps:

1. Application details, including the Purpose field.
2. The target: an HTTP endpoint, a model or custom code.
3. Plugins (or a preset) and attack strategies.
4. Review, then save the config.

promptfoo redteam run then generates several hundred adversarial inputs into redteam.yaml and runs them against the target, and promptfoo redteam report presents the findings. For a headless setup, promptfoo redteam init --no-gui skips the browser, which is the route for scripted pipelines. Attack generation runs on a separate provider, OpenAI by default, not on the model under test.

Reviewer’s Notes: The YAML-first workflow suits engineers who already review config in pull requests, while product managers will lean on the web viewer. LangSmith and Braintrust start in a hosted UI instead, which is friendlier to non-engineers and harder to version alongside code.

Evals: Assertions, Model Graders and CI

Promptfoo’s assertions split into deterministic and model-assisted families, and the grader default is the trap to watch. Use deterministic checks wherever the requirement is exact, and keep model grading for judgments string matching cannot make.

• Deterministic: equals, contains, icontains, regex and starts-with check text; is-json, contains-json, is-sql and is-html check structured output; custom JavaScript, Python and Ruby assertions plus cost and latency thresholds cover the rest.
• Model-assisted: similar uses embeddings; llm-rubric, g-eval, factuality and model-graded-closedqa grade answer quality; answer-relevance, context-faithfulness and context-recall cover RAG; trajectory:goal-success and select-best handle agent and comparative evals.

Each assertion has a default weight of 1, and a test’s score is the weighted average. assert-set groups checks under a pass threshold, so weight deliberately: several minor format checks should not outvote one essential correctness check.

With no grader configured, llm-rubric and factuality fall back to the provider under test, so a model can grade itself. Users raised this in GitHub issue #10581 in September 2026. I treat the grader as part of the test definition and pin it explicitly, especially when a suite compares rival models.

Promptfoo also has execution controls for repeatable suites:

• Disk cache: bypass it with --no-cache or empty it with promptfoo cache clear.
• Concurrency: --max-concurrency (or -j) caps parallel requests.
• Resume: interrupted evals pick up where they stopped instead of restarting.
• Exports: results save as JSON, YAML, HTML or a spreadsheet.

Running Promptfoo in CI

The official promptfoo/promptfoo-action@v1 runs a before/after eval on pull requests that touch prompts and posts a link to the results. Any other CI system runs the CLI directly:

npx promptfoo@latest eval -c promptfooconfig.yaml -o results.json

Keep results.json as a build artifact so the eval evidence stays attached to the prompt change, and pin the grader in the committed config so CI never lets a model grade itself. DeepEval offers a similar metric library for teams that would rather define tests in Python.

Promptfoo also ingests OpenTelemetry (OTLP) traces at eval time, and trace-aware assertions check tool calls, execution paths, timing and errors. That helps agent evals, but it is not production monitoring of the kind LangSmith provides.

Red Teaming: Plugins, Strategies and the Probe Cap

Promptfoo ships 157 red-team plugins across six categories: brand, compliance and legal, dataset, security and access control, trust and safety, and custom. Pick plugins around what the application actually does, because a support bot and a coding agent fail in different ways, and every extra plugin spends probes.

Plugins decide which vulnerabilities get probed; strategies decide how attacks reach the target. The recommended pair is Meta Agent (single-turn agentic) and Hydra (adaptive multi-turn); the rest include Goblin multi-turn, jailbreak templates and static encodings such as Base64, hex, leetspeak, homoglyphs and audio or image encoding.

strategies:
  - jailbreak:meta
  - jailbreak:hydra

Hydra is my pick when the risk builds over a conversation rather than arriving in one malicious request. Framework presets give security teams a starting taxonomy:

Preset	Coverage
owasp:llm	OWASP LLM Top 10 2025, LLM01 through LLM10
nist:ai:measure	NIST AI measurement mapping
mitre:atlas	MITRE ATLAS mapping

Beyond chat targets, Promptfoo covers three more surfaces:

• MCP servers: tests for tool poisoning, data exfiltration, auth hijacking and tool shadowing.
• Code scanning: a GitHub Action traces data flows in pull requests to flag prompt injection, PII exposure and excessive agency; the VS Code extension is Enterprise-only.
• Model files: promptfoo scan-model (ModelAudit) statically checks pickle opcodes, Keras Lambda layers, TensorFlow ops, embedded executables and hidden credentials.

The 10,000-probe cap counts target requests, not scans. A run of several hundred inputs across multiple strategies, repeated after every prompt change, eats the monthly allowance far faster than the headline number suggests.

Logged into Promptfoo Cloud, red teaming prefers remote generation through Promptfoo’s API. PROMPTFOO_DISABLE_REDTEAM_REMOTE_GENERATION=true opts out at the cost of lower attack quality. In GitHub issue #8199 (March 2026), users saw jailbreak:composite and jailbreak:likert generate zero tests when the hosted API was blocked, so check that dependency before scheduling scans on a restricted network.

Garak (NVIDIA, Apache 2.0) and PyRIT (Microsoft, MIT) are free alternatives for security teams, and Giskard’s open-source library runs a basic vulnerability scan, but none pairs attack scans with an eval matrix. Regression tests and red teaming in one config is the stronger reason to pick Promptfoo.

Privacy, Telemetry and Self-Hosting

Promptfoo runs locally, but three paths send data out, so set the opt-outs before sensitive test cases go in.

• Telemetry (on by default): events for each command run and assertion type used, with package version and a CI flag; with an account, also user ID, email and cloud login status. Prompts, outputs, test cases, API keys and full configs are excluded. Disable it with PROMPTFOO_DISABLE_TELEMETRY=1 and update checks with PROMPTFOO_DISABLE_UPDATE=1.
• Remote attack generation: opt out with PROMPTFOO_DISABLE_REDTEAM_REMOTE_GENERATION=true, then configure your own attack provider and expect lower-quality attacks.
• Cloud sharing: after promptfoo auth login, sharing uploads the eval snapshot (prompts, vars, outputs, traces, metadata, provider config fields) to promptfoo.app as links private to you and your organization.

PROMPTFOO_DISABLE_REMOTE_GENERATION is not an egress firewall: it does not stop telemetry, account checks, sharing or Cloud sync.

The self-hosted Docker image is an Express server on SQLite with no built-in auth or SSO, no multi-team RBAC and one replica only; a second replica causes “Job not found” errors. Enterprise buyers get a separate enterprise image, while Langfuse self-hosts as a full production observability stack. Set the env vars in CI before the first run, because telemetry starts with the very first command, and keep cloud sharing off for any eval that must stay internal.

What Changed After OpenAI Bought Promptfoo?

So far, neither the license nor the release pace has changed. OpenAI announced on March 9, 2026 that Promptfoo agreed to be acquired, with terms undisclosed; founders Ian Webster (CEO) and Michael D’Angelo (CTO) ran a 23-person company. The README now says Promptfoo is part of OpenAI and remains open source and MIT licensed.

The company committed to maintaining the suite for any AI model or application, supporting a diverse range of providers, and serving existing customers. OpenAI plans to fold Promptfoo’s core tech into its model and infrastructure layers, including OpenAI Frontier, its enterprise agent platform.

Releases kept coming after the deal: five in six weeks, from 0.122.2 on August 28 to 0.124.1 on October 8, 2026. The repo sits at about 26,000 stars and roughly 390 contributors, and the provider index still lists:

• Anthropic, plus the Claude Agent SDK
• Google models
• Amazon Bedrock
• Ollama for locally served models

Users on Hacker News asked what the deal means for teams testing Claude or Gemini through an OpenAI-owned tool. I read that as a procurement risk, not evidence of removed providers. The MIT license keeps a fork possible if neutrality slips; Enterprise buyers should get multi-provider support written into the contract.

How Does Promptfoo Compare to Competitors?

Promptfoo wins on free CLI evals and red teaming; LangSmith, Braintrust and Langfuse win when production tracing or hosted collaboration is the main requirement.

Tool	Starting paid price	Free tier	Best at
Promptfoo	Quote-only Enterprise	Community, 10,000 red-team probes/month	CI evals and red teaming
Confident AI / DeepEval	Starter $200/month; Team $2,000/month	Free tier	Python metrics and hosted reports
LangSmith	Plus $39/seat/month	Developer: 1 seat, 5,000 base traces/month	Production tracing for LangChain applications
Braintrust	Pro $249/month	Starter: 1 GB, 10,000 scores	Hosted evals and experiments, unlimited users
Langfuse	Core $29/month; Pro $199/month	Hobby	Self-hostable production tracing
Arize AX	Pro $50/month, 50,000 spans	AX Free	Production observability

• Promptfoo vs DeepEval: DeepEval wins when the team wants a Python metric library rather than YAML tests. Its hosted companion, Confident AI, adds reports and observability, but the jump from Starter at $200/month to Team at $2,000/month is steep. Promptfoo fits better when CI config and security scans belong together.
• Promptfoo vs LangSmith: LangSmith wins on production tracing, especially for LangChain apps. Developer is free for one seat and 5,000 base traces a month, while Plus at $39 per seat includes 10,000 traces and adds pay-as-you-go usage on top, so a ten-person team pays $390 a month before usage. Promptfoo’s OTLP assertions judge test runs, not a deployed service.
• Promptfoo vs Braintrust: Braintrust wins for hosted experiments and datasets shared between technical and non-technical people, with unlimited users on every tier. Promptfoo avoids the subscription, and the overage charges per GB and per 1,000 scores, when engineers mainly need version-controlled tests.
• Promptfoo vs Langfuse: Langfuse wins when self-hosted production tracing is mandatory, since Promptfoo’s Community server lacks authentication and scaling. Langfuse’s hosted plans add usage-unit charges to budget for, and SSO and RBAC sit in a paid add-on.
• Promptfoo vs Garak and PyRIT: NVIDIA’s Apache 2.0 Garak and Microsoft’s MIT-licensed PyRIT suit security teams that want free, code-first attack tooling. Promptfoo wins when scans must sit beside prompt comparisons in one results matrix and one CI job.

Arize AX is another production-observability pick, with Pro at $50/month for 50,000 spans. Galileo, now part of Splunk’s agent observability line, belongs on the same shortlist. My recommendation is Promptfoo before deployment and one tracing platform after release, rather than forcing either category to do both jobs.

How I Reviewed Promptfoo

I installed Promptfoo 0.124.1, ran the getting-started eval and the red-team setup flow, and worked through the assertion types, grader settings and the GitHub Action. I checked the telemetry, remote-generation and sharing settings against what a privacy-sensitive team needs before adoption, including what the remote-generation switch does not block.

I compared Community against Enterprise and On-Premise on probe limits, collaboration and deployment, then set Promptfoo against DeepEval/Confident AI, LangSmith, Braintrust, Langfuse, Arize, Garak and PyRIT. I also reviewed the release history from 0.122.2 to 0.124.1. I weighed sustained user sentiment from GitHub issues and Hacker News as specific failure cases and procurement questions, not as a measure of how often every team hits them.

Prices current as of October 2026.

Should You Use Promptfoo for LLM Evals and Red Teaming?

Promptfoo Community gives engineering teams the complete eval toolkit for free, plus pre-launch red teaming mapped to the OWASP LLM Top 10. Its gaps are production monitoring, the 10,000-probe cap and outbound-data defaults that need deliberate configuration before the first run.

• Engineering teams shipping LLM features: yes. Start on Community with YAML regression tests in CI and an explicit grader provider.
• Security teams red-teaming chatbots and agents: yes, with the owasp:llm preset, but track target requests against the 10,000-probe monthly cap.
• Teams needing production monitoring or non-technical reviewers: pair Promptfoo with Langfuse or LangSmith, or pick Braintrust for a hosted workspace with unlimited users.
• Teams wary of OpenAI ownership: the MIT license is fork insurance; get multi-provider commitments into any Enterprise contract.

Start with Promptfoo by running npx promptfoo@latest init --example getting-started, then replace the example prompts with your own application’s prompts and expected behavior.

FAQs

Is Promptfoo free?

Yes, Promptfoo Community is free forever, with every LLM eval feature and 10,000 red-team probes per month. Enterprise and On-Premise are custom quotes, and there is no self-serve paid tier between them. Model API calls on your own provider keys are billed separately by your provider, including model-graded assertions.

Is Promptfoo still open source after the OpenAI acquisition?

Yes, Promptfoo remains open source under the MIT license after joining OpenAI. Its provider index still includes Anthropic, Google, Bedrock and Ollama. Current support is not a contractual guarantee, though, so Enterprise buyers worried about model neutrality should ask for explicit multi-provider commitments.

Does Promptfoo send my data to its servers?

Yes, telemetry is on by default, red teaming can use hosted attack generation, and cloud sharing uploads eval snapshots. Telemetry excludes prompts and outputs, but shared snapshots contain them. Set PROMPTFOO_DISABLE_TELEMETRY=1 and PROMPTFOO_DISABLE_REDTEAM_REMOTE_GENERATION=true, and skip cloud sharing for sensitive evals.

What is a probe in Promptfoo?

A probe is one request to your target system during red teaming. Community includes 10,000 probes a month, not 10,000 scans, so multiple strategies and repeated scans drain the allowance through their target requests. Enterprise sets custom limits and sells additional probes when the Community cap is too tight.

Can Promptfoo replace LangSmith or Langfuse?

No, Promptfoo does not replace production observability from LangSmith or Langfuse. Its OpenTelemetry support feeds eval-time assertions about agent behavior, such as tool calls and execution paths. Use Promptfoo for regression tests and pre-launch scans, then a tracing platform for deployed applications.