Promptfoo Review (2026): Features, Pricing & Verdict
Promptfoo is an open-source, developer-first tool for evaluating and — increasingly its headline act — red-teaming LLM applications, and it has quietly become one of the most widely adopted tools of its kind. At its simplest, it’s a command-line tool and library that lets you replace the risky habit of “vibe checks” — eyeballing two or three model outputs and calling it done — with systematic, repeatable testing: you write declarative YAML test cases, run them against your prompts, agents and RAG pipelines across more than 50 model providers, and get a clear side-by-side comparison of what works. But what has made Promptfoo genuinely distinctive in 2026 is its red-teaming module: with a single command it auto-generates hundreds of context-aware adversarial attacks tailored to your specific application — prompt injection, jailbreaks, PII leakage, SSRF, SQL and shell injection, excessive agency and more — drawing on 50-plus attack plugins and shipping presets that map directly to the OWASP LLM Top 10, the OWASP Agentic Top 10, the NIST AI Risk Management Framework and MITRE ATLAS, then grades each response and renders a compliance-style vulnerability report. It’s the same class of adversarial testing the big foundation labs run before releasing models — and, tellingly, both OpenAI and Anthropic use Promptfoo internally. Its pedigree backs that up: around 10,400 GitHub stars, more than 350,000 developers, 130,000 monthly active users and adoption at roughly a quarter of the Fortune 500, all built on tooling originally created for LLM apps serving over 10 million users in production. It’s MIT-licensed, runs completely locally (your data never leaves your machine), is CI/CD-native so a failing prompt can block a deploy, and mixes deterministic assertions (regex, contains, JSON schema, cost and latency thresholds) with LLM-as-a-judge grading. One major development shapes any 2026 assessment: OpenAI acquired the Promptfoo company in March 2026 — the project remains open source under MIT, but the acquisition, and Promptfoo’s deliberate scope as a pre-deployment tool with no production monitoring, are the two things to weigh, both covered below.
- Best for
- Developers, SDETs and AI security teams who want systematic, CI/CD-native prompt evaluation, side-by-side model comparison, and — above all — automated LLM red teaming and vulnerability scanning mapped to OWASP, NIST and MITRE frameworks, all running locally
- Platform
- Open-source CLI & library (Node.js; Python/JS/any language) for LLM evaluation and red teaming — declarative YAML tests, 50+ providers, deterministic + LLM-as-a-judge assertions, 50+ attack plugins, web UI viewer & compliance reports; runs locally, plus an optional cloud/enterprise platform
- Key differentiator
- The best-in-class open-source LLM red-teaming engine — one command auto-generates hundreds of context-aware adversarial attacks with OWASP LLM Top 10, NIST AI RMF and MITRE ATLAS presets — combined with a developer experience teams actually adopt
- Pricing
- Open-source Community free (MIT, all core features, self-host); Cloud free tier for individuals; Team ~$50/month; Enterprise custom. Plus your own LLM API token costs for eval & grading runs
- Vendor
- Promptfoo — acquired by OpenAI (March 2026; project stays MIT open source). Previously valued at $86M (July 2025 Series A). ~10.4k GitHub stars, 350k+ developers, ~25% of Fortune 500; used internally by OpenAI & Anthropic
What Is Promptfoo?
Promptfoo is a CLI-first, open-source framework for evaluating LLM outputs and testing AI applications before they ship. The core idea is to bring the discipline of software testing to prompt engineering: instead of manually reviewing a handful of outputs and hoping for the best, you define your test cases declaratively in YAML, specify the providers you want to test against (OpenAI, Anthropic, Ollama, Azure, Bedrock, Mistral, Groq, Cohere and dozens more), attach assertions that encode what a good answer looks like, and run everything from the command line to get a clear, shareable comparison. It serves two closely related jobs. The first is evaluation — test-driven prompt engineering, model selection and regression testing, so you can see objectively which prompt or which model performs best for your task and catch quality regressions before they reach users. The second, and the one that has driven much of its 2026 prominence, is red teaming and vulnerability scanning: systematically probing your application with adversarial inputs to find security and safety weaknesses before attackers or users do. Both run the same way — locally, from the CLI, wired into CI/CD — which is central to Promptfoo’s philosophy: security and quality tools that only specialists run don’t stop bugs from shipping, whereas a tool that sits in the build pipeline and can block a deploy actually does. That developer-workflow fit is why it has spread so widely; it’s described as “the tool your dev team will actually adopt,” and it now appears in senior SDET and AI Test Engineer job descriptions more than any other LLM evaluation framework. Within this site’s Machine Learning & MLOps category, Promptfoo occupies a distinctive niche in the LLM-evaluation cluster. Where tools like Confident AI, Braintrust and Langfuse aim to be broad quality platforms spanning evaluation, observability and production monitoring, Promptfoo is deliberately a pre-deployment testing-and-security tool — narrower in scope, but best-in-class within that scope, and unmatched on developer adoption and red-teaming depth. It’s the tool you reach for to test and harden an application before it goes live, and to keep testing it in CI as it evolves.
Core Features
Evaluation: declarative YAML tests and side-by-side model comparison
Promptfoo’s evaluation engine is built around a simple, powerful premise: define your tests once, in plain declarative YAML, and run them anywhere. You don’t need to write code or wrangle heavy notebooks to get started — you describe your prompts, the providers you want to test, and a set of assertions, and Promptfoo executes the matrix and shows you the results in a clean web viewer. The assertion system is where a lot of the depth lives, and it’s deliberately two-sided. Deterministic assertions cover the things that have a right answer: exact match, contains, regex, JSON schema validation, plus cost thresholds and latency limits so you can fail a test not just on wrong content but on a response that’s too expensive or too slow. Model-graded assertions cover the things that don’t: LLM-as-a-judge evaluations, where you use one model to assess another’s output on quality, tone, correctness or any criterion you define, and you can even write fully custom assertion functions in JavaScript or Python when the built-ins aren’t enough. The practical guidance that experienced users follow is to combine the two — deterministic assertions for safety and format validation, LLM-as-a-judge for quality and tone — because together they give coverage neither approach achieves alone. The other standout evaluation capability is side-by-side model comparison across more than 50 providers. This is genuinely valuable in 2026’s crowded model market: when you’re deciding between Claude, GPT, Gemini, DeepSeek and open-source alternatives for a specific task, Promptfoo runs the same test suite against all of them and produces a comparison table showing exactly which performs best for your use case, on your data — a far more reliable basis for a decision than public benchmarks. Because it outputs CI-compatible formats, the whole thing slots into GitHub Actions, GitLab CI or Jenkins, so a prompt change that causes a regression fails the build before it reaches production. And because it runs completely locally, talking directly to the LLM providers, your prompts and test data never leave your environment — a real advantage for privacy-sensitive teams.
Red teaming: automated adversarial testing and compliance mapping
Red teaming is Promptfoo’s most distinctive feature in 2026, and it’s genuinely best-in-class among open-source tools. The problem it addresses is that LLM applications have an enormous, stochastic attack surface: depending on your architecture, you face off-topic or harmful content generation, information leakage and access-control failures in RAG systems, misuse of connected APIs and databases in agents, and classic injection attacks adapted for AI. Manually probing for all of that is impossible, so Promptfoo automates it. With a single command it reads your prompts, targets and business logic, then generates hundreds of context-aware adversarial test cases tailored specifically to your application — not generic ones. The engine is built on three concepts. Plugins generate adversarial inputs for specific vulnerability classes: there are more than 50, covering prompt injection, jailbreaks, PII leakage through direct, session and social vectors, SSRF, SQL injection, shell injection, hijacking, excessive agency, hallucination and harmful-content elicitation, plus custom policy violations you define. Framework presets bundle related plugins into whole compliance suites — one line enabling the full OWASP LLM Top 10, or the newer OWASP Agentic Top 10, with additional mappings to the NIST AI Risk Management Framework and MITRE ATLAS. And strategies determine how the attacks are delivered: simple injection framing, DAN-style jailbreak techniques, or multi-turn “crescendo” escalation where each message builds on the last. Once generated, the scan runs every test case against your target — including live API endpoints and custom RAG or agent flows — grades each response with an LLM judge (with the grader model configurable, so you can run it locally), and renders everything in a web UI: red where your model broke, green where it held, with click-through to the exact adversarial prompt, the response and the grader’s reasoning. The payoff is the compliance report card: your results mapped against OWASP, NIST and MITRE controls, with severity ratings and evidence trails, so you can hand stakeholders a document that says precisely which categories passed and which failed. It can run as a one-off assessment or continuously in CI/CD, which is where it delivers the most value — turning AI security from a periodic audit into a standing gate.
Developer experience, integrations and deployment
Much of Promptfoo’s success comes down to a deceptively simple thing: developers actually use it, because it’s built to fit the way they already work. It’s a CLI and a library, not a heavyweight platform you have to log into — you install it with a single npm, brew or pip command (or run it with npx without installing at all), configure everything in version-controlled YAML that lives alongside your code, and run it locally or in CI. That local-first, declarative design is why it slots so naturally into existing pipelines and why teams adopt it rather than working around it. It’s language-agnostic for the application under test — your app can be written in Python, JavaScript or anything else, since Promptfoo talks to it over its interface rather than requiring you to rewrite in a particular framework — and it supports a very wide provider list, so whether you’re on OpenAI, Anthropic, Mistral, Azure, Bedrock, Groq, Perplexity, Cohere, Hugging Face or local models via Ollama, you set an API key and go. For applications built on LangChain or LlamaIndex, Promptfoo can test the prompts those frameworks generate; for live systems, you point it at an HTTP endpoint with a placeholder variable and it pentests the running API; and it now extends into adjacent areas including code scanning, model security, guardrails and MCP support. Collaboration is handled through a built-in share function and web viewer, and there’s an optional cloud platform and enterprise tier that add team-based access control, SSO, granular permissions, shared configurations, real-time security alerts across all your projects, remediation tracking and compliance verification for organisations managing many applications. On deployment, the open-source Community version includes all core features for local testing, evaluation and vulnerability scanning at no cost, while the enterprise option can run entirely within your own infrastructure for organisations that require full control. The one piece of friction worth naming: Promptfoo requires Node.js, so Java- or Python-only teams have to add it to their stack — the Python package is a wrapper that still runs the underlying Node implementation. For most teams that’s a minor cost against what they get; for some it’s a genuine consideration.
Scored Categories
Pricing
| Tier | Price | Notes |
|---|---|---|
| Community (open source) | Free (MIT) | All core features for local evaluation, red teaming & vulnerability scanning — no feature restrictions, self-hosted, runs entirely on your machine |
| Cloud — Free | Free | Hosted free tier for individual use — shared results and web viewer without self-managing infrastructure |
| Team | ~$50 / month | Collaboration, shared results, team management and cloud convenience for small teams (confirm current pricing with Promptfoo) |
| Enterprise | Custom quote | Full infrastructure control / self-hosting, SSO, RBAC & granular permissions, real-time cross-project alerts, remediation tracking, compliance verification & support |
Strengths
- Best-in-class open-source LLM red teaming — 50+ attack plugins, one-command adversarial scans
- Compliance presets for OWASP LLM Top 10, OWASP Agentic, NIST AI RMF & MITRE ATLAS
- Vulnerability report cards with severity ratings and click-through evidence trails
- Exceptional developer experience — declarative YAML, CLI-first, version-controlled config
- CI/CD-native — fail a build on a regression or a security finding before it ships
- Side-by-side model comparison across 50+ providers for objective model selection
- Deterministic assertions and LLM-as-a-judge grading, plus custom JS/Python assertions
- Open-source (MIT), free self-host, all core features, runs completely locally (private)
- Massive adoption & pedigree — 350k+ developers, ~25% of Fortune 500, used by OpenAI & Anthropic
- Pentests live API endpoints and custom RAG/agent flows; code scanning, guardrails & MCP
Weaknesses
- Pre-deployment tool by design — no production monitoring, live traffic tracking or observability
- OpenAI acquisition (March 2026) raises a neutrality question for multi-provider comparison
- Requires Node.js — added friction for Java- or Python-only teams (pip is a wrapper)
- LLM-as-a-judge evals and red-team grading incur API token costs that scale with suite size
- Overkill for teams using a single model and single prompt where eval overhead isn’t worth it
- Not a broad quality platform — pair it with an observability tool for production coverage
- Web collaboration and enterprise governance sit in paid tiers
Verdict: 8.3 / 10 — The Developer-First Standard for LLM Testing & Red Teaming
Promptfoo earns a strong 8.3 and a clear identity: it is the open-source, developer-first standard for evaluating and — above all — red-teaming LLM applications before they ship. Its red-teaming engine is genuinely best-in-class among open tools: one command generates hundreds of context-aware adversarial attacks from a library of 50-plus plugins, maps them to the OWASP LLM Top 10, OWASP Agentic, NIST AI RMF and MITRE ATLAS, grades every response and hands you a compliance report card with severity ratings and evidence — the same kind of adversarial testing the frontier labs run, packaged so an ordinary dev team can run it in CI. Around that sits a superb evaluation framework: declarative YAML tests, deterministic plus LLM-as-a-judge assertions, and side-by-side comparison across more than 50 providers that makes model selection an evidence-based decision. What truly sets Promptfoo apart, though, is adoption — 350,000-plus developers, roughly a quarter of the Fortune 500, and internal use at both OpenAI and Anthropic — earned by a local-first, CI/CD-native design that developers actually embrace rather than route around. Two honest factors keep it a notch below the broadest quality platforms rather than at the very top. First, scope: Promptfoo is deliberately a pre-deployment tool, with no production monitoring, no live-traffic tracking and no observability — so it can’t be your single quality platform the way Confident AI or Langfuse can; you’ll pair it with an observability tool for the production half. Second, the March 2026 OpenAI acquisition: the project remains MIT open source and there’s no sign of bias, but a neutral cross-provider comparison tool now owned by one of the providers it compares is a fair thing to weigh, even if the MIT licence keeps a community fork always available. Add the minor Node.js requirement and the per-run judge costs, and you have the full picture. The clean verdict: if you build LLM applications and care about shipping them securely and reliably, Promptfoo is close to essential — the best open-source red-teaming tool available and an excellent evaluation framework, with a developer experience that’s second to none. Just treat it as the pre-deployment testing-and-security layer it is, pair it with production observability, and stay aware of the ownership change.
Frequently Asked Questions
What does the OpenAI acquisition mean for Promptfoo?
OpenAI acquired the Promptfoo company on 9 March 2026, with the terms undisclosed; Promptfoo had been valued at around $86 million at its Series A in July 2025. The single most important practical fact is that the project remains open source under the permissive MIT licence and continues to develop actively — the acquisition changed who owns the company, not the licensing or availability of the tool, and its repository stays public. It’s also worth noting that both OpenAI and Anthropic were already using Promptfoo internally before the acquisition, which speaks to its quality and neutrality up to that point. The consideration the acquisition raises is a reasonable one about neutrality, and it’s specific to one part of what Promptfoo does. A core Promptfoo use case is neutral, side-by-side comparison of competing model providers — running the same test suite against GPT, Claude, Gemini, DeepSeek and others to see which performs best for your task. When the tool that arbitrates those comparisons is owned by one of the providers being compared, some users understandably want to keep an eye on whether that independence is maintained over time. To be clear, there’s no evidence of any bias having been introduced, the grading models are configurable (you can use a non-OpenAI or local model as the judge), and the tool’s methodology is open for inspection precisely because it’s open source. And the MIT licence provides a genuine backstop: if the community ever felt the project’s direction compromised its neutrality, the code can be forked and maintained independently, as has happened with other open-source projects after acquisitions. So the honest summary is: the acquisition brings more resources and doesn’t change the licence or availability, both major labs trust the tool, and while a neutrality question is fair to hold in mind for cross-provider comparisons, the open-source nature and configurable graders substantially mitigate it. For red teaming and evaluation of your own application — as opposed to comparing providers — the acquisition is largely immaterial.
Does Promptfoo monitor LLM applications in production?
No — and this is the single most important thing to understand about where Promptfoo fits, because it’s a deliberate design choice rather than a gap. Promptfoo is intentionally a pre-deployment tool. It sits between your development workflow and production deployment, and its job is to test and harden your application before it goes live, and to keep testing it in CI as it evolves. It does not monitor live production traffic, it does not alert on real-time quality degradation in production, and it has no observability or tracing features for inspecting what’s happening to requests once your app is serving real users. What Promptfoo does exceptionally well is everything up to the deploy gate: systematic evaluation of prompts and models, side-by-side comparison, regression testing that can fail a build, and automated red-team vulnerability scanning — run as one-off assessments or continuously in your CI/CD pipeline. Think of it as the equivalent of unit tests, integration tests and a security scanner in traditional software: they run before and during the release process to stop bad code shipping, but they’re not your production monitoring dashboard. The practical implication is that if you want full lifecycle coverage, you pair Promptfoo with a production observability platform. Tools like Langfuse, Arize, or an evaluation platform such as Confident AI provide the live tracing, real-time monitoring and alerting that Promptfoo deliberately doesn’t — capturing every production request, tracking quality over time, and alerting you when something degrades after launch. A common and effective setup is Promptfoo in the pipeline as the pre-deployment testing-and-security gate, plus an observability tool watching production. This isn’t a weakness so much as a scoping decision: by staying focused on pre-deployment testing and red teaming, Promptfoo does that job better than almost anything else, and it doesn’t try to be a sprawling all-in-one platform. Just go in knowing you’ll need a second tool for the production-monitoring half of the lifecycle.
Do I need to know how to code to use Promptfoo?
Mostly no for basic use, with some nuance. Promptfoo is built around declarative YAML configuration, which means that for a great many use cases you define your tests — your prompts, the providers to test against, and the assertions that describe a good answer — in a structured text file rather than by writing programs. Setting up a straightforward evaluation or kicking off a red-team scan is largely a matter of editing that YAML config and running a command, and the red-team setup in particular ships with an init command and sensible defaults that guide you through configuring a scan for your use case, so you can get a meaningful vulnerability report without writing evaluation logic yourself. In that sense, the barrier is much lower than notebook-based or code-first evaluation frameworks, and the YAML approach maps neatly onto skills that QA engineers and SDETs already have from traditional test automation — which is part of why Promptfoo appears so often in AI test-engineer job descriptions. That said, there are a few practical realities. First, Promptfoo is a command-line tool that requires Node.js to be installed, so you do need to be comfortable working in a terminal and setting environment variables for your LLM provider API keys — that’s more technical than a pure point-and-click web app. Second, while the built-in assertion types (exact match, contains, regex, JSON schema, cost and latency limits, and LLM-as-a-judge grading) cover a lot of ground declaratively, if you want fully custom evaluation logic you can write assertion functions in JavaScript or Python, and that part does require coding. Third, the initial integration — pointing Promptfoo at a live API endpoint or a custom RAG/agent flow — benefits from some engineering familiarity. So the honest answer: prompt writers and QA specialists can absolutely define and run tests and scans through YAML without being software engineers, especially once the initial setup exists, and there’s an optional cloud UI that lowers the barrier further; but the tool lives in the developer’s world of terminals, config files and CI pipelines, so a completely non-technical user may want an engineer to handle the initial setup.