Promptfoo

What Promptfoo is

Promptfoo is an open source CLI and library for evaluating and red-teaming LLM applications, RAG systems, and agents. It was founded by Ian Webster and Michael D’Angelo and built on tooling originally designed to serve production systems at over 10 million users.

On March 9, 2026, Promptfoo agreed to be acquired by OpenAI. The open-source project continues under the same maintainers, now as part of OpenAI. As of that date the project had 21.8k GitHub stars and was in active use by 156 Fortune 500 companies.

Source: Promptfoo | Joining OpenAI announcement

Product surface

Promptfoo is not just an eval runner it has grown into a full AI security testing platform with six distinct product areas:

Evals

YAML-defined test suites that run prompts through one or more LLM providers and score outputs against configurable assertions and metrics. Supports custom Python/JS scorers, LLM-as-judge, factual grounding checks, and regression detection. Runs via CLI, as a library, or in CI/CD.

Red Teaming

Automated adversarial testing for agents and RAGs. Promptfoo generates application-specific attack scenarios and simulates real user behavior to surface vulnerabilities such as prompt injection, tool misuse, data exfiltration, and jailbreaks. Supports indirect prompt injection via web-browsing agents (the indirect-web-pwn strategy).

Code Scanning

A GitHub Action that scans pull requests for LLM-specific vulnerabilities. Traces data flows through application code to find prompt injection risks that traditional static analysis tools miss.

Model Audit (ModelAudit)

Open-sourced scanner for ML model files. Checks 42+ model formats for unsafe loading behaviors, known CVEs, and suspicious artifacts. Useful for supply chain security when consuming third-party models.

Guardrails

Runtime protection layer that can be configured to block unsafe model outputs in production.

MCP Proxy

Security layer for Model Context Protocol (MCP) connections. Tests and mediates tool use by agents operating over MCP.

Why it fits Ikidna

Ikidna treats quality and governance as first-class concerns across its context strategy. Promptfoo addresses several layers of that concern:

• Eval-driven delivery YAML-defined test suites can become release gates, so skill and context changes are validated before rollout
• Regression detection catches quality or behavior drift as models, context, or workflows evolve
• CI/CD integration runs natively in delivery pipelines alongside other quality checks
• Agent-focused security red teaming and MCP proxy coverage addresses Ikidna’s agentic architecture directly
• Code scanning catches injection risks at the PR level before they reach runtime
• OpenAI alignment as Ikidna consumes OpenAI models, Promptfoo’s integration with OpenAI’s ecosystem (including the OpenAI Build Hours endorsement) is a relevant signal

In Ikidna terms, Promptfoo best serves as the application-level and security evaluation layer validating end-to-end agent behavior, prompt quality, and security posture.

Relationship to other evaluation tooling

Promptfoo operates at the application and safety layer, distinct from skill-level evaluation:

• Promptfoo application-centric: validates end-to-end agent behavior, prompt quality, and security posture across the full request pipeline
• Tessl context packaging and distribution: versioned skill registry with evaluation hooks

A complete Ikidna evaluation pipeline can use both: Promptfoo for application-level and security checks, and Tessl for cross-agent context distribution.