Promptfoo

PromptfooSince 2026-01

Promptfoo is an open-source CLI and evaluation engine designed for LLM quality assurance, automated red teaming, and security vulnerability scanning. It enables engineering teams to systematically test prompts, agents, and RAG pipelines against prompt injections, jailbreaks, PII leakage, and hallucinations before releasing to production. With Promptfoo, developers write declarative test suites in YAML or JSON, defining test cases, assertion criteria (semantic similarity, regex, LLM-as-a-judge, toxicity), and scoring matrices that integrate directly into GitHub Actions CI/CD pipelines.

Promptfoo supports over 30 LLM providers and custom HTTP endpoints, allowing teams to test OpenAI, Anthropic, Gemini, Bedrock, and self-hosted models side by side. It includes automated adversarial red teaming plugins that generate hundreds of dynamic attack vectors (OWASP Top 10 for LLMs, prompt leak, SSRF, indirect prompt injection). Results can be viewed in a local web viewer or exported as JUnit XML, JSON, and CSV for automated pull request quality gates.

RagasSince 2026-01

Ragas (Retrieval Augmented Generation Assessment) is the industry-standard evaluation framework designed specifically to measure the performance of RAG pipelines without requiring human-annotated ground truth datasets. Ragas evaluates RAG systems across critical dimensions: Faithfulness (hallucination detection), Answer Relevance (query alignment), Context Precision (signal-to-noise ratio in retrieved chunks), and Context Recall (measuring whether all necessary information was retrieved).

Ragas also includes powerful synthetic test data generation capabilities (Ragas Testset Generation), creating diverse multi-hop questions, reasoning challenges, and adversarial probes from raw document corpora automatically. It integrates natively with LangChain, LlamaIndex, Haystack, and DSPy, enabling continuous evaluation loops in production monitoring and pre-deployment automated CI gates.