DeepEval

DeepEvalSince 2026-01

DeepEval is an open-source LLM evaluation framework built by Confident AI that feels like 'Pytest for LLMs'. It allows AI engineers to write unit tests for prompts, RAG pipelines, and conversational agents with production-ready evaluation metrics that execute seamlessly in local terminals and CI/CD pipelines. DeepEval includes 14+ research-backed evaluation metrics covering hallucination, G-Eval, toxicity, answer relevancy, bias, summarization quality, and tool correctness.

DeepEval tests can be run directly using the standard 'pytest' command line. It integrates with Confident AI's cloud dashboard to track metric drift over time, analyze test run regressions, and debug failing test cases with root-cause explanations. It supports custom LLM judges, synthetic dataset synthesis, and synthetic edge-case generation to stress-test AI systems prior to enterprise deployment.

PromptfooSince 2026-01

Promptfoo is an open-source CLI and evaluation engine designed for LLM quality assurance, automated red teaming, and security vulnerability scanning. It enables engineering teams to systematically test prompts, agents, and RAG pipelines against prompt injections, jailbreaks, PII leakage, and hallucinations before releasing to production. With Promptfoo, developers write declarative test suites in YAML or JSON, defining test cases, assertion criteria (semantic similarity, regex, LLM-as-a-judge, toxicity), and scoring matrices that integrate directly into GitHub Actions CI/CD pipelines.

Promptfoo supports over 30 LLM providers and custom HTTP endpoints, allowing teams to test OpenAI, Anthropic, Gemini, Bedrock, and self-hosted models side by side. It includes automated adversarial red teaming plugins that generate hundreds of dynamic attack vectors (OWASP Top 10 for LLMs, prompt leak, SSRF, indirect prompt injection). Results can be viewed in a local web viewer or exported as JUnit XML, JSON, and CSV for automated pull request quality gates.