Promptfoo

PromptfooSince 2026-01

Promptfoo is an open-source CLI and evaluation engine designed for LLM quality assurance, automated red teaming, and security vulnerability scanning. It enables engineering teams to systematically test prompts, agents, and RAG pipelines against prompt injections, jailbreaks, PII leakage, and hallucinations before releasing to production. With Promptfoo, developers write declarative test suites in YAML or JSON, defining test cases, assertion criteria (semantic similarity, regex, LLM-as-a-judge, toxicity), and scoring matrices that integrate directly into GitHub Actions CI/CD pipelines.

Promptfoo supports over 30 LLM providers and custom HTTP endpoints, allowing teams to test OpenAI, Anthropic, Gemini, Bedrock, and self-hosted models side by side. It includes automated adversarial red teaming plugins that generate hundreds of dynamic attack vectors (OWASP Top 10 for LLMs, prompt leak, SSRF, indirect prompt injection). Results can be viewed in a local web viewer or exported as JUnit XML, JSON, and CSV for automated pull request quality gates.

vLLMSince 2023-06

vLLM is the industry-standard open-source LLM serving and inference engine designed for ultra-high throughput and minimal memory waste. Developed by UC Berkeley researchers, vLLM introduced PagedAttention—a revolutionary memory management algorithm that manages attention key-value (KV) cache like virtual memory in operating systems, virtually eliminating memory fragmentation. Capable of delivering 2x to 4x higher throughput than Hugging Face TGI and standard PyTorch runtimes, vLLM powers production AI inference infrastructure across enterprise cloud clusters and high-volume API providers worldwide.

vLLM features state-of-the-art inference optimizations including continuous request batching, Chunked Prefill, speculative decoding, prefix caching, and native quantization support (AWQ, GPTQ, FP8, INT4, SqueezeLLM). It provides drop-in OpenAI-compatible REST API endpoints, supports multi-GPU distributed tensor parallelism with Ray/NCCL, and serves all major model architectures including DeepSeek-V3, Llama 3.3, Mistral, Qwen 2.5, and Command R+.