Promptfoo

PromptfooSince 2026-01

Promptfoo is an open-source CLI and evaluation engine designed for LLM quality assurance, automated red teaming, and security vulnerability scanning. It enables engineering teams to systematically test prompts, agents, and RAG pipelines against prompt injections, jailbreaks, PII leakage, and hallucinations before releasing to production. With Promptfoo, developers write declarative test suites in YAML or JSON, defining test cases, assertion criteria (semantic similarity, regex, LLM-as-a-judge, toxicity), and scoring matrices that integrate directly into GitHub Actions CI/CD pipelines.

Promptfoo supports over 30 LLM providers and custom HTTP endpoints, allowing teams to test OpenAI, Anthropic, Gemini, Bedrock, and self-hosted models side by side. It includes automated adversarial red teaming plugins that generate hundreds of dynamic attack vectors (OWASP Top 10 for LLMs, prompt leak, SSRF, indirect prompt injection). Results can be viewed in a local web viewer or exported as JUnit XML, JSON, and CSV for automated pull request quality gates.

Cosine GenieSince 2024-08

Cosine Genie is a next-generation autonomous software engineering agent built to tackle complex software bugs, refactors, and feature requests. Operating with deep semantic understanding of massive codebases, Genie analyzes repository architectures, creates comprehensive execution plans, and generates multi-file diffs that pass existing continuous integration test suites. Designed to bridge the gap between AI code completion and full software development lifecycle automation, Genie mimics human engineering workflows by navigating dependency graphs, testing assumptions in sandboxed environments, and autonomously self-correcting logic errors before opening pull requests.

Genie is powered by Cosine's proprietary fine-tuned reasoning models and multi-agent orchestration framework, benchmarked at top-tier performance on the industry-standard SWE-bench. The system integrates directly with GitHub and GitLab, indexing abstract syntax trees (ASTs), commit histories, and documentation to maintain contextual awareness. Its sandboxed runtime spins up localized build containers to execute unit tests, compile artifacts, and measure regression risks in real time. Engineering teams use Genie to triage backlogs, automate dependency upgrades, and accelerate code reviews with verified pull request generation.