Pipecat

PipecatSince 2024-03

Pipecat is a high-performance, open-source Python and TypeScript framework for building real-time voice, video, and multimodal conversational AI agents. Maintained by Daily.co, Pipecat abstracts the intricate pipeline of WebRTC transport, audio turn-taking, speech-to-text (STT), LLM streaming, and text-to-speech (TTS) into modular, composable services.

Pipecat solves the hardest challenges in real-time conversational agents: human interruption handling, sub-second latency, voice activity detection (VAD), and network jitter over WebRTC and WebSockets. It offers plug-and-play integrations with Deepgram, Cartesia, ElevenLabs, OpenAI Realtime API, Whisper, and Anthropic Claude, allowing developers to construct voice bots for telephony, customer support, and interactive robotics.

Moshi by KyutaiSince 2024-07

Moshi is an open-source real-time conversational voice AI foundation model developed by Kyutai, the non-profit AI research lab based in Paris. Engineered to revolutionize human-AI verbal communication, Moshi operates on a full-duplex architecture capable of listening, thinking, and speaking simultaneously with sub-200ms end-to-end latency. Unlike traditional voice assistants that chain separate Speech-to-Text (STT), Large Language Model (LLM), and Text-to-Speech (TTS) pipelines together, Moshi processes raw multi-stream audio natively as continuous speech tokens. This allows Moshi to understand emotional nuances, interrupt and be interrupted naturally, chuckle, whisper, and express genuine conversational timing. Moshi is fully open-source with openly accessible weights, training recipes, and inference code, serving as a foundational milestone for research in real-time spoken language modeling and multi-modal conversational systems.

Moshi’s architecture is built on Helium, a 7-billion parameter language model coupled with Mimi, a cutting-edge neural audio codec that compresses 24kHz audio into multi-stream discrete tokens at just 1.1 kbps. By operating on a joint text-audio token stream, Moshi predicts both conversational text tokens and acoustic speech tokens in parallel. This end-to-end audio modeling eliminates the latency bottlenecks and acoustic information loss inherent in cascading STT-LLM-TTS pipelines. The Kyutai team provides full PyTorch and Rust-based inference engines optimized for local GPU execution, enabling real-time full-duplex conversations on consumer-grade hardware (NVIDIA RTX 4090 or Apple Silicon Mac).