Moshi by Kyutai

Moshi by KyutaiSince 2024-07

Moshi is an open-source real-time conversational voice AI foundation model developed by Kyutai, the non-profit AI research lab based in Paris. Engineered to revolutionize human-AI verbal communication, Moshi operates on a full-duplex architecture capable of listening, thinking, and speaking simultaneously with sub-200ms end-to-end latency. Unlike traditional voice assistants that chain separate Speech-to-Text (STT), Large Language Model (LLM), and Text-to-Speech (TTS) pipelines together, Moshi processes raw multi-stream audio natively as continuous speech tokens. This allows Moshi to understand emotional nuances, interrupt and be interrupted naturally, chuckle, whisper, and express genuine conversational timing. Moshi is fully open-source with openly accessible weights, training recipes, and inference code, serving as a foundational milestone for research in real-time spoken language modeling and multi-modal conversational systems.

Moshi’s architecture is built on Helium, a 7-billion parameter language model coupled with Mimi, a cutting-edge neural audio codec that compresses 24kHz audio into multi-stream discrete tokens at just 1.1 kbps. By operating on a joint text-audio token stream, Moshi predicts both conversational text tokens and acoustic speech tokens in parallel. This end-to-end audio modeling eliminates the latency bottlenecks and acoustic information loss inherent in cascading STT-LLM-TTS pipelines. The Kyutai team provides full PyTorch and Rust-based inference engines optimized for local GPU execution, enabling real-time full-duplex conversations on consumer-grade hardware (NVIDIA RTX 4090 or Apple Silicon Mac).

LiveKit AgentsSince 2026-08

LiveKit Agents is an open-source real-time communication framework engineered to build conversational voice, video, and multimodal AI agents with sub-500ms latency. Leveraging WebRTC, it connects speech-to-text (Deepgram, Whisper), LLMs (OpenAI, Anthropic), and text-to-speech (Cartesia, ElevenLabs) in a tightly synchronized bidirectional stream. From customer service avatars to interactive language tutors and hands-free coding copilots, LiveKit Agents provides the enterprise infrastructure for real-time human-AI interaction.

Building real-time voice agents requires solving audio interruption, packet jitter, and latency stacking. LiveKit Agents abstracts these challenges with native Voice Activity Detection (VAD), turn-taking management, and edge-routed audio pipelines. Available in Python and Node.js with client SDKs across React, iOS, Android, and Flutter, LiveKit enables developers to self-host their agent backend or deploy onto LiveKit Cloud.