Moshi by Kyutai

Moshi by KyutaiSince 2024-07

Moshi is an open-source real-time conversational voice AI foundation model developed by Kyutai, the non-profit AI research lab based in Paris. Engineered to revolutionize human-AI verbal communication, Moshi operates on a full-duplex architecture capable of listening, thinking, and speaking simultaneously with sub-200ms end-to-end latency. Unlike traditional voice assistants that chain separate Speech-to-Text (STT), Large Language Model (LLM), and Text-to-Speech (TTS) pipelines together, Moshi processes raw multi-stream audio natively as continuous speech tokens. This allows Moshi to understand emotional nuances, interrupt and be interrupted naturally, chuckle, whisper, and express genuine conversational timing. Moshi is fully open-source with openly accessible weights, training recipes, and inference code, serving as a foundational milestone for research in real-time spoken language modeling and multi-modal conversational systems.

Moshi’s architecture is built on Helium, a 7-billion parameter language model coupled with Mimi, a cutting-edge neural audio codec that compresses 24kHz audio into multi-stream discrete tokens at just 1.1 kbps. By operating on a joint text-audio token stream, Moshi predicts both conversational text tokens and acoustic speech tokens in parallel. This end-to-end audio modeling eliminates the latency bottlenecks and acoustic information loss inherent in cascading STT-LLM-TTS pipelines. The Kyutai team provides full PyTorch and Rust-based inference engines optimized for local GPU execution, enabling real-time full-duplex conversations on consumer-grade hardware (NVIDIA RTX 4090 or Apple Silicon Mac).

ElevenLabsSince 2022-10

A leading AI voice platform for generating ultra-realistic speech, voice cloning, and multilingual audio content.

Supports 30+ languages with emotion control. Used in podcasts, audiobooks, and games.