Moshi by Kyutai

Moshi by KyutaiSince 2024-07

Moshi is an open-source real-time conversational voice AI foundation model developed by Kyutai, the non-profit AI research lab based in Paris. Engineered to revolutionize human-AI verbal communication, Moshi operates on a full-duplex architecture capable of listening, thinking, and speaking simultaneously with sub-200ms end-to-end latency. Unlike traditional voice assistants that chain separate Speech-to-Text (STT), Large Language Model (LLM), and Text-to-Speech (TTS) pipelines together, Moshi processes raw multi-stream audio natively as continuous speech tokens. This allows Moshi to understand emotional nuances, interrupt and be interrupted naturally, chuckle, whisper, and express genuine conversational timing. Moshi is fully open-source with openly accessible weights, training recipes, and inference code, serving as a foundational milestone for research in real-time spoken language modeling and multi-modal conversational systems.

Moshi’s architecture is built on Helium, a 7-billion parameter language model coupled with Mimi, a cutting-edge neural audio codec that compresses 24kHz audio into multi-stream discrete tokens at just 1.1 kbps. By operating on a joint text-audio token stream, Moshi predicts both conversational text tokens and acoustic speech tokens in parallel. This end-to-end audio modeling eliminates the latency bottlenecks and acoustic information loss inherent in cascading STT-LLM-TTS pipelines. The Kyutai team provides full PyTorch and Rust-based inference engines optimized for local GPU execution, enabling real-time full-duplex conversations on consumer-grade hardware (NVIDIA RTX 4090 or Apple Silicon Mac).

Fish AudioSince 2024-03

Fish Audio is an open-source text-to-speech (TTS) and voice cloning platform built on state-of-the-art auto-regressive transformer models. Engineered to deliver sub-150ms voice generation latencies with natural human inflection, Fish Audio allows developers, creators, and voice agents to generate studio-grade audio across dozens of global languages from a single reference sample. The platform's zero-shot voice cloning engine requires as little as 10 to 30 seconds of clean reference audio to accurately replicate pitch, accent, emotional timbre, and speaking rhythm. With support for bilingual code-switching, emotional tone control (excited, whisper, solemn), and real-time streaming WebSockets, Fish Audio powers conversational voice AI applications, video game character voiceovers, and dynamic audiobook narration. Fish Audio provides both a managed cloud platform with intuitive web interfaces and self-hostable open-weight model checkpoints, giving enterprise developers full control over data privacy, on-premise compute deployment, and model fine-tuning.

Fish Audio is architected around a dual-component neural pipeline comprising an auto-regressive acoustic model and a high-fidelity vocoder. By tokenizing reference audio through a continuous vector-quantized latent space, the model preserves acoustic subtleties without requiring extensive multi-speaker training datasets. Developers interact with Fish Audio via high-performance REST APIs, Python/TypeScript SDKs, and streaming WebSockets designed for live conversational AI pipelines. The API provides granular parameters for temperature, top-p sampling, pitch scaling, speech rate modulation, and custom pronunciation lexicons via SSML tags. For high-throughput production environments, Fish Audio supports GPU-accelerated inference across NVIDIA TensorRT and vLLM-compatible backends, enabling parallel synthesis across thousands of concurrent agent dialogue streams.