MiniCPM-o 4.5

MiniCPM-o 4.5 delivers strong speech capability through its integrated audio encoder and decoder, supporting bilingual real-time speech conversation in both English and Chinese. The model features configurable voices and the ability to clone any speaker's voice using a simple reference audio clip, a capability that surpasses dedicated TTS tools like CosyVoice2. This is made possible by a configurable speech modeling design that inherits a multimodal system prompt structure: a traditional text system prompt combined with an audio system prompt specifying the assistant's voice. During inference, the model can switch between voices or adopt new ones based on the provided reference, enabling creative applications such as role-play or personalized assistants. The result is more natural, expressive, and stable speech interactions that feel genuinely human-like.

A defining feature of MiniCPM-o 4.5 is its new full-duplex multimodal live streaming capability, which allows it to process real-time continuous video and audio input streams while generating concurrent text and speech output streams in an end-to-end fashion. This is achieved through a time-division multiplexing mechanism that synchronizes all input and output streams on the millisecond timeline within the LLM backbone. The model can simultaneously see, listen, and speak without mutual blocking, creating a fluid conversational experience. Additionally, the proactive interaction mechanism enables the LLM to continuously monitor the input streams and decide at a frequency of 1 Hz whether to speak, allowing it to initiate reminders or comments based on its understanding of the live scene. This goes beyond reactive question-answering to create a truly interactive and attentive AI companion.

The overall workflow of MiniCPM-o 4.5 begins with its end-to-end omni-modal architecture, where modality encoders and decoders are densely connected to the LLM via hidden states, enabling robust information flow and exploitation of multimodal knowledge during training. For real-time interaction, the model can be converted from its default half-duplex mode to full-duplex mode using the as_duplex() method. The streaming process involves two key steps: streaming_prefill ingests audio waveforms and video frames in chunks, while streaming_generate produces text and audio output tokens with configurable parameters like decode_mode and max_new_speak_tokens_per_chunk. The model supports session management via session_id, allowing long-running conversations with KV-cache clearing. Voice style can be set using a reference audio and a multimodal system prompt, enabling consistent character voices across interactions. This structured approach makes deployment straightforward for both offline and streaming scenarios.

MiniCPM-o 4.5 enables compelling use cases that showcase its multimodal capabilities. In a voice roleplay scenario, a reference audio clip of Elon Musk is used to clone his voice, and the model engages in a conversation about Mars colonization, responding with accurate content and natural intonation that matches the persona. Another use case is emotion shift, where the assistant is instructed to describe the excitement of successfully buying concert tickets followed by disappointment when the system fails, and it generates speech that dynamically shifts tone to reflect these emotions. For real-time video understanding, a user streams a skiing video through the model, and it provides spoken commentary describing the action, demonstrating vision and speech integration. In document parsing, the model processes complex English documents with tables and formulas, achieving state-of-the-art results on OmniDocBench and outperforming specialized pipeline tools. These scenarios highlight the model's ability to handle diverse interaction modes with high accuracy and naturalness.

MiniCPM-o 4.5 is designed for AI researchers, developers, and enthusiasts working on multimodal applications that require vision, speech, and real-time interaction. It runs on various platforms: NVIDIA GPUs for PyTorch inference, CPU devices via llama.cpp and Ollama, and it offers quantized int4 and GGUF formats in 16 sizes for memory-constrained environments. Additionally, it supports high-throughput inference with vLLM and SGLang, and has a unified multi-chip backend via FlagOS. The model is completely open-source, with pre-trained weights available on Hugging Face and an API service that supports full-duplex realtime interaction. This accessibility empowers innovators to build custom interactive systems without cloud dependency. In summary, MiniCPM-o 4.5 redefines what is possible at the edge by packing powerful omni-modal capabilities into a compact, deployable package that rivals much larger proprietary models.