VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction

📝 Paper Summary

Multimodal Large Language Models (MLLMs) Speech-to-Speech Interaction

VITA-1.5 integrates vision, text, and speech into a single model via a three-stage training process, enabling real-time spoken dialogue about visual content without external speech recognition or text-to-speech modules.

Core Problem

Integrating speech into visual language models often causes modality conflicts (e.g., speech data degrading vision performance) and relies on slow external cascading modules for speech processing.

Why it matters:

Current open-source models lag behind proprietary ones like GPT-4o in handling real-time, natural speech interactions alongside vision.
Cascaded systems (ASR + LLM + TTS) suffer from high latency and lose paralinguistic features like emotion and tone.
Simultaneous optimization of vision (spatial) and speech (temporal) modalities is difficult due to their fundamental differences.

Concrete Example: A traditional system answering a question about a video must first transcribe audio (ASR), process text (LLM), and generate speech (TTS), causing noticeable delays. VITA-1.5 processes audio tokens directly to generate speech tokens, skipping these steps for faster response.

Key Novelty

Three-Stage Progressive Training Strategy

Aligns vision and text first to establish strong visual understanding foundations before introducing audio.
Incorporates audio input capability via an encoder trained with ASR (Automatic Speech Recognition) tasks, preventing interference with visual knowledge.
Enables end-to-end speech output by training a native speech decoder, removing the need for external Text-to-Speech (TTS) systems.

Architecture

Overall architecture of VITA-1.5 showing the Multimodal Encoder-Adapter-LLM setup and the specialized Speech Decoder branch.

Evaluation Highlights

Achieves comparable perception and reasoning capabilities to leading image/video-based MLLMs while adding speech functionality.
Successfully eliminates the need for separate ASR and TTS modules, significantly accelerating multimodal end-to-end response speed.
Demonstrates robust performance across image, video, and speech benchmarks, bridging the gap between open-source models and GPT-4o.

Breakthrough Assessment

8/10

Significant step for open-source MLLMs by achieving native end-to-end speech interaction with vision, a capability previously dominated by proprietary models like GPT-4o.

⚙️ Technical Details

Problem Definition

Setting: Omni-modal understanding and generation (Image/Video + Audio + Text -> Audio/Text)

Inputs: Interleaved image, video, text, and audio stream

Outputs: Text or speech response

Pipeline Flow

Input Processing (Visual/Audio Encoders + Adapters)
LLM Processing (Text Generation / Semantic Modeling)
Output Generation (Speech Decoders + Codec)

System Modules

Visual Encoder (Input Processing)

Extract spatial features from images/video

Model or implementation: InternViT-300M (448x448 input)

Audio Encoder (Input Processing)

Extract temporal features from speech inputs

Model or implementation: 24 Transformer blocks (1024 hidden size) + Conv downsampling

LLM Core

Multimodal understanding and text generation

Model or implementation: Vicuna-7B (implied from VITA-1.0 context or similar LLaMA-based)

Speech Decoder (NAR + AR) (Output Generation)

Convert LLM text features into discrete speech codes

Model or implementation: Two 4-layer LLaMA decoders (NAR + AR), hidden size 896

Codec Decoder (Output Generation)

Convert discrete speech tokens to waveform

Model or implementation: TiCodec (single codebook, size 1024)

Novel Architectural Elements

Integration of dual speech decoders (NAR + AR) specifically attached to LLM output for direct speech synthesis without external TTS
Unified three-stage training pipeline enabling single-model processing of video, image, text, and audio inputs and outputs

Modeling

Base Model: LLM (likely Vicuna-7B based on VITA-1.0 heritage) + InternViT-300M + Custom Audio Encoder

Training Method: Three-stage Progressive Training (Alignment -> Understanding -> SFT)

Objective Functions:

Purpose: Align audio encoder with text.

Formally: Connectionist Temporal Classification (CTC) loss.
Purpose: Train LLM to generate text from multimodal inputs.

Formally: Standard Next Token Prediction (Cross-Entropy Loss).
Purpose: Train speech decoder.

Formally: Cross-Entropy Loss on discrete speech tokens.

Training Data:

Stage 1 (Vision): ShareGPT4V, LLaVA-150K, ShareGemini (Video)
Stage 2 (Audio Input): 110k hours internal speech-transcription pairs, TTS-generated speech QA
Stage 3 (Speech Output): 3k hours text-speech paired data

Key Hyperparameters:

audio_encoder_params: 350M
speech_decoder_params: 120M
visual_tokens_per_image: 256
+ 1 more
audio_output_frame_rate: 12.5Hz (encoder), 40Hz (decoder)

Compute: Not reported in the paper

Comparison to Prior Work

vs. GPT-4o: VITA-1.5 is open-source and follows a similar end-to-end speech interaction paradigm.
vs. LLaMA-Omni: VITA-1.5 integrates vision (image/video) in addition to speech, whereas LLaMA-Omni focuses on speech-text.
vs. VITA-1.0: VITA-1.5 replaces external TTS with native token-based speech generation for lower latency.

Limitations

Heavy reliance on large-scale internal speech data (110k hours) which may hinder full reproduction.
Video processing uses uniform sampling, which might miss key events in long videos compared to more complex sampling strategies.
Speech generation quality depends on the TiCodec and limited 3k hours of TTS training data.

Reproducibility

Code: https://github.com/VITA-MLLM/VITA

Code is publicly available at https://github.com/VITA-MLLM/VITA. Training data sources are listed (ShareGPT4V, LLaVA-150K, etc.) but 110k hours of internal speech data is likely private.

📊 Experiments & Results

Evaluation Setup

Evaluation across Image, Video, and Speech benchmarks

Benchmarks:

MME, MMBench, MMMU, HallusionBench (Image Understanding & Reasoning)
Video-MME, TempCompass (Video Understanding)
Wenetspeech, Librispeech (Automatic Speech Recognition (ASR))

Metrics:

Accuracy
Word Error Rate (WER)
Statistical methodology: Not explicitly reported in the paper

Key Results

Benchmark	Metric	Baseline	This Paper	Δ
Not reported in the paper	Not reported in the paper	Not reported in the paper	Not reported in the paper	Not reported in the paper

Experiment Figures

The three-stage training strategy: 1) Vision-Language, 2) Audio Input Alignment, 3) Speech Output Training.

Main Takeaways

VITA-1.5 demonstrates that a single model can handle Vision, Text, and Audio inputs and generate Audio/Text outputs effectively.
The three-stage training strategy (Vision -> Audio Input -> Audio Output) effectively mitigates modality conflicts, preserving visual performance while adding speech capabilities.
End-to-end speech generation is viable in open-source models without relying on external TTS APIs, reducing system latency.

📚 Prerequisite Knowledge

Prerequisites

Transformer architecture
Multimodal Large Language Models (MLLM)
Audio/Visual Encoders (ViT, Wav2Vec style models)
Vector Quantization / Codec models

Key Terms

ASR: Automatic Speech Recognition—converting spoken audio into text

TTS: Text-to-Speech—converting text into spoken audio

MLLM: Multimodal Large Language Model—AI that processes multiple data types (text, images, audio) simultaneously

InternViT: A specific vision transformer model used as the visual encoder

TiCodec: A codec model used to compress continuous audio into discrete tokens and decode them back to waveforms

NAR Decoder: Non-Autoregressive Decoder—generates outputs in parallel (globally) rather than sequentially

AR Decoder: Autoregressive Decoder—generates outputs sequentially, one token at a time

CTC loss: Connectionist Temporal Classification—a loss function used to align sequences of different lengths, common in speech recognition

SFT: Supervised Fine-Tuning—training on labeled instruction-response pairs

Dynamic Patching: A technique to handle high-resolution images by splitting them into smaller grids (patches)