Confucius4-R2T2: A Low Latency and High Accuracy Real-Time Speech Recognition Model

Real Real-Time Transcription

                             

Confucius4-R2T2 is a low-latency and high-accuracy true streaming Automatic Speech Recognition (ASR) model that features fine-grained and configurable decoding chunks from 80 ms to 2 s. The model operates in append-only output mode: committing transcript text permanently without revising previous words, which is critical for applications where text must be processed or acted upon instantly. This results in a smoother user experience, avoiding disruptive text revisions and visual flickering in real-time applications, such as Real-Time Live Captioning & Subtitling, Downstream NLP Pipelines & LLM Agents, Simultaneous Speech Translation, etc.

R2T2, short for Real Real-Time Transcription, is built upon the Qwen3-ASR model. And it is trained with a unique set of data construction techniques including stable-prefix data, forced time-alignment data, and token-level audio segmentation. Combined with a Longest Stable Prefix (LSP) learning paradigm (tech report will be released soon), R2T2 can dynamically determine when a stable prefix can be safely emitted and when additional audio context is needed. By exposing only stable prefixes, the model provides high-quality context that conditions subsequent predictions while guaranteeing that previously emitted text remains unchanged. Despite its streaming design, R2T2 maintains strong accuracy in offline recognition.

• Low-latency and high accuracy streaming recognition — The model achieves accuracy close to that of offline recognition, with only 200 to 600 milliseconds average latency.
• Stable streaming output — Emitted text is committed as it arrives and remains unchanged.
• Configurable low-latency chunking - Supports decoding chunks from 80 ms to 2 s for different latency/accuracy trade-offs.
• No loss in offline accuracy — Adding streaming support does not degrade offline recognition accuracy.
• vLLM backend — Provides high-throughput inference. A Hugging Face transformers backend is also available.
• Context and hotword prompts — Natively supported.
• Multilingual support — Optimized for Chinese and English, while also supporting a broad range of additional languages.

Experimental results show that R2T2 achieves state-of-the-art (SOTA) performance in both latency and recognition quality among a range of open-source models, while remaining competitive with leading closed-source systems. This repository provides inference code, a minimal usage example, and a vLLM-based backend supporting both offline and real-time streaming inference.

• Overview
• Demo
  • Side-by-side comparison with GPT-Live-Transcribe
  • Additional resources
• Evaluation
  • Streaming performance
  • Accuracy
    • English
    • Chinese
• Model Checkpoints
• Installation
  • Clone the repository
  • Option 1: Conda
  • Option 2: uv
• Docker (recommended)
  • 1. Start a container
  • 2. Run the example inside the container
  • 3. Manage the container
• Quick Start
  • Configuration
• Python API
  • vLLM backend
    • Offline transcription
    • Streaming transcription
  • llama.cpp backend
• WebSocket Server
  • Start and stop the server
  • WebSocket endpoint
  • Message format
  • Example client
• Supported Languages
• Community & Contact
  • WeChat Group
  • Discord Server
  • Business contact
  • GitHub Issues
• Acknowledgements
• Citation
• License

Figure 1. Overall framework of R2T2.

More demonstrations and comparisons are available at r2t2.youdao.com/examples, covering chunk size, context and hotwords, code-switch, languages, noise, long-form speech.

If you are an author or maintainer of a model included in these comparisons and have questions or concerns about the results, please feel free to contact us through the repository issue tracker. We are happy to share evaluation details and work with you to verify or correct them.

The streaming API supports decoding chunks from 80 ms to 2 s; the figures below show representative WER/latency trade-offs at 160 ms.

Figure 3. English WER and retrospective chunk-wise latency across model and configuration settings.

Figure 4. Chinese CER and retrospective chunk-wise latency across model and configuration settings.

Figure 5. Accuracy-latency Pareto frontier. Lower-left is better; the frontier uses retrospective chunk-wise mean fuzzy latency.

English results use WER (%), and Chinese results use CER (%); lower is better.

※ Pseudo-streaming model: its partial transcript may revise previously emitted text; unmarked models use true streaming, append-only output.

The model weights are published on Hugging Face and ModelScope. Choose the checkpoint that matches the inference backend:

Inference backend	Model	Hugging Face	ModelScope	Description
vLLM	Confucius4-R2T2			Standard checkpoint for the vLLM backend
llama.cpp	Confucius4-R2T2-GGUF			GGUF repository with several quantization variants

We recommend using a fresh, isolated environment. For local development and source installation, use the Conda or uv environment below. Docker is recommended for quickly running the project with a preconfigured CUDA and runtime environment — see Docker.

git clone https://github.com/netease-youdao/Confucius4-R2T2.git
cd Confucius4-R2T2

conda create -n confucius4-r2t2 python=3.12 -y
conda activate confucius4-r2t2

# Install the package with the vLLM backend
pip install -e .

uv venv --python 3.12
source .venv/bin/activate

# Install the package with the vLLM backend
uv pip install -e .

Python 3.10+ is supported. Python 3.12 is the version we test against.

vLLM has strict CUDA / PyTorch compatibility requirements. If the install fails to resolve, check the version matrix on the vLLM website and pin a combination that matches your CUDA runtime.

R2T2 runs out of the box on the official Qwen3-ASR Docker image, which already ships every runtime library we need.

Pre-built image: qwenllm/qwen3-asr.

Before you begin, install the NVIDIA Container Toolkit to enable GPU access from Docker. If Docker Hub access is slow or unreliable in your region, you may need to configure a registry mirror.

LOCAL_WORKDIR=/path/to/your/workspace   # host path that will be mounted into the container
HOST_PORT=8000
CONTAINER_PORT=80

docker run --gpus all --name confucius4-r2t2 \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -p $HOST_PORT:$CONTAINER_PORT \
    --mount type=bind,source=$LOCAL_WORKDIR,target=/data/shared/confucius4-r2t2 \
    --shm-size=4gb \
    -it qwenllm/qwen3-asr:latest

Your local workspace ($LOCAL_WORKDIR) — including a checkout of this repository and the R2T2 checkpoint — will be mounted inside the container at /data/shared/confucius4-r2t2. Host port 8000 is mapped to container port 80; services running inside the container must bind to 0.0.0.0 (not 127.0.0.1) for port forwarding to work.

Once inside the container's shell:

cd /data/shared/confucius4-r2t2/Confucius4-R2T2
MODEL_PATH=/data/shared/confucius4-r2t2/Confucius4-R2T2 \
    ./run_example.sh /path/to/audio.wav

# re-enter after exiting
docker start confucius4-r2t2
docker exec -it confucius4-r2t2 bash

# remove completely
docker rm -f confucius4-r2t2

Grab any audio file (mono or stereo, any sample rate — it is resampled to 16 kHz internally) and run:

./run_example.sh /path/to/audio.wav \
    --model_path /path/to/Confucius4-R2T2 \
    --infer_mode stream_vllm \
    --language Chinese \
    --chunk_size_ms 160

Logs are written to run_example.log by default. Run ./run_example.sh --help to see the full flag list.

run_example.sh reads the following environment variables (all optional):

You can also call example.py directly and pass any of these as flags (--audio, --model_path, --infer_mode, --language, --chunk_size_ms, --lookahead_ms, --unfixed_token_num, --context).

Audio inputs can be passed as a local path, a URL, base64 data, or a (np.ndarray, sr) tuple. Batched inference is supported. Remember to wrap vLLM code under if __name__ == '__main__': to avoid the spawn error described in vLLM Troubleshooting.

import librosa
from r2t2 import R2T2ASRModel

if __name__ == "__main__":
    asr = R2T2ASRModel.LLM(
        model="/path/to/Confucius4-R2T2",
        gpu_memory_utilization=0.5,
        max_model_len=16384,
        max_inference_batch_size=32,
        max_new_tokens=4096,
    )

    wav, sr = librosa.load("path/to/audio.wav", sr=16000, mono=True)

    results = asr.transcribe(
        audio=[(wav, 16000)],
        language=["Chinese"],       # or [None]
        return_time_stamps=False,
    )
    print(results[0].language, results[0].text)

import librosa
from r2t2 import R2T2ASRModel

if __name__ == "__main__":
    asr = R2T2ASRModel.LLM(
        model="/path/to/Confucius4-R2T2",
        gpu_memory_utilization=0.4,
        max_model_len=16384,
        max_new_tokens=4,           # keep small for low-latency streaming
    )

    wav, sr = librosa.load("path/to/audio.wav", sr=16000, mono=True)

    state = asr.init_streaming_state(
        context="",                 # optional hotword / topic hint
        language="Chinese",         # or None
        unfixed_chunk_num=0,
        unfixed_token_num=1,
        chunk_size_sec=0.16,
    )

    step = int(0.16 * 16000)
    for pos in range(0, len(wav), step):
        seg = wav[pos : pos + step]
        _, text = asr.streaming_transcribe(seg, state, max_new_tokens=4)
        print("text:", text)

    asr.finish_streaming_transcribe(state)
    print("final:", state.text)

For a complete streaming example with adaptive max_new_tokens and initial-chunk lookahead handling, see example.py.

The r2t2_llama/ package adds decoding support for deployments where a llama.cpp runtime is preferred. Three routes are available:

GGUF_DIR=/path/to/gguf python -m r2t2_llama.example_llama \
    --audio resources/test.wav \
    --model_path /path/to/Confucius4-R2T2 \
    --infer_mode stream_llama_hybrid

Prebuilt CUDA artifacts for CPython 3.12 ship in the package, so no llama.cpp checkout is needed to run it. See r2t2_llama/README.md for the pinned llama.cpp version, the Python API, and rebuild instructions.

For real-time, multi-client streaming ASR, this repo ships a ready-to-run WebSocket server (ws_server.py), a launcher script (run_start_server.sh), and a reference Python client (ws_client.py).

# Start with a VAD model
./run_start_server.sh start \
    --model_path /path/to/Confucius4-R2T2 \
    --vad_model_path /path/to/Stream-VAD \
    --port 8272 \
    --gpu 0

# Stop
./run_start_server.sh kill

# Restart in one step
./run_start_server.sh restart \
    --model_path /path/to/Confucius4-R2T2 \
    --vad_model_path /path/to/Stream-VAD \
    --port 8272 \
    --gpu 0

The launcher resolves its own directory, so it can be invoked from anywhere. Logs are written to nohup_service_ws_<host_tag>_<port>.log in the current directory. The FireRedVAD model is available from Hugging Face. We recommend downloading the model files into this repository's checkpoints directory:

# The FireRedVAD repo ships several detectors, but only the streaming one is
# needed. Both commands below keep the `Stream-VAD/` folder name, so the files
# land in checkpoints/vad/Stream-VAD with no extra nesting.

# Option A — hf CLI (pip install -U "huggingface_hub[cli]")
hf download FireRedTeam/FireRedVAD \
    --include "Stream-VAD/*" \
    --local-dir checkpoints/vad

# Option B — git clone
git clone https://huggingface.co/FireRedTeam/FireRedVAD
cp -r FireRedVAD/Stream-VAD checkpoints/vad/

Either command leaves the model at checkpoints/vad/Stream-VAD, which is exactly what --vad_model_path defaults to — so you can drop the flag entirely.

Client → Server:

• Send raw 16 kHz mono PCM as int16 binary frames (the reference client uses ≈160 ms per frame, i.e. 2560 samples × 2 bytes).
• Send the string "YOUDAO_ONETIME_ASR_STREAM_EOS" to signal end-of-audio; the server will emit any final text and close.

Server → Client: JSON messages of the form

{
  "status": "success",
  "requestId": "<uuid>",
  "msg": {
    "text": "hello",
    "reset": false,
    "asr_cost_ms": 35.4,
    "total_cost_ms": 42.0
  }
}

• text is the newly recognized (incremental) segment since the previous message. Concatenate them client-side to get the full transcript.

ws_client.py is a minimal example that streams a WAV file to the server and prints the responses.

# Uses the default URI (ws://localhost:8272/asr_stream_api_v1) and built-in sample audio
python ws_client.py

# Point at a custom endpoint and audio file
python ws_client.py \
    --uri wss://your.host/asr_stream_api_v1 \
    --audio resources/test.wav \
    --save service_ws_test \
    --audio-id test.wav

Command-line options:

R2T2 is optimized for streaming recognition in Chinese and English. Beyond these primary languages, it retains useful cross-lingual streaming capability on languages such as French, German, Italian, Japanese, Korean, Portuguese, Russian, Spanish, Arabic, etc.

Join our community to ask questions, share ideas, and connect with other users and developers.

Scan the QR code below to join our WeChat group:

Join our Discord server

For high-concurrency, production-grade, domestically deployable, or private deployment solutions, as well as business inquiries and partnership opportunities, please feel free to contact us through the channels below.

• Phone: +86 010-82558901
• Email: AIcloud_Business@corp.youdao.com

We also welcome discussions in this repository's Issues section. Feel free to ask questions, report bugs, or suggest improvements!

We sincerely thank the Alibaba Qwen team for open-sourcing the Qwen3-ASR modeling code, which provides the architectural foundation for R2T2.

If you use this repository or the R2T2 checkpoint in your research, please cite Confucius4-R2T2 (this project):

@misc{Confucius4-R2T2,
  title        = {Confucius4-R2T2: A Low Latency and High Accuracy Real-Time Speech Recognition Model},
  author       = {NetEase Youdao},
  year         = {2026},
  howpublished = {https://github.com/netease-youdao/Confucius4-R2T2}
}

R2T2 uses dual licensing to distinguish the source code from the model weights:

• Code in this repository is released under the Apache License 2.0 and is free to use, modify, and redistribute (including commercially) under the terms of that license.
• Model weights are released under the NetEase Model Use License Agreement.