Xiangyu Zeng^1,*, Yuandong Yang^1,*, Zhiqiu Zhang^2,4,*, Yuhan Zhu^1,*, Xinhao Li^1,*, Qingyi Si^3,*
Dingyu Yao⁶, Changlian Ma^1,2, Haoran Chen^1,2, Xinyu Chen^1,2, Yansong Shi^2,5, Junhao Zhou⁶
Yifei Li⁹, Jun Zhang^1,3, Chuanyu Qin⁶, Chenxu Yang⁶, Xinlei Yu⁷, Kun Ouyang⁸
Yuchen Shao⁶, Qianshan Wei⁶, Changhai Zhou¹⁰, Jun Gao¹¹, Jiaqi Wang³, Limin Wang^1,†

¹NJU · ²PJLAB · ³JD · ⁴SJTU · ⁵USTC · ⁶CAS
⁷CUHK · ⁸PKU · ⁹THU · ¹⁰FDU · ¹¹ZJU
_{* Equal contribution. † Corresponding author.}

Overview · Method · Models and data · Results · Quick start · Citation

Perceive the present. Remember the past. Respond at the right time.

OneStreamer is a streaming video LLM that jointly learns to record evidence and respond through a shared proactive generation process. It preserves time-grounded factual memory before future questions are known, interprets incoming visual evidence, and decides when enough evidence is available to respond.

Built on Qwen3-VL-4B-Instruct, OneStreamer-4B achieves the best aggregate results among the compared methods on all eight benchmarks in the paper, spanning online perception, memory, and proactive response.

_{Blue bars show OneStreamer; the lavender portions mark the Qwen3-VL baseline. Each panel uses its benchmark's metric.}

• Reusable memory: Proactive Hierarchical Caption Memory (PHCM) retains local details and completed-event summaries alongside a recent visual window.
• Timely responses: Proactive State Transition Learning (PSTL) preserves output anchors and selects representative state changes and persistence, outperforming dense state supervision with only 27.5% of annotated state tokens supervised.
• Evidence-aligned training: OneStreamer-1M combines streaming captions, streaming QA, and cleaned open-source data into over one million interaction records.

Proactive Hierarchical Caption Memory. As video arrives, the model writes local-detail captions with </Observe> and summaries of completed events with </Summary>. These time-aligned records remain in text history after their source frames leave the recent visual window, supplying context for later questions without revisiting historical visual features. During training, causal caption targets also supervise the interpretation of observed video prefixes.

Proactive State Transition Learning. PSTL keeps supervision at every output anchor and selects representative tokens for state changes and state persistence. Other state tokens remain in the training sequence but are masked from the state loss; caption and answer text retain full supervision. At inference, the model directly generates control states and text from the available context.

For proactive QA, </Silence> means continued observation, </Standby> indicates emerging but insufficient evidence, and </Response> introduces a user-visible answer.

OneStreamer-1M covers proactive caption memory, perception and memory QA, proactive QA, and proactive interaction. Its synthesis pipeline aligns both the content and release time of each target with the evidence observed so far.

The caption branch curates videos, verifies captions at multiple temporal scales, and releases each caption after its supporting evidence is observed. The QA branch generates task-directed questions, verifies supporting intervals, and calibrates response times to the earliest point that satisfies the reliability criterion. Synthesized examples are combined with cleaned open-source data in a common streaming format.

Selected comparisons from the paper's main results table are shown below. All metrics are higher-is-better; evaluation settings are documented in the evaluation guide.

ProactiveVideoQA is abbreviated as ProactiveVQA in the paper. OneStreamer's OmniMMI result uses ASR for AP, SI, MD, and SG; PA uses visual input only. ViSpeak retains its native 0–5 score scale.

_{Top: caption records retain evidence for a later question. Bottom: the model updates its count when each target event occurs.}

The inference example includes the counting video and runs the model on incoming frames, printing proactive responses as they are generated.

Follow Evaluation / Preparation to create the onestreamer-eval environment. The evaluated stack uses Python 3.10, PyTorch 2.6.0 with CUDA 12.4, Transformers 4.57.6, and FlashAttention-2. The guide includes the required installation order and exact dependency versions.

Place the OneStreamer-4B checkpoint under path_to/models/OneStreamer-4B/ at the repository root, or pass --model-path to the inference script. For evaluation, set model paths in models.json. Commands below run from the repository root.

conda activate onestreamer-eval
CUDA_VISIBLE_DEVICES=0 python Inference/inference.py

This runs the included video with the question: “How many times does the event: showing something to the camera happen?” Responses are printed immediately, and all updates are saved to Inference/outputs/responses.jsonl.

To use your own input:

CUDA_VISIBLE_DEVICES=0 python Inference/inference.py \
  --video ../path_to/videos/example.mp4 \
  --question "Tell me when the person picks up the cup." \
  --output outputs/my_video.jsonl

The inference script resolves relative paths from Inference/. See the inference guide for model-path overrides and additional options.

Download OVOBench, StreamingBench, OVBench, ODVBench, ProactiveVideoQA, OmniMMI, OVO-Timing, and ViSpeak, then configure paths.json following the data layout. OVO-Timing shares the OVOBench videos; the selected OmniMMI ASR caches are included.

# Run prediction jobs sequentially across eight GPUs.
bash Eval/run_script/run_eval_8gpu.sh \
  --model OneStreamer-4B --mode infer --run-id evaluation

# Score the saved predictions; configure text-judge credentials first.
bash Eval/run_script/run_eval_8gpu.sh \
  --model OneStreamer-4B --mode judge --run-id evaluation

Independent scoring uses saved annotations and predictions and needs no GPU, checkpoint, video, or ASR cache. Prediction outputs are grouped under Eval/outputs/<model>/<bench>/, and metrics and judge caches under Eval/judge_outputs/<model>/<bench>/. See the evaluation guide for individual benchmarks, resume, model switching, and judge configuration.

Project code is licensed under Apache-2.0. Third-party components retain their original licenses; see NOTICE. Dataset media and model checkpoints remain subject to their respective source terms.

We thank the authors of Qwen3-VL, VLMEvalKit, and the benchmark and data resources used in this project. Figures in this README are taken from the OneStreamer paper; their sources are listed in assets/README.md.

@misc{zeng2026onestreamerunifyingperceptionmemory,
  title={OneStreamer: Unifying Perception, Memory, and Proactive Response in Streaming Video Interaction},
  author={Xiangyu Zeng and Yuandong Yang and Zhiqiu Zhang and Yuhan Zhu and Xinhao Li and Qingyi Si and Dingyu Yao and Changlian Ma and Haoran Chen and Xinyu Chen and Yansong Shi and Junhao Zhou and Yifei Li and Jun Zhang and Chuanyu Qin and Chenxu Yang and Xinlei Yu and Kun Ouyang and Yuchen Shao and Qianshan Wei and Changhai Zhou and Jun Gao and Jiaqi Wang and Limin Wang},
  year={2026},
  eprint={2610.01762},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2610.01762}
}