StreamingBench

Introduction

The rapid development of Multimodal Large Language Models (MLLMs) has expanded their capabilities from image comprehension to video understanding. However, most of these MLLMs focus primarily on offline video comprehension, necessitating extensive processing of all video frames before any queries can be made. This presents a significant gap compared to the human ability to watch, listen, think, and respond to streaming inputs in real time, highlighting the limitations of current MLLMs. In this paper, we introduce StreamingBench, the first comprehensive benchmark designed to evaluate the streaming video understanding capabilities of MLLMs. StreamingBench assesses three core aspects of streaming video understanding: real-time visual understanding, omni-source understanding, and contextual understanding. The benchmark consists of 18 tasks, featuring 900 videos and 4,500 human-curated QA pairs. Each video features five questions presented at different time points to simulate a continuous streaming scenario. We conduct experiments on StreamingBench with 13 open-source and proprietary MLLMs and find that even the most advanced proprietary MLLMs like Gemini 1.5 Pro and GPT-4o perform significantly below human-level streaming video understanding capabilities. We hope our work can facilitate further advancements for MLLMs, empowering them to approach human-level video comprehension and interaction in more realistic scenarios.

Leaderboard *

Model	LLM Params	Frames	Date	Overall (%)	Real-Time Visual Understanding (%)	Omni-Source Understanding (%)	Contextual Understanding (%)
Gemini 1.5 Pro Google	-	Video	2024-06-15	70.26	77.39	67.80	51.06
MiniCPM-o 2.6 OpenBMB	8B	60	2025-01-14	66.01	79.88	53.40	38.45
GPT-4o OpenAI	-	60	2024-06-15	64.10	74.54	50.95	47.94
InternLM-XC2.5-OL Shanghai AI Lab	7B	64	2024-12-12	60.80	75.36	46.20	33.58
Claude 3.5 Sonnet Anthropic	-	20	2024-07-30	59.71	74.04	41.40	37.83
LLaVA-OneVision Bytedance & NTU S-Lab	7B	32	2024-08-08	58.43	74.27	40.83	30.96
Qwen2-VL Alibaba	7B	768	2024-08-19	56.99	71.15	40.73	33.08
MiniCPM-V 2.6 OpenBMB	8B	64	2024-08-12	57.67	72.43	40.23	33.39
VITA-1.5 NJU	7B	16	2025-01-03	57.36	70.88	40.80	35.83
InternVL2 Shanghai AI Lab	8B	32	2024-07-18	57.04	70.11	42.73	34.10
LLaVA-NeXT-Video Bytedance & NTU S-Lab	32B	32	2024-05-10	56.68	69.83	41.73	34.29
Kangaroo Meituan & UCAS	8B	64	2024-07-23	53.34	65.76	40.04	31.18
LongVA NTU S-Lab	7B	128	2024-06-25	50.66	63.11	35.93	30.21
VILA-1.5 NVIDIA & MIT	8B	14	2024-07-21	49.46	61.54	37.53	26.66
Video-CCAM QQMM	14B	32	2024-07-16	42.53	53.42	32.22	24.06
VideoLLaMA 2 Alibaba	7B	32	2024-08-29	43.33	52.58	35.92	23.69

Rank	Model	LLM Params	Frames	Date	Overall (%)	Real-Time Visual Understanding (%)
Rank	Model	LLM Params	Frames	Date	Overall (%)	OP	CR	CS	ATP	EU	TR	PR	SU	ACP	CT
	MiniCPM-o 2.6 OpenBMB	8B	60	2025-01-14	79.88	85.01	85.94	89.91	85.95	80.12	86.60	76.85	74.80	74.79	46.11
	Gemini 1.5 Pro Google	-	1 fps	2024-06-15	77.39	83.43	77.94	89.24	81.65	79.17	83.92	83.93	60.32	74.87	49.22
	InternLM-XC2.5-OL Shanghai AI Lab	7B	64	2024-12-12	75.36	84.74	79.69	83.91	87.91	75.78	80.06	67.59	65.85	70.25	38.34
	GPT-4o OpenAI	-	60	2024-06-15	74.54	80.66	76.98	86.67	73.81	75.95	85.48	75.00	70.66	65.99	43.09
	LLaVA-OneVision Bytedance & NTU S-Lab	7B	32	2024-08-08	74.27	82.83	77.34	83.23	83.33	72.05	74.77	73.15	68.29	71.10	41.97
	Claude 3.5 Sonnet Anthropic	-	20	2024-07-30	74.04	82.45	73.77	82.43	82.40	76.39	85.56	61.68	60.73	67.88	47.62
	MiniCPM-V 2.6 OpenBMB	8B	64	2024-08-12	72.43	78.20	71.88	84.18	83.99	75.16	75.39	72.22	56.50	67.14	47.15
	Qwen2-VL Alibaba	7B	0.2-1 fps	2024-08-19	71.15	75.75	79.69	76.58	79.08	74.53	75.08	74.07	65.85	65.16	41.97
	VITA-1.5 NJU	7B	16	2025-01-03	70.88	77.66	82.81	82.33	79.34	72.67	71.34	67.59	62.20	69.12	32.12
	InternVL2 Shanghai AI Lab	8B	32	2024-07-18	70.11	73.84	65.63	78.80	82.03	71.43	72.90	73.15	63.01	65.44	42.49
	LLaVA-NeXT-Video Bytedance & NTU S-Lab	32B	32	2024-05-10	69.83	80.11	71.09	80.70	80.72	71.43	73.21	62.96	59.35	63.17	36.79
	Kangaroo Meituan & UCAS	7B	64	2024-07-23	65.76	77.57	74.22	75.70	74.38	70.91	62.86	52.63	50.54	63.97	33.16
	LongVA NTU S-Lab	7B	128	2024-06-25	63.11	73.02	66.41	66.46	74.84	65.22	62.93	60.19	56.91	56.66	37.82
	VILA-1.5 NVIDIA & MIT	8B	14	2024-07-21	61.54	71.12	57.81	74.68	72.22	70.81	62.62	51.85	51.22	60.34	18.65
	Video-CCAM QQMM	14B	32	2024-07-16	53.42	54.50	63.28	73.73	63.73	57.76	47.35	50.93	41.87	48.44	26.94
	VideoLLaMA 2 Alibaba	7B	32	2024-08-29	52.58	59.95	60.16	62.97	60.46	54.66	46.11	41.67	46.75	48.16	34.72

Rank	Model	LLM Params	Frames	Date	Overall (%)	Omni-Source Understanding (%)
Rank	Model	LLM Params	Frames	Date	Overall (%)	ER	SCU	SD	MA
	Gemini 1.5 Pro Google	-	1 fps	2024-06-15	67.80	52.40	50.80	80.40	87.60
	MiniCPM-o 2.6 OpenBMB	8B	60	2025-01-14	53.40	48.40	24.40	63.20	77.60
	GPT-4o OpenAI	-	60	2024-06-15	50.95	53.60	32.40	49.00	68.80
	InternLM-XC2.5-OL Shanghai AI Lab	7B	64	2024-12-12	46.20	45.20	30.40	48.40	60.80
	InternVL-V2 Shanghai AI Lab	8B	32	2024-07-18	42.73	44.80	28.11	47.20	50.80
	LLaVA-NeXT-Video Bytedance & NTU S-Lab	32B	32	2024-05-10	41.73	41.60	24.50	44.40	56.40
	Claude 3.5 Sonnet Anthropic	-	20	2024-07-30	41.40	39.60	35.60	34.40	56.00
	LLaVA-OneVision Bytedance & NTU S-Lab	7B	32	2024-08-08	40.83	41.20	26.10	43.20	52.80
	VITA-1.5 NJU	7B	16	2025-01-03	40.80	42.80	28.40	39.60	52.40
	Qwen2-VL Alibaba	7B	0.2-1 fps	2024-07-16	40.73	40.80	25.30	41.20	55.60
	MiniCPM-V 2.6 OpenBMB	8B	64	2024-08-12	40.23	42.00	27.71	40.40	50.80
	Kangaroo Meituan & UCAS	7B	64	2024-07-23	40.04	40.80	34.14	40.00	45.20
	VILA-1.5 NVIDIA & MIT	8B	14	2024-07-21	37.53	40.40	28.51	37.20	44.00
	LongVA NTU S-Lab	7B	128	2024-06-25	35.93	41.20	28.51	32.00	42.00
	Video-LLaMA2 DAMO NLP	7B	32	2024-08-29	35.92	43.60	23.29	35.20	41.60
	Video-CCAM QQMM	14B	32	2024-07-16	32.22	38.00	21.29	31.20	38.40

Rank	Model	LLM Params	Frames	Date	Overall (%)	Contextual Understanding (%)
Rank	Model	LLM Params	Frames	Date	Overall (%)	MCU	ACU	SQA	PO
	Gemini 1.5 Pro Google	-	1 fps	2024-06-15	51.06	42.40	52.80	59.20	45.10
	GPT-4o OpenAI	-	60	2024-06-15	47.94	42.80	50.40	52.40	39.22
	MiniCPM-o 2.6 OpenBMB	8B	60	2025-01-14	38.45	38.00	36.80	42.40	29.41
	Claude 3.5 Sonnet Anthropic	-	20	2024-07-30	37.83	43.20	36.00	34.80	35.29
	VITA-1.5 NJU	7B	16	2025-01-03	35.83	32.00	33.60	44.00	25.49
	LLaVA-NeXT-Video Bytedance & NTU S-Lab	32B	32	2024-05-10	34.29	28.80	34.27	44.00	13.64
	InternVL-V2 Shanghai AI Lab	8B	32	2024-07-18	34.10	27.20	35.08	42.80	20.45
	InternLM-XC2.5-OL Shanghai AI Lab	7B	64	2024-12-12	33.58	38.00	28.80	38.00	13.73
	MiniCPM-V 2.6 OpenBMB	8B	64	2024-08-12	33.39	27.20	37.50	40.00	11.11
	Qwen2-VL Alibaba	7B	0.2-1 fps	2024-08-19	33.08	26.40	34.27	44.40	4.55
	Kangaroo Meituan & UCAS	7B	64	2024-07-23	31.18	32.40	35.89	30.80	4.00
	LLaVA-OneVision Bytedance & NTU S-Lab	7B	32	2024-08-08	30.96	30.40	35.08	30.00	18.18
	LongVA NTU S-Lab	7B	128	2024-06-25	30.21	28.00	33.47	34.40	4.55
	VILA-1.5 NVIDIA & MIT	8B	14	2024-07-21	26.66	28.00	29.03	26.00	11.76
	Video-CCAM QQMM	14B	32	2024-07-16	24.06	22.80	26.21	27.60	2.27
	Video-LLaMA2 DAMO NLP	7B	32	2024-08-29	23.69	26.00	28.23	21.20	2.27

* Experiment Setting: 60 seconds of context preceding the query time

Visualization

Data Statistics

Option	Count	Percentage

Category	Count	Percentage

Question Length Distribution

Minimum length: 4 words
Maximum length: 34 words
Average length: 11.57 words

Options Length Distribution

Citation


      @article{lin2024streaming,
        title={StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding},
        author={Junming Lin and Zheng Fang and Chi Chen and Zihao Wan and Fuwen Luo and Peng Li and Yang Liu and Maosong Sun},
        journal={arXiv preprint arXiv:2411.03628},
        year={2024}
      }