🥇 VideoEval-Pro Leaderboard

A More Robust and Realistic QA Evaluation benchmark of Multi-modal LLMs in long video understanding

Introduction

Do existing long video benchmarks faithfully reflect model's real capacity to understand long video content? Do the gains reported by newer models genuinely translate into stronger long video comprehension capability, or are they illusional? To probe these questions, we present VideoEval-Pro, a more robust and realistic long video understanding benchmark containing open-ended, short-answer QA problems. To construct VideoEval-Pro, we source the questions from four existing long video understanding MCQ benchmarks, and reformat these questions into free-form questions. We apply a series of filtering methods based on video duration, question and answer type, answerability and QA difficulty to ensure the quality of our benchmark. Our final benchmark contains a total of 1,289 short-answer questions based on 465 videos, with an average duration of 38 minutes.

Minimum number of parameters (B)

0 10

Maximum number of parameters (B)

0 10

Select Model Type

Rank	Models	Model Size(B)	Type	Frames	LP_Open	LR_Open	HP_Open	HR_Open	Overall_Open
10	Gemini-2.5-Flash	unknown	Proprietary	256	47.2	29.9	28.1	34.5	40.8

Rank	Models	Model Size(B)	Type	Frames	LP_Open	LR_Open	HP_Open	HR_Open	Overall_Open
1	GPT-4.1	unknown	Proprietary	256	47.2	29.9	28.1	34.5	40.8
2	GPT-4.1-mini	unknown	Proprietary	256	46	32	27.3	32.6	39.9
3	Gemini-1.5-Pro	unknown	Proprietary	512	43.7	32.7	35.5	31.8	39.3
4	Gemini-2.5-Flash	unknown	Proprietary	256	42.4	30.6	25.6	26.9	36.3
5	Gemini-1.5-Flash	unknown	Proprietary	512	41.5	25.9	27.3	25.8	35.1
6	GPT-4o	unknown	Proprietary	256	39.4	23.1	26.4	29.2	34.2
7	Qwen2.5-VL	7	Open-source	512	33.9	15.6	24.8	17.8	27.7
8	InternVideo2.5	8	Open-source	512	33.6	17	19.8	18.2	27.2
9	VideoChat-Flash	7	Open-source	512	33.3	16.3	21.5	17.4	27
10	Qwen2-VL	7	Open-source	512	31.7	14.3	21.5	20.5	26.5
11	InternVL3	8	Open-source	64	30.3	17	24	13.3	24.7
12	InternVL2.5	8	Open-source	64	28.8	19.7	21.5	16.7	24.6
13	LLaVA-Video	7	Open-source	64	28.5	13.6	20.7	19.3	24.2
14	Vamba	10	Open-source	512	28.1	10.9	21.5	12.5	22.3
15	LongVU	7	Open-source	512	25.9	12.9	19.8	17.4	22.1
16	Video-XL	7	Open-source	512	22.3	15	18.2	10.2	18.6
17	LongLLaVA	9	Open-source	512	21.7	15	14	10.2	17.8
18	Phi-4-Mini	5.6	Open-source	128	19.2	12.9	18.2	10.2	16.5
19	LongVA	7	Open-source	64	20.5	6.8	19	9.5	16.5
20	Mantis-Idefics2	8	Open-source	24	17.8	9.5	16.5	8.3	14.8
21	Video-LLaVA	8	Open-source	8	13.2	6.1	14	6.1	11

Models are ranked based on Overall_Open.

🥇 VideoEval-Pro Leaderboard

A More Robust and Realistic QA Evaluation benchmark of Multi-modal LLMs in long video understanding

Introduction

Dataset Statistics and Tasks Info

Submit on VideoEval-Pro Leaderboard Introduction

The evaluattion model should be used is GPT-4o-0806

⚠ Please note that you need to submit the JSON file with the following format: