Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering
Apple researchers reframe video captioning evals: forget lexical matching, measure information fidelity instead.
Why it matters
A methodological advancement in how we measure VLLM caption quality—moving from reference-matching to information-fidelity tests. Practitioners building video understanding systems and eval researchers will care; this could reshape how captioning models are benchmarked.
The key facts
10 to knowResearch from Apple Machine Learning
Proposes evaluation paradigm shift: lexical matching → information fidelity
Addresses 'one-to-many' caption problem: valid captions penalized for text variation
Application: Visual Large Language Models (VLLMs)
Method: multiple-choice question answering as evaluation proxy
Addresses 'one-to-many' problem in video description evaluation
Proposes information fidelity as caption quality metric
Targets Visual Large Language Models (VLLMs)
Moves beyond lexical matching for fine-grained quality assessment
Published September 2026
Go to the source
Apple Machine Learningmachinelearning.apple.com
Publisher excerpt: Evaluating video captioning remains a critical challenge for Visual Large Language Models (VLLMs). Existing metrics primarily rely on matching generated text against ground-truth references. This paradigm suffers from the “one-to-many” nature of video description, where high-quality captions are…