Show Me Examples: Inferring Visual Concepts from Image Sets
Apple's new benchmark exposes a blind spot in every major vision-language model.
Why it matters
Apple ML Research introduces VICIS, a new evaluation task revealing that state-of-the-art VLMs fail at visual concept inference—a capability gap that matters for real-world AI applications requiring few-shot visual reasoning.
The key facts
8 to knowNew benchmark: Visual Concept Inference from Sets (VICIS)
Tests VLM ability to infer shared concepts from image sets and apply to new inputs
State-of-the-art VLMs perform poorly on this task
Published by Apple Machine Learning Research
Addresses gap in visual reasoning from purely visual context (without text)
Apple introduces VICIS (Visual Concept Inference from Sets) benchmark
Tests VLM ability to infer shared concepts from example image sets
Requires models to generate new images preserving context-defined concepts
Go to the source
Apple Machine Learningmachinelearning.apple.com
Publisher excerpt: Vision-language models (VLMs) can follow complex textual instructions, yet they struggle to reason from purely visual context. In particular, current models fail to infer shared concepts from sets of example images and apply them to new inputs. We introduce Visual Concept Inference from Sets…