DeepAmbigQA: Ambiguous Multi-hop Questions for Benchmarking LLM Answer Completeness
Apple's new benchmark exposes a blind spot: LLMs with search tools still botch ambiguous multi-hop questions that require reasoning across multiple interpretations.
Why it matters
DeepAmbigQA is a capability benchmark that measures how well LLMs handle a specific class of complex reasoning—disambiguating entities and integrating evidence across multiple hops. It identifies a gap in existing eval suites and provides practitioners a tool to stress-test RAG and search-augmented systems.
The key facts
10 to knowBenchmark targets multi-hop QA with ambiguous entities (e.g., 'Heat' — multiple films with same title)
Tests both disambiguation AND multi-step reasoning in a single eval
Apple's automatic data generation pipeline (DeepAmbigQAGen) constructs the benchmark
Existing QA benchmarks don't jointly evaluate both challenges
Relevant to LLMs with integrated search tools and open-domain QA systems
Benchmark focuses on ambiguous multi-hop QA (e.g., 'Which actor from Heat won an Oscar')
Addresses two joint challenges: distinguishing between multiple entities with same name + reasoning across large evidence sets
Automatic data generation pipeline (DeepAmbigQAGen) to scale benchmark construction
Existing QA benchmarks rarely evaluate both disambiguation and multi-hop reasoning together
Published by Apple Machine Learning Research (Aug 2026)
Go to the source
Apple Machine Learningmachinelearning.apple.com
Publisher excerpt: Large language models (LLMs) with integrated search tools show strong promise in open-domain question answering (QA), yet they often struggle to produce complete answer set to complex questions such as “Which actor from the film Heat won at least one Academy Award?”, which requires (1)…