AI & ML
FondueQA: A Benchmark for Evaluating Parametric Compensation in Multi-Hop QA
arXiv cs.IR0 views
arXiv:2611.09274v1 Announce Type: cross Abstract: Large Language Models (LLMs) show remarkable performance based on parametric knowledge, i.e., their ability to recall facts and associations internalized in a model's weights. However, studying this capability on established multi-hop benchmarks may be complicated by benchmark contamination, as many of these benchmarks have been publicly available long enough to plausibly appear in modern pretraining corpora. As a result, correct answers under missing evidence may reflect either general parametric knowledge or memorization of benchmark-specific instances. To study how models use internal knowledge when retrieval evidence is incomplete, we introduce FondueQA, in which every required multi-hop fact is uniquely localized to a designated evidence document. This enables controlled document-level evidence ablations to study parametric compensation under incomplete retrieval. Our empirical evaluation reveals that parametric compensation strongly depends on both entity popularity and the structural role of missing evidence. Generally, models compensate more successfully for facts involving popular entities. On FondueQA, we show that bridging evidence can be more useful than the answer-bearing document itself. Moreover, models may recall a missing fact in isolation but fail to use it under partial evidence, particularly when the context contains plausible competing entities, which can reduce Exact Match by up to 15 percentage points.
Source:https://arxiv.org/abs/2611.09274PreviousLarge Language Model Consensus and Corpus-Adaptive Topic Modeling for Classifying and Mapping Patent InnovationNextENGRAM: From Agent Experience to Weights through Verified Consolidation
Related posts

Agents draft. People decide.
0dev.to UX

โค้ดเพิ่ม 400% แต่คุณไม่ได้เร็วขึ้น 400%: GitHub ชี้คอขวดใหม่อยู่ที่การรีวิว
0dev.to GitHub

Drago: Your GitHub Dragon
0dev.to GitHub
Case Study: Automating Tableau to Power BI Migration with Up to 90% Accuracy.
0dev.to Data
Vom Bluttest ins Molekül: Warum die Alamar/Nightingale-Studie räumliches Denken braucht...
0dev.to Data
10x Genomics가 Anthropic의 Claude AI를 Chromium 플랫폼에 통합한 'Sentira'를 발표했습니다. 단일세포 데이터 분석을 가...
0dev.to Data