AI & ML
Asking but Missing: Decision Coverage in Natural-Language Data Querying
arXiv cs.SE0 views
arXiv:2611.23864v1 Announce Type: cross Abstract: Large language models are increasingly used to translate natural language questions about structured data into executable programs. A fundamental challenge is recognizing when a question admits multiple computational interpretations and identifying the decisions needed to resolve them. We investigate whether language models recognize these unresolved decisions before generating code, or silently adopt assumptions that may lead to incorrect results. We refer to the latter behavior as Underspecified Decision Completion (UDC). To distinguish different causes of UDC, we introduce Grounded Code Generation (GCG), a diagnostic protocol that separates the identification of unresolved decisions, their resolution using evidence, and their translation into executable code. By examining each stage separately, we can determine where the decisions responsible for incorrect results were first missed or mishandled. We evaluate this framework on BIRD-Pandas with twelve models. The results show that answering clarification questions consistently improves execution accuracy, yet substantial errors remain. Our findings suggest that the main difficulty lies not in answering clarification questions, but in recognizing which questions need to be asked in the first place.
Source:https://arxiv.org/abs/2611.23864PreviousValidating Concurrent Correctness Witnesses with Ghosts by Thread-Modular Abstract InterpretationNextInference-Time Scaling in Agentic Hardware Verification Workflows: A Multi-Strategy Characterization Across Protocol Complexity
Related posts
Fuzzy name matching for sanctions screening: normalisation, transliteration and Jaro-Winkler
0dev.to
The Claude Architect Professional exam changes on 7 December - Here is what's different
0dev.to
SEO Isn't Enough Anymore. Meet GEO.
0dev.to

Prompt Engineering vs. Context Engineering vs. Harness Engineering: What’s the Difference?
0dev.to
P_SAPEA Exam Practice Questions: The Comparisons That Decide SAP Enterprise Architect Scenarios
0dev.to
Olala 7A1B says 8 GB, but the BF16 file is 13.59 GB: what a 1B-active hybrid MoE needs to run
0dev.to