AI & ML
PS-HalluBench: Taxonomy-Driven Benchmarking of LLM Hallucinations in Power Systems
arXiv cs.SY0 views
arXiv:2611.22012v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly explored for power-system applications, yet their reliability in engineering analysis remains uncertain. We introduce PS-HalluBench, a manually constructed benchmark of 200 questions that evaluates hallucination-related failures across five power-system task categories: numerical reasoning, network topology, constraint verification, false-premise acceptance, and abstention. We evaluate ten LLMs over the benchmark, yielding 39.35\% overall correctness, with constraint verification achieving only 21.75\%, and highlighting persistent difficulties in engineering reasoning. We further investigate engineering-grounded reasoning through numerical solver support and verified network information both of which show significant performance improvement. The findings reveal important limitations of standalone LLM reasoning and motivate integrating LLMs with verified engineering tools for reliable power-system analysis.
Source:https://arxiv.org/abs/2611.22012PreviousSafe Reinforcement Learning with Hard Constraints: Policy Gradient Methods with Intersample GuaranteesNextAn Explainable Agentic AI Framework for Distribution System Load Restoration
Related posts
BEVCast: Path-Adaptive BEV for Efficient End-to-End Autonomous Driving
0arXiv cs.RO
World-Model Gradient Refinement in CEM: An Empirical Study
0arXiv cs.RO
PRIME: Perception Feedback with Situational Memory Embeddings in VLA Models
0arXiv cs.RO
Rollback the World, Keep the Reflection: Rollback-Induced Reflection for Long-Horizon LLM Agents
0arXiv cs.RO
ToM-2 Understanding Statements for Teacher-Guided Preference Learning
0arXiv cs.RO
Orbis 2: A Hierarchical World Model for Driving
0arXiv cs.RO