AI & ML
PS-HalluBench: Taxonomy-Driven Benchmarking of LLM Hallucinations in Power Systems
arXiv cs.SY0 views
arXiv:2611.22012v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly explored for power-system applications, yet their reliability in engineering analysis remains uncertain. We introduce PS-HalluBench, a manually constructed benchmark of 200 questions that evaluates hallucination-related failures across five power-system task categories: numerical reasoning, network topology, constraint verification, false-premise acceptance, and abstention. We evaluate ten LLMs over the benchmark, yielding 39.35\% overall correctness, with constraint verification achieving only 21.75\%, and highlighting persistent difficulties in engineering reasoning. We further investigate engineering-grounded reasoning through numerical solver support and verified network information both of which show significant performance improvement. The findings reveal important limitations of standalone LLM reasoning and motivate integrating LLMs with verified engineering tools for reliable power-system analysis.
Source:https://arxiv.org/abs/2611.22012PreviousSafe Reinforcement Learning with Hard Constraints: Policy Gradient Methods with Intersample GuaranteesNextAn Explainable Agentic AI Framework for Distribution System Load Restoration
Related posts
The Songless Archive Is a Better Listening Log Than a Playlist
0dev.to
AI makes it easier to understand the whole project
0dev.to
ChatGPT Has No Secret Codes. These Features Are What Help
0dev.to
How to Check a Freelancer's AI Work Before You Pay Them
0dev.to
I scanned 277 public MCP servers. Here's what 90% of them miss.
0dev.to

The JSON Parsed. Is It Right? How to Score LLM Data Extraction
0dev.to