AI & ML
DiagDBench: Evaluating Agentic LLMs on Interactive Database Diagnosis and Anomaly Remediation
arXiv cs.DB0 views
arXiv:2611.07339v1 Announce Type: new Abstract: Database diagnosis and anomaly remediation are essential to reliable database management system (DBMS) operation. Despite advances in large language model (LLM)-based database agents, constructing live DBMS environments that reproduce diverse anomalies at scale remains challenging, limiting the availability of executable samples for database agent training and evaluation. To this end, we introduce DiagDBench, a large-scale benchmark for training and evaluating database agents on interactive diagnosis, anomaly remediation, and optional workload performance optimization across diverse, realistic anomaly scenarios. Each sample pairs a user request, comprising an issue description and an anomaly-verified workload under which the issue was observed, with a self-contained, executable DBMS environment that supports real-time monitoring, interactive diagnosis, remediation, and workload execution. To construct these samples at scale, we propose an LLM-based framework with two complementary pipelines: Manual-Guided Anomaly Synthesis, which sequentially synthesizes user requests, database modifications, and anomaly-injection scripts grounded in DBA manuals and official DBMS documentation; and Incident-Guided Anomaly Reproduction, which reconstructs anomaly samples end to end from real-world enterprise online diagnosis incidents. Built using this framework with sample filtering, expert review, and refinement, DiagDBench comprises 6,940 synthesized training samples and a held-out test set of 454 expert-annotated samples, spanning three hardware configurations, two DBMSs, and three database schemas. Extensive evaluations of database agents powered by multiple LLMs demonstrate that even the best-performing agent, ReAct with GPT-6 Astra, scores only 56.24, whereas most agents score below 45, underscoring the difficulty of DiagDBench.
Source:https://arxiv.org/abs/2611.07339PreviousRecursively Testing and Vibe Debugging a Google AI Studio Generated Vibe-Coded Database Software ApplicationNextSelective Retrieval Routing for Multi-Violation Knowledge Graph Repair
Related posts
Geometric Functional Inequalities for Markov Processes: An Invitation via the Langevin Diffusion
0arXiv stat.CO
A Unified Coordinate Descent Framework for Penalized Longitudinal Log-Contrast Quantile Regression
0arXiv stat.CO
Fast Functional Clustering via Closed-Form Covariance Updates
0arXiv stat.CO
marp: Model-Averaged Inference for Parametric Renewal Processes in R
0arXiv stat.CO
dsp: An R Package for Bayesian Time Series Modeling with the Dynamic Shrinkage Process
0arXiv stat.CO
Machine Learning Fundamentals: Understanding How Machines Learn from Data
0dev.to Discuss