AI & ML
Monitoring Machine Learning Systems in Production: A Multivocal Literature Review
arXiv cs.SE0 views
arXiv:2611.23293v1 Announce Type: new Abstract: Monitoring is essential for maintaining reliable machine learning (ML) systems after deployment. However, knowledge about its practical implementation and effectiveness remains fragmented across academic and gray literature. We present a multivocal literature review of 94 sources (37 academic and 57 gray) covering both traditional ML and generative AI systems. We develop taxonomies covering monitored objects and properties, techniques, input signals, tools, automation, frequency, evaluation, and challenges. Our findings reveal an imbalanced monitoring process. Existing approaches provide broad and largely automated support for observing system behavior, detecting problems, and communicating findings, while monitor configuration, investigation, and response retain substantial human involvement. Monitoring is concentrated on inference, with no source covering the entire production ML system, limiting end-to-end diagnosis. Evidence of effectiveness is also limited: among 78 sources presenting an approach or tool, 45 provide no formal evaluation, 24 report offline or controlled evaluations, and only nine evaluate monitoring in live production. This gap is particularly evident for generative AI, where evidence is dominated by gray literature and vendor claims. We propose six directions for improving monitor design, cross-layer diagnosis, proxy validation, corrective responses, production-oriented evaluation, and organizational practices. Our review provides a structured basis for improving the detection, diagnosis, and resolution of production ML problems.
Source:https://arxiv.org/abs/2611.23293PreviousAn Empirical Study of Visual Artifacts in Pull Request DescriptionsNextFrom Ambiguous Tickets to Ranked Hypotheses: Multi-Agent Root-Cause Analysis for Autonomous Heavy-Vehicle Software
Related posts
Ruby2ßhell: Pre-authentication RCE on stock Rails
0Hacker News
Generalizing self-normalized importance sampling with couplings
0arXiv stat.CO
Bayesian Computation of Copula-based Multivariate Models with an Application to Doping Detection
0arXiv stat.CO
WAIVES: Weak-form Asymptotically-unbiased Instrumental-Variable Estimation for SDEs
0arXiv stat.CO
Treat Your Honeypot Analysis Like Code: Git, Pull Requests and a Human Gate for AI [Guest Diary], (Wed, Nov 18th)
0SANS ISC
ISC Stormcast For Thursday, November 19th, 2026 https://isc.sans.edu/podcastdetail/10186, (Thu, Nov 19th)
0SANS ISC