AI & ML
A Theory of Cross-Validation Variance
arXiv stat.ML0 views
arXiv:2611.14463v1 Announce Type: new Abstract: Cross-validation estimates predictive performance by repeatedly splitting the data into training and validation sets, fitting the model on the first and evaluating it on the second. Every observation influences the estimate in two roles, through its own evaluation when it is held out and through the models it helps fit when it trains. The random splits determine its weight in each role. From this representation, we prove an exact finite-sample identity for the variance of the cross-validated estimate. For a given splitting scheme with a common training set size, the variance is a function of these weights, with constants of the model, the metric, and the population. It covers metrics averaged over observations, such as mean squared error, and over case--control pairs, such as the AUC. The identity shows which schemes to use. The variance decomposes into sampling variance, which depends only on the training size, and variance due to the random fold splitting, which arises when the weights change from one draw of the splits to the next. K-fold cross-validation, whose weights are equal across observations by design, is therefore variance optimal at first order; Monte Carlo and bootstrap cross-validation are not. The first-order variance of K-fold does not depend on K, and under reasonable conditions on the model leave-one-out has the smallest variance of any K-fold scheme. For the AUC the same holds for leave-pair-out, which evaluates every case--control pair. Finally, we show the cross-validated estimate is asymptotically normal with the identity's full variance. Together, these results provide a general theoretical foundation for inference on cross-validated performance and establish principles for selecting variance-optimal cross-validation designs.
Source:https://arxiv.org/abs/2611.14463PreviousCompositional random forestNextGeometry and Stochastic Analysis of Flow and Diffusion Models: A Survey
Related posts
ragR: Retrieval-Augmented Generation and RAG Assessment in R
0arXiv stat.CO
Convergence and global optimality of Wasserstein--Fisher--Rao descent for Gaussian mixtures
0arXiv stat.CO
Pseudo-marginal Sampling for Selective Inference
0arXiv stat.CO
Microsoft Patch Tuesday November 2027 , (Tue, Nov 10th)
0SANS ISC
ISC Stormcast For Wednesday, November 11th, 2026 https://isc.sans.edu/podcastdetail/10174, (Wed, Nov 11th)
0SANS ISC

The Interface Questions That Matter in a Disposable-Inbox Workflow
0dev.to UX