Backend
Training Neural Networks to be Locally Adaptive
arXiv stat.ML0 views
arXiv:2611.22198v1 Announce Type: cross Abstract: Deep neural networks fitted by global empirical risk minimization commit to a single model for the entire covariate space. In a regression problem, if the regression function $f_0$ is inhomogeneous --- varying rapidly in one region, while being nearly flat in another --- no single model can be appropriate everywhere, and the fit degrades where the function varies most rapidly. Here we study an alternative in which, instead of fitting the network globally, we fit it locally. For each query point $x$, a \emph{local neural network} is trained on the $k$ nearest-neighbours of $x$ alone, making the training sample and the fitted function query-dependent. We analyze the performance of the resulting estimator through a localization approach, establishing that the pointwise risk of the estimator with an appropriate choice of $k$ attains the rate $n^{-2\alpha/(2\alpha+d^{*})}$, up to logarithmic factors, when $f_0$ is locally $\alpha$-H\"older with $\alpha\in(0,1]$ and the covariates are supported on a $d^{*}$-dimensional manifold embedded in $\mathbb{R}^{d}$. A matching Le Cam lower bound shows that this rate is pointwise minimax optimal. However, the local rate-optimality requires $k$ to be calibrated to the smoothness of $f_0$, which is unknown in practice. We therefore provide a locally adaptive choice of $k$ that makes the neural network adapt to the local H\"older exponent of $f_0$ and to the intrinsic dimension simultaneously, both of which are potentially varying over the covariate domain. An extensive simulation study demonstrates the dominance of the locally adaptive neural network over a globally trained network and several state-of-the-art methods.
Source:https://arxiv.org/abs/2611.22198PreviousCentroid Spectral Coupling: Implicit Alignment and Representation Diversity in Self-Supervised LearningNextSOTA or Luck? The Winner's Curse in LLM Leaderboards
Related posts

China threatens 'countermeasures' in the face of new AI GPU rental and export restrictions — Beijing threatens retaliation ahead of Thursday's Xi Jinping meeting, Sino-American trade war may be renewe
0Tom's Hardware
At-home microbiome tests are taking off. Can they be trusted?
0Nature News
Signature Webhook HMAC-SHA256 Tidak Cocok? Hex vs Base64 dan 4 Jebakan yang Sering Terjadi
0dev.to
How do I make sure my agent drops outdated instructions instead of getting confused?
0dev.to

Klastro Walkthrough: Real Kubernetes Labs, in Your Browser
0dev.to
Part 2: The Main Problem - My Kali Download Was Too Small
0dev.to