Backend
Training Neural Networks to be Locally Adaptive
arXiv stat.ML0 views
arXiv:2611.22198v1 Announce Type: cross Abstract: Deep neural networks fitted by global empirical risk minimization commit to a single model for the entire covariate space. In a regression problem, if the regression function $f_0$ is inhomogeneous --- varying rapidly in one region, while being nearly flat in another --- no single model can be appropriate everywhere, and the fit degrades where the function varies most rapidly. Here we study an alternative in which, instead of fitting the network globally, we fit it locally. For each query point $x$, a \emph{local neural network} is trained on the $k$ nearest-neighbours of $x$ alone, making the training sample and the fitted function query-dependent. We analyze the performance of the resulting estimator through a localization approach, establishing that the pointwise risk of the estimator with an appropriate choice of $k$ attains the rate $n^{-2\alpha/(2\alpha+d^{*})}$, up to logarithmic factors, when $f_0$ is locally $\alpha$-H\"older with $\alpha\in(0,1]$ and the covariates are supported on a $d^{*}$-dimensional manifold embedded in $\mathbb{R}^{d}$. A matching Le Cam lower bound shows that this rate is pointwise minimax optimal. However, the local rate-optimality requires $k$ to be calibrated to the smoothness of $f_0$, which is unknown in practice. We therefore provide a locally adaptive choice of $k$ that makes the neural network adapt to the local H\"older exponent of $f_0$ and to the intrinsic dimension simultaneously, both of which are potentially varying over the covariate domain. An extensive simulation study demonstrates the dominance of the locally adaptive neural network over a globally trained network and several state-of-the-art methods.
Source:https://arxiv.org/abs/2611.22198PreviousCentroid Spectral Coupling: Implicit Alignment and Representation Diversity in Self-Supervised LearningNextSOTA or Luck? The Winner's Curse in LLM Leaderboards
Related posts

AWS Lambda now supports GPU instances for Event Functions on Lambda Managed Instances
0AWS Blog
Learn Data Structures and Algorithms in Python with Visualizations
0freeCodeCamp
The night the queue ran dry at 1:40
0dev.to
WAFio v2, SafeLine and ModSecurity through three test tools
0dev.to
A review checklist for n8n workflows before you release them
0dev.to
Building a Gherkin editor and two-way Git sync on Atlassian Forge: eight things I learned
0dev.to