AI & ML
Benign Overfitting for Regression with Trained Two-Layer ReLU Networks
arXiv stat.ML0 views
arXiv:2410.06191v2 Announce Type: replace-cross Abstract: We study the least-square regression problem with a two-layer fully-connected neural network, with ReLU activation function, trained by gradient flow. Our first result is a generalization result, that requires no assumptions on the underlying regression function or the noise other than that they are bounded. We operate in the neural tangent kernel regime, and our generalization result is developed via a decomposition of the excess risk into estimation and approximation errors, viewing gradient flow as an implicit regularizer. This decomposition in the context of neural networks is a novel perspective of gradient descent, and helps us avoid uniform convergence traps. In this work, we also establish that under the same setting, the trained network overfits to the data. Together, these results, establishes the first result on benign overfitting for finite-width ReLU networks for arbitrary regression functions.
Source:https://arxiv.org/abs/2410.06191PreviousAttentionSwitch: Jailbreak Defense for LLMs via Joint Learning of Sparse Attention-Head Selection and ScalingNextBias-Optimal Bounds for Stochastic Gradient Descent
Related posts
Software jobs are back, but not for juniors. What I'd build if I were starting in 2026.
0dev.to Discuss
How to Calculate Real Throughput in an Automated Filling Line
0dev.to Productivity
WeightBench Tests Language Models as Weight-Space Programmers
0dev.to Machine Learning
Data Cleaning for messy dates dd-mm-yyyy
0dev.to Machine Learning

How Much GPU Memory Do I Need?
0dev.to Machine Learning
The Hugging Face Ecosystem in 2026: Transformers, Datasets, Trainer, LoRA/PEFT, Gradio & Spaces (Real Code)
0dev.to Machine Learning