AI & ML
Gradient Descent on Two ReLU Neurons: Global Landscape and Bifurcation Dynamics
arXiv stat.ML0 views
arXiv:2610.42118v1 Announce Type: cross Abstract: Understanding how gradient descent learns features remains a central challenge in neural network theory. We study this question in a minimal multi-index setting that already exhibits rich multi-phase gradient dynamics: a well-specified two-neuron ReLU teacher-student model under isotropic Gaussian inputs, trained by population gradient flow from small initialization. We show that the dynamics are organized by two directions: the "easy" bisector direction, which carries the leading signal, and the "hard" splitting direction, which governs specialization. To characterize the loss structure and learning behavior, we analyze the population squared-loss landscape and show that every critical point with both student weights nonzero is either a global minimum or a saddle. We then track the gradient-flow trajectory, showing that student neurons first collapse toward a bisector saddle before escaping and specializing to teacher neurons. Our results provide both a landscape and dynamics account of the multi-phase symmetry-breaking behavior that arises in a simple multi-index model and under standard algorithmic and architectural choices.
Source:https://arxiv.org/abs/2610.42118PreviousCausalWAM: Learning the Causes of Actions in World Action ModelsNextTransport-Free Lower Bounds for Ollivier--Ricci Curvature in Higher-Order Networks: Scope, Tightness, and Computational Applications
Related posts

Apple's New HomeView Smart Home Hub: Here's How Much RAM It Has
0MacRumors
Rewind on an Older Apple Watch: Memova's Upcoming Feature
0Hacker News New
Plain Vanilla Web
0Hacker News New
Hedge Funds Were 'Massive' Sellers of AI Debt, RBC's Skiba Says
0Hacker News New
Claude Code sends your email address to websites
0Hacker News New
AgentSight: System-wide AI agent profiling and monitoring with eBPF
0Hacker News New