AI & ML
Gradient Descent on Two ReLU Neurons: Global Landscape and Bifurcation Dynamics
arXiv stat.ML0 views
arXiv:2610.42118v1 Announce Type: cross Abstract: Understanding how gradient descent learns features remains a central challenge in neural network theory. We study this question in a minimal multi-index setting that already exhibits rich multi-phase gradient dynamics: a well-specified two-neuron ReLU teacher-student model under isotropic Gaussian inputs, trained by population gradient flow from small initialization. We show that the dynamics are organized by two directions: the "easy" bisector direction, which carries the leading signal, and the "hard" splitting direction, which governs specialization. To characterize the loss structure and learning behavior, we analyze the population squared-loss landscape and show that every critical point with both student weights nonzero is either a global minimum or a saddle. We then track the gradient-flow trajectory, showing that student neurons first collapse toward a bisector saddle before escaping and specializing to teacher neurons. Our results provide both a landscape and dynamics account of the multi-phase symmetry-breaking behavior that arises in a simple multi-index model and under standard algorithmic and architectural choices.
Source:https://arxiv.org/abs/2610.42118PreviousCausalWAM: Learning the Causes of Actions in World Action ModelsNextTransport-Free Lower Bounds for Ollivier--Ricci Curvature in Higher-Order Networks: Scope, Tightness, and Computational Applications
Related posts
The Agent Made the Test Green by Weakening It
0dev.to Testing
Goodbye, ChatGPT - Why I Cancelled My ChatGPT Subscription
0dev.to Programming
Your Next Big Idea Could Win ₹2,50,000 — INCEPTRIX 3.0 Is Calling Innovators
0dev.to Programming
Stop Feeding Raw HTML to LLMs: How to Convert Any URL to Clean Markdown
0dev.to Programming
How to run multiple coding agents in parallel without them fighting
0dev.to Git
Queue heavy builds for parallel Claude Code agents
0dev.to Git