AI & ML
Unifying Central Flow, Rod Flow, and the Edge Coupling
arXiv stat.ML0 views
arXiv:2611.08358v1 Announce Type: new Abstract: At the edge of stability, the iterates of gradient descent oscillate along the sharp directions of the Hessian. This poses a significant challenge for continuous-time models as the usual continuous-time approximation -- the gradient flow -- admits no oscillations. Several continuous-time models have been proposed to capture the behavior at the edge of stability, among them central flow and rod flow. Although both models can be interpreted as performing a coarse-graining in time, the connection between them remains poorly understood. Here we show that both flows can be derived from a third object, the edge coupling: a functional on consecutive iterates whose critical points are the fixed points and period-two orbits of gradient descent. It is revealed that the two flows correspond to coarse-grainings at different time scales. Rod flow coarse-grains over only the fastest time scale, while central flow coarse-grains over both the fast and intermediate time scales. Furthermore, using the edge coupling, we derive rod flows and central flows for momentum methods such as heavy ball and Adam. We verify the derived flows experimentally on representative neural network architectures, where they track the discrete iterates throughout all phases of training -- including the transition from the stable regime to the edge of stability.
Source:https://arxiv.org/abs/2611.08358PreviousSeries-wise Cross-validation for Global Time Series ForecastingNextDouble Machine Learning for Panel Data: Identification, Specification, and a Bank Regulation Effect
Related posts
Avrea’s Faster Runners for GitHub Actions Reflect How AI Coding Is Changing CI
0HackerNoon
From Telecom Expense Management to AI Governance: How IntraTEM Is Expanding TEM
0HackerNoon

Your Marketplace's Most Valuable Asset Is Hiding in Its Conversational Data
0HackerNoon

AI Reasoning Models Collapse Completely Once Puzzles Get Hard Enough
0HackerNoon
XAUH: A Swiss Boutique Bets Gold's Next Buyers Will Carry It in a Chat App
0HackerNoon
Upwind Unveils Upwind Response to Stop AI-Driven Cloud Attacks Within Seconds
0HackerNoon