AI & ML
Unifying Central Flow, Rod Flow, and the Edge Coupling
arXiv stat.ML0 views
arXiv:2611.08358v1 Announce Type: new Abstract: At the edge of stability, the iterates of gradient descent oscillate along the sharp directions of the Hessian. This poses a significant challenge for continuous-time models as the usual continuous-time approximation -- the gradient flow -- admits no oscillations. Several continuous-time models have been proposed to capture the behavior at the edge of stability, among them central flow and rod flow. Although both models can be interpreted as performing a coarse-graining in time, the connection between them remains poorly understood. Here we show that both flows can be derived from a third object, the edge coupling: a functional on consecutive iterates whose critical points are the fixed points and period-two orbits of gradient descent. It is revealed that the two flows correspond to coarse-grainings at different time scales. Rod flow coarse-grains over only the fastest time scale, while central flow coarse-grains over both the fast and intermediate time scales. Furthermore, using the edge coupling, we derive rod flows and central flows for momentum methods such as heavy ball and Adam. We verify the derived flows experimentally on representative neural network architectures, where they track the discrete iterates throughout all phases of training -- including the transition from the stable regime to the edge of stability.
Source:https://arxiv.org/abs/2611.08358PreviousSeries-wise Cross-validation for Global Time Series ForecastingNextDouble Machine Learning for Panel Data: Identification, Specification, and a Bank Regulation Effect
Related posts

I’ve reviewed almost every Android foldable this year — here’s how the iPhone Duo compares
0The Verge
Physicl raises $10M to make the digital world real enough for robots
0TechCrunch

Kubernetes, mark II? Why Google’s giving away gVisor, a code sandbox used by Anthropic and OpenAI
0The New Stack
AI helps find better recipes for 3D-printing titanium parts
0Phys.org Tech
Newly-developed paint could keep your home cooler and lower energy bills
0Phys.org Tech

Warhammer 40,000: Dawn of War 4 creative director explains how core 40K tabletop features like ‘Armor of Contempt’ were adapted to the RTS — ‘It’s not the same mechanic, but it does kind of the same t
0TechRadar