M. Mahdi Mostafaei · Shahid Beheshti University
Why Does SGD Prefer Flat Minima? Through the Lens of Dynamical Systems
Abstract
We show that stochastic gradient descent (SGD) escapes sharp minima exponentially fast, even before reaching stationarity. While SGD is the standard optimizer for non-convex tasks like neural networks, its ability to find generalizable solutions remains unclear. Escaping analysis which measures how quickly SGD leaves poorly generalizing sharp minima has been insightful but previously required stationarity. In this work, we prove exponential escape in the non-stationary regime using Large Deviation Theory. Specifically, we identify the quasipotential as a key tool for characterizing SGD dynamics throughout training.
