ISyE Seminar Series: Yingbin Liang
"Why Adam Can Beat SGD: Second-Moment Normalization Yields Sharper Tails"
Yingbin Liang
Professor at the Department of Electrical and Computer Engineering at the Ohio State University (OSU), and a core faculty of the Ohio State Translational Data Analytics Institute (TDAI)
About the Seminar:
Adam is one of the most widely used optimization methods in modern machine learning, yet a basic theoretical question remains unresolved: why can Adam outperform stochastic gradient descent (SGD)? Existing theory often provides comparable convergence guarantees for the two methods, leaving Adam’s empirical advantage largely unexplained. In this talk, I will present a new high-probability perspective showing that, under standard smoothness and bounded-variance assumptions, Adam exhibits fundamentally sharper tail behavior than SGD and consequently achieves a faster convergence rate in terms of the confidence parameter. We identify second-moment normalization as the key mechanism underlying this advantage: it transforms cumulative stochastic-gradient energy into a self-normalized logarithmic quantity, making the optimization trajectory substantially less sensitive to rare but large stochastic-gradient shocks. I will explain how this mechanism benefits Adam and how a stopping-time and martingale-based analysis captures this effect, providing a concrete theoretical explanation for why Adam can outperform SGD.
About the Speaker:
Dr. Yingbin Liang is currently a Professor at the Department of Electrical and Computer Engineering at the Ohio State University (OSU), and a core faculty of the Ohio State Translational Data Analytics Institute (TDAI). She also serves as the Deputy Director of the NSF AI-EDGE Institute and the Co-Lead for Foundational AI Pillar of OSU AI^X Hub. Dr. Liang received the Ph.D. degree in Electrical Engineering from the University of Illinois at Urbana-Champaign in 2005, and served on the faculty of University of Hawaii and Syracuse University before she joined OSU. Dr. Liang's research lies at the intersection of machine learning, large-scale optimization, statistical signal processing, information theory, and wireless networks, with their growing applications to other scientific domains. She received the National Science Foundation CAREER Award and the State of Hawaii Governor Innovation Award in 2009. She also received EURASIP Best Paper Award in 2014. She is currently an Information Theory Society Distinguished Lecturer for 2026–2027. Dr. Liang is an IEEE fellow.
If you wish to be added to the ISyE Graduate Seminar Series emailing list, please email Event Coordinator Emily Rice at [email protected].