CSE DSI Machine Learning Seminar with Mingrui Liu (CS, George Mason University)

Beyond Black-Box Analysis: Understanding the Role of Local Updates in Distributed Learning

Local updates are a fundamental technique in distributed learning, widely adopted for their ability to reduce communication overhead and improve scalability. Empirical evidence suggests that methods like local stochastic gradient descent (SGD) can enhance convergence rates and improve generalization across various distributed machine learning tasks. Despite its practical success, the theoretical foundations behind these benefits remain unclear. Notably, even for heterogeneous quadratic objectives, local updates are provably ineffective, highlighting a significant gap between empirical observations and theoretical understanding.

In this talk, Liu argues that the disconnect between the theory of distributed optimization and the practical success of distributed machine learning stems from the limitations of traditional black-box analysis. Conventional approaches often overlook the structural properties of machine learning models, leading to theoretical results that fail to explain empirical observations. To bridge this gap, he will present three concrete cases where local updates are provably beneficial: logistic regression, a two-layer neural network with Gaussian inputs, and a two-layer neural network with feature heterogeneity. In the first two cases, he will demonstrate how local steps accelerate convergence to a global minimum. In the third case, he will show that local updates enable more efficient learning of heterogeneous features, leading to improved generalization. Finally, he will discuss open challenges and future directions for distributed optimization in the era of foundation models.

Mingrui Liu has been an assistant professor at the Department of Computer Science, George Mason University since August 2021. From September 2020—August 2021, he was a postdoc at Rafik B. Hariri Institute at Boston University. He received his Ph.D. at the Department of Computer Science at the University of Iowa in August 2020. His research interests include machine learning, optimization, deep learning, and statistical learning theory. He serves as an area chair for NeurIPS/ICML/AISTATS/IJCAI. His research is recognized by an AAAI new faculty highlight.