CS&E Colloquium: Bilevel Optimization: Recent Algorithmic & Theoretical Advances, and Emerging Applications in Training Large Language Models

The computer science colloquium takes place on Mondays from 11:15 a.m. - 12:15 p.m. This week's speaker, Mingyi Hong (UMN), will be giving a talk titled "Bilevel Optimization: Recent Algorithmic & Theoretical Advances, and Emerging Applications in Training Large Language Models".

Abstract

Abstract: Bilevel Optimization (BLO) is a class of challenging optimization problems that has two levels of nested optimization subproblems. It can be used to model applications in signal processing, machine learning and game theory, and more recently, in large language model training. In the first part of this talk, we will discuss recent advances that addressed two of its key challenges, (1) efficient implementation (e.g., how to efficiently deal with stochasticity, Hessian computation, etc.); (2) structural complexity (e.g., how to deal with non-convexity in lower-level problems). These works together provide a set of useful tools for the practitioners to customize for different application domains. In the second part of this talk, we will dive deep into a recent application – aligning Large Language Models (LLMs) with human values. We will show that the challenging LLM alignment problem can be cast as a special BLO problem – the inverse reinforcement learning problem – whose upper-level recovers a human reward model while the lower-level solves for the optimal policy. This perspective unifies popular alignment pipelines, such as the reinforcement learning with human feedback (RLHF) and its direct preference optimization (DPO) variants. Leveraging recent BLO advances yields algorithms that (i) outperform standard RLHF baselines in both sample- and compute-efficiency, and (ii) reveal key design principles that have already influenced industrial practice. Finally, we will discuss open problems and opportunities for future BLO research.

Biography

Mingyi is an Associate professor in the Department of Electrical and Computer Engineering at the University of Minnesota. His research has been focused on developing optimization theory and algorithms for applications in signal processing, machine learning and foundation models. His work has received two IEEE Signal Processing Society Best Paper Awards (2021, 2022), an International Consortium of Chinese Mathematicians Best Paper Award (2020), among others. He is an Amazon Scholar, and he is the recipient of the 2022 Pierre-Simon Laplace Early Career Technical Achievement Award from IEEE, and the 2025 Egon Balas Prize from INFORMS Optimization Society. He is a Fellow of IEEE.