ISyE Seminar: “Concentration of contractive stochastic approximation and applications to reinforcement learning”
Our first seminar of spring semester has been rescheduled to next Wednesday, March 1. You will still be able to join in person or via Zoom. This seminar will feature Martin Zubeldia from the University of Minnesota who will discuss concentration of contractive stochastic approximation and applications to reinforcement learning.
Wednesday, March 1, 2023
3:30 p.m. - Graduate seminar
4:30 p.m. - Reception, coffee and cookies
Lind Hall, Room 325
Watch the seminar via Zoom
Martin Zubeldia
Assistant Professor
Department of Industrial and Systems Engineering
University of Minnesota
About the seminar
Professor Zubeldia and his colleagues study the concentration behavior of a stochastic approximation algorithm under a contractive operator with respect to an arbitrary norm. The researchers consider two settings where the iterates are potentially unbounded: (1) bounded multiplicative noise, and (2) additive sub-Gaussian noise. They obtain concentration bounds on the convergence errors and show that these errors have sub-Gaussian tails in the additive noise setting, and super-polynomial tails (faster than polynomial decay) in the multiplicative noise setting. Moreover, their bounds hold anytime in the sense that the entire sample path lies within a tube of decaying radius with high probability.
To demonstrate the applicability of their theoretical results, the researchers use them to provide anytime high probability bounds for a large class of reinforcement learning algorithms, including but not limited to on-policy TD-learning with linear function approximation, off-policy TD-learning with generalized importance sampling factors, and Q-learning. To the best of their knowledge, super-polynomial concentration bounds for off-policy TD-learning have not been established in the literature due to the challenge of handling the combination of unbounded iterates and multiplicative noise.
About the speaker
Martin Zubeldia is an assistant professor in the Department of Industrial and Systems Engineering (ISyE) at the University of Minnesota. Before joining the department, he spent a year as a postdoctoral fellow in the Department of Industrial and Systems Engineering at the Georgia Institute of Technology and two years as a postdoc in the Department of Mathematics and Computer Science at the Eindhoven University of Technology and the Korteweg-de Vries Institute for Mathematics at the University of Amsterdam. He received a Ph.D. in electrical engineering from the Massachusetts Institute of Technology (2019), and a master's in engineering (2014) and bachelor's in electronics engineering (2012) from Universidad ORT Uruguay.
His research primarily focuses on using applied probability for the modeling, analysis, and control of large-scale stochastic decision and learning systems. He is particularly interested in exploring the fundamental tradeoffs between performance, efficiency, and scalability that arise in these systems, and in how traditional model-based analysis can be combined with newer data-centric approaches to get the best of both worlds. He has been recognized as a finalist in the 2019 INFORMS APS Best Student Paper Award, and as a finalist in the 2016 INFORMS George E. Nicholson Student Paper Competition.