CSE DSI Machine Learning Seminar with Geoff Pleiss (Stats, UBC)

Ensemble methods have historically used either high-bias base learners (e.g. through boosting) or high-variance base learners (e.g. through bagging). Modern neural networks cannot be understood through this classic bias-variance tradeoff, yet “deep ensembles” are pervasive in safety-critical and high-uncertainty application domains. This talk will cover counterintuitive phenomena that emerge when ensembling overparameterized base models like neural networks. The first set of results will establish an empirical and theoretical equivalence between neural network ensembles and single (but larger) models, with implications for uncertainty quantification, robustness, and decision making. The second set of results will explore effective ensembling strategies for overparameterized models given this equivalence. Classic ensembling techniques—like bagging or feature subsetting—prove detrimental while seemingly pathological strategies—like averaging predictions of high- and low-accuracy neural networks—are beneficial. I will conclude with future directions and an outlook on ensembling in the age of large foundation models.

 

Geoff Pleiss is an assistant professor in the Department of Statistics at the University of British Columbia, as well as a Canada CIFAR AI Chair affiliated with the Vector Institute. He earned a Ph.D. in Computer Science from Cornell University under the supervision of Kilian Weinberger. Geoff specializes in uncertainty quantification in machine learning, especially within the contexts of Bayesian optimization, spatiotemporal modelling, and scientific discovery. His work has been recognized with the Blackwell-Rosenbluth Award from the International Society for Bayesian Analysis. Additionally, Geoff has co-founded several widely-used open source software projects, including the GPyTorch, LinearOperator, and CoLA libraries.