ML Seminar: Zhiqi Bu

CSE DSI Machine Learning seminars will be held Tuesdays 11a.m. - 12 p.m. Central Time in hybrid mode. We hope facilitate face-to-face interactions among faculty, students, and partners from industry, government, and NGOs by hosting some of the seminars in-person. See individual dates for more information.

This week's speaker, Zhiqi (Woody) Bu (Amazon AWS AI), will be giving a talk titled, "On the Computational Efficiency of Differentially Private Deep Learning".

Abstract

Differentially private (DP) optimization is the standard paradigm to learn large neural networks that are accurate and privacy-preserving. The computational cost for DP deep learning, however, is notoriously heavy due to the per-sample gradient clipping. Existing DP implementations are 2−1000× more costly in time and space complexity than the standard (non-private) training. In this work, we develop a novel Book-Keeping (BK) technique that implements existing DP optimizers (thus achieving the same accuracy), with a substantial improvement on the computational cost. Specifically, BK enables DP training on large models and high dimensional data to be roughly as efficient as the standard training, whereas previous DP algorithms can be inefficient or incapable of training due to memory error. The computational advantage of BK is supported by the complexity analysis as well as extensive experiments on vision and language tasks. Our implementation achieves state-of-the-art (SOTA) accuracy with very small extra cost: on GPT2 and at the same memory cost, BK has 1.0× the time complexity of the standard training (0.75× training speed in practice), and 0.6× the time complexity of the most efficient DP implementation (1.24× training speed in practice). We will open-source the codebase for the BK algorithm.

Biography

Dr. Zhiqi Bu is an Applied Research Scientist at Amazon AWS AI, focusing on the optimization of large-scale deep learning, especially with differential privacy. Dr. Bu obtained his Ph.D. in the Applied Math and Computational Science program (AMCS) at the University of Pennsylvania in 2021, under Benjamin Franklin Fellowship, where he also obtained his M.A. in Statistics from Wharton School. Dr. Bu completed his B.A. (Honors) in Mathematics at the University of Cambridge in 2015.