Turbo-charging AI

More than ten years ago, Professor Zhaosong Lu foresaw the burgeoning impact of machine learning (ML) models as a fertile ground where his technical optimization skills could play a vital role. His journey into the realm of ML commenced with collaborations with experts from various domains, particularly in sparse and low-rank learning. Together with his collaborators, he developed innovative optimization models and methods for dimension reduction and variable selection. These advancements have found extensive applications in prediction, a central theme in ML. A pivotal point came when he embarked on a sabbatical at the ML division of Microsoft Research in Redmond, WA in 2013. This experience solidified his interests in ML and laid the foundation for his current research. 

For Lu, ML and optimization are inherently intertwined. ML often begins by creating models that need precise calibration, and optimization provides various avenues for designing efficient methods to perform this calibration. For example, imagine designing a model to differentiate between pictures of gophers and chipmunks (a tool that could have been helpful when Goldy was first drawn). The model consists of a series of calculations, each making use of some parameters. By altering these parameters, a model can be created that is better or worse at identifying the critters. To determine these parameters, pictures of gophers and chipmunks can be added into the model, and adjust the parameters when the model does not make a correct prediction. As more and more pictures are used (a process often called training), the model progressively becomes better at recognizing gophers and chipmunks. Adjusting the parameters to produce the best resulting models is an optimization problem, and this is where Lu’s expertise in optimization comes into play. One facet of his research aims at designing methods that find higher quality solutions to models in order to make sureoptimality conditions are satisfied. These optimality conditions are more stringent and can be guaranteed to correspond to the best solution to the problem.

Another facet of Lu’s work aims to improve the robustness of the process of selecting model hyperparameters so that minor changes in training do not significantly alter the calibrated model’s results. To address this issue, Lu collaborates with a doctoral student to develop innovative solution algorithms for bilevel optimization problems. Using these algorithms, the ML community can create more accurate and reliable predictive models by performing training in a way that considers the possibility of adversarial entities making slight modifications to the data.

Turbocharged by powerful optimization methods, AI techniques can solve a wide array of global challenges. In a project recently funded by the National Science Foundation (NSF), Lu collaborates with colleagues from the University of Pittsburgh and Rutgers to explore how AI can be used to predict properties of nanomaterials – tiny materials with versatile applications, including nanomedicine. His expertise is crucial in designing the very optimization techniques that power these models. Armed with these tailored tools, the team works on predicting the properties and behavior of nanostructures with the goal of discovering some that minimize side effects, while ensuring safe design and therapeutic effectiveness. Lu envisions many more applications for his work, especially in healthcare and precision medicine.

For the immediate future, Lu plans to incorporate elements of stochastic optimization into his approach to ML problems. Reminiscing on his days in graduate school, Lu warmly recalls: “I took many courses in probability during my PhD. I enjoyed them tremendously and always thought I would make use of them, after I was happy with how well I had explored deterministic algorithms. I think the time has come.” Lu plans to leverage the power of stochastic methods to tackle issues arising from vast datasets and large samples. Additionally, he is actively researching how these algorithms can be applied in federated learning and in decentralized optimization. With an expanding and ambitious research agenda on the horizon, Lu is driven by an unwavering desire to push the boundaries of what optimization algorithms can do to speed up AI advancements across a multitude of fields. His work embodies the benefits of establishing synergies between optimization and ML, offering pathways to more accurate, reliable, and impactful AI models.

Graphic courtesy of Sysouk Khambounmy