Sampling diffusion models in the era of generative AI

Industrial Problems Seminar 

Morteza Mardani (NVIDIA Corporation)

Abstract

In the rapidly evolving landscape of AI, a transformative shift from content retrieval to content generation is underway. Central to this transformation are diffusion models, wielding remarkable power in visual data generation. My talk touches upon the nexus of generative AI and NVIDIA's influential role therein. I will then navigate through diffusion models, elucidating how they establish the bedrock for leveraging foundational models. An important question arises: how to integrate the rich prior of foundation models in a plug-and-play fashion for solving downstream tasks such as inverse problems and parametric models? Through the lens of variational sampling, I present an optimization framework for sampling diffusion models that only needs diffusion score evaluation. Not only does it provide controllable generation, but the framework also establishes a connection with the well-known regularization by denoising (RED) framework, unveiling its extensive implications for text-to-image/3D generation.