A diffusion-model-based ensemble score filter approach for solving high dimensional nonlinear data assimilation problems

Data Science Seminar

Feng Bao
Florida State University

Abstract

In this talk, we present the diffusion-model-based Ensemble Score Filter (EnSF) for accurate and efficient high-dimensional nonlinear filtering. 

Nonlinear filtering, also known as data assimilation, is the process of estimating the evolving state of a dynamical system by optimally combining noisy observations with predictions from a numerical model. Conventional particle filters and ensemble Kalman filters lose accuracy in highly nonlinear, large-scale settings. EnSF overcomes this by representing the filtering density via a score-based diffusion model in a pseudo-temporal domain, storing information in the score function rather than finite Monte Carlo samples. A training-free, mini-batch Monte Carlo estimator directly approximates the score function at any pseudo-spatial–temporal location, avoiding costly neural network training while retaining high accuracy. Numerical results on Lorenz-96 systems with up to one million dimensions show EnSF’s substantial gains over the state-of-the-art Kalman type Filter. We further demonstrate the method for data assimilation in calibrating benchmark SPDE solutions and atmosphere–ocean simulation models.