Despite its importance and ubiquity, filtering remains a challenging
task, particularly when (i) the state is high dimensional; (ii) the
state transition dynamic is nonlinear, expensive to simulate, and
intractable, e.g., if it involves the integration of a chaotic partial
differential equation; and (iii) observations of the state are
sparse, both in space and time
[1]}, [2]}. These constraints
reflect typical challenges faced in numerical weather prediction or
geophysical data assimilation, and pose severe challenges for consistent sequential Monte Carlo (SMC) algorithms, which invariably
face particle degeneracy or impoverishment
[3]}, [4]}.
In such scenarios, state-of-the-art resultsAt least in terms
of time-averaged errors in point estimates of the state
[5]}. are typically
obtained with the ensemble Kalman filter (EnKF), which is the
workhorse of modern ensemble-based data assimilation
[6]}.
The EnKF implements a two-step Monte Carlo approximation of the
classical Kalman recursions. In the forecast step, a particle
approximation of the filtering distribution is propagated
through the transition dynamic to yield a “forecast” ensemble at the
next observation time. In the analysis step, the forecast
ensemble is updated via the action of a linear
transformation to yield an empirical approximation of the new
filtering marginal. The linear transformation is estimated under
Gaussian assumptions.
Hence, the EnKF cannot yield consistent estimators of the filtering
distribution for non-Gaussian models [7]}.
In essence, the EnKF trades consistency of the estimators for lower
variance, and thus robustness in high dimensions. Yet the intrinsic
bias of the EnKF—due both to the linear transformation and the way
it is estimated—implies that increasing the ensemble size beyond a
certain threshold does not improve accuracy;
increasing computational effort does not yield better inference.
We wish to address these limitations.
