In some misspecified settings, the posterior distribution in Bayesian statistics may lead to inconsistent estimates. To fix this issue, it has been suggested to replace the likelihood by a pseudo-likelihood, that is the exponential of a loss function enjoying suitable robustness properties. In this paper, we build a pseudo-likelihood based on the Maximum Mean Discrepancy, defined via an embedding of probability distributions into a reproducing kernel Hilbert space. We show that this MMD-Bayes posterior is consistent and robust to model misspecification. As the posterior obtained in this way might be intractable, we also prove that reasonable variational approximations of this posterior enjoy the same properties. We provide details on a stochastic gradient algorithm to compute these variational approximations. Numerical simulations indeed suggest that our estimator is more robust to misspecification than the ones based on the likelihood.
 Bayesian methods are very popular in statistics and machine learning as they provide a natural way to model uncertainty. Some subjective prior distribution π is updated using the negative log-likelihood n via Bayes' rule to give the posterior π n (θ) ∝ π(θ) exp(− n (θ)). Nevertheless, the classical Bayesian methodology is not robust to model misspecification. There are many cases where the posterior is not consistent (Barron et al., 1999; Grünwald and Van Ommen, 2017) , and there is a need to develop methodologies yielding robust estimates. A way to fix this problem is to replace the log-likelihood n by a relevant risk measure. This idea is at the core of the PAC-Bayes theory (Catoni, 2007) and Gibbs posteriors (Syring and Martin, 2018) ; its connection with Bayesian principles are discussed in Bissiri et al. (2016) . Knoblauch et al (2019) builds a general representation of Bayesian inference in the spirit of Bissiri et al. (2016) and extends the representation to the approximate inference case. In particular, the use of a robust divergence has been shown to provide an estimator that is robust to misspecification (Knoblauch et al, 2019) . For instance, Hooker and Vidyashankar (2014) investigated the case of Hellinger-based divergences, Ghosal and Basu (2016) , Futami et al (2017), and Nakagawa et al. (2019) used robust β-and γ-divergences, while Catoni (2012) , Baraud and Birgé (2017) and Holland (2019) replaced the logarithm of the log-likelihood by wisely chosen bounded functions. Refer to Jewson et al (2018) for a complete survey on robust divergence-based Bayes inference. In this paper, we consider the Maximum Mean Discrepancy (MMD) as the alternative loss used in Bayes' formula, leading to a pseudo-posterior that we shall call MMD-Bayes in the following. MMD is built upon an embedding of distributions into a reproducing kernel Hilbert space (RKHS) that generalizes the original feature map to probability measures, and allows to apply tools from kernel methods in parametric estimation. Our MMD-Bayes posterior is related to the kernel-based posteriors in Fukumizu et al. (2013) , Park et al. (2016) and Ridgway (2017) , even though it is different. More recently, Briol et al. (2019) introduced a frequentist minimum distance estimator based on the MMD distance, that is shown to be consistent and robust to small deviations from the model. We show that our MMD-Bayes retains the same properties, i.e is consistent at the minimax optimal rate of convergence as the minimum MMD estimator, and is also robust to misspecification, including data contamination and outliers. Moreover, we show that these guarantees are still valid when considering a tractable approximation of the MMD-Bayes via variational inference, and we support our theoretical results with experiments showing that our approximation is robust to outliers for various estimation problems. All the proofs are deferred to the appendix. In this paper, we showed that the MMD-Bayes posterior concentrates at the minimax convergence rate and is robust to model misspecification. We also proved that reasonable variational approximations of this posterior retain the same properties, and we proposed a stochastic gradient algorithm to compute such approximations that we supported with numerical simulations. An interesting future line of research would be to investigate if the i.i.d assumption can be relaxed and if the MMD-based estimator is also robust to dependency in the data. Appendix A. Proof of Theorem 1. In order to prove Theorem 1, we first need two preliminary lemmas. The first one ensures the convergence of the empirical measureP n to the true distribution P 0 (in MMD distance D k ) at the minimax rate n −1/2 , and which is an expectation variant of Lemma 1 in Briol et al. (2019) that holds with high probability: The rate n −1/2 is known to be minimax in this case, see Theorem 1 in Tolstikhin et al. (2017) . The second lemma is a simple triangle-like inequality that will be widely used throughout the proofs of the paper: Lemma 6 We have for any distributions P , P and Q: Proof The chain of inequalities follow directly from the triangle inequality and inequality 2ab ≤ a 2 + b 2 . Let us come back to the proof of Theorem 1. An important point is that the MMDBayes can also be defined using an argmin over the set M 1 + (Θ) of all probability distributions absolutely continuous with respect to π and the Kullback-Leibler divergence KL(· ·): This is an immediate consequence of Donsker and Varadhan's variational inequality, see e.g Catoni (2007) . Using the triangle inequality, Lemma 5, Lemma 6 for different settings of P , P and Q, and Jensen's inequality: which gives, using Lemma 5 and the triangle inequality again: We remind that θ * = arg min θ∈Θ D k (P θ , P 0 ). This bound can be formulated in the following way when ρ is chosen to be equal to π restricted to B n : Finally, as soon as the prior mass condition C(π, β) is satisfied, we get: Appendix B. Proof of Theorem 2. In case of well-specification, Formula (3.1) simply becomes according to Jensen's inequality: Hence, it is sufficient to show that the inequality above implies the concentration of the MMD-Bayes to the true distribution. This is a simple consequence of Markov's inequality. Indeed, for any M n → +∞: which guarantees the convergence in mean of π β n D k (P θ , P 0 ) > M n · n −1/2 to 0, which leads to the convergence in probability of π β n D k (P θ , P 0 ) > M n ·n −1/2 to 0, i.e. the concentration of MMD-Bayes to P 0 at rate n −1/2 .
Robust Bayesian Estimation via Maximum Mean Discrepancy
{"src": "scitldr-AIC_train", "paper_id": "SygyKk24FH"}