As the former MCMC approach requires revisiting each data point for each update, the latter approximate approach of Variational inference scales better, and thus this approach gained a lot of popularity in the context of BNNs.
Variational Bayes approximates the complicated posterior distribution with a “simpler” variational approximation \(q({\mathbf {\omega }})\) , for example a Gaussian posterior with a diagonal covariance, (i.e. fully factorized Gaussian), and in that case each parameter of the model has its own mean and variance.
Analogously to variational autoencoders [1]}, we define a variational lower bound:
\(\log p(\mathcal {D}) \ge \mathcal {F}(q) \triangleq \underbrace{ \mathbb {E}_{q({\mathbf {\omega }})} [ \log p(\mathcal {D} | {\mathbf {\omega }}) ]}_{\text{Likelihood term}} - \underbrace{{D}_{KL} \big (q({\mathbf {\omega }}) || p({\mathbf {\omega }})\big )}_{\text{KL term}} \,,\) 
