We can recognize that inside the expectation on the RHS is exactly the update rule for the variance in the bayes by backprop method [1]}. Thus, we can interpret bayes by backprop as a one-sample Monte Carlo estimation of that expectation value. The equations shows that in the limit of infinite samples, the bayes by backprop update step \(\nabla _\Sigma V(\mu + \epsilon \Sigma )\)  converges to the (averaged) Laplace approximation step \(\mathbb {E}_{\theta \sim q}[\nabla ^2_\theta V(\theta )]\)  up to a factor 2. This motivates both the power and the instability of bayes by backprop.
