Choosing \(\pi (\theta )\)  is rather standard in the literature on Bayesian neural networks and complies with choosing a regularization for \(\theta \) , cf. [1]}. We will here follow the common choice of a centered normal distribution \(\pi (\theta ) = \mathcal {N}(\theta | 0, \lambda _\theta ^2)\) . For Bernoulli dropout [2]} with rate \(p\)  the term \(\frac{1}{N}D_{\mathrm {KL}}(q_{\phi }(\theta )\Vert \pi (\theta ))\)  in (REF ) then equals, up to a constant, \(\frac{(1-p)}{2N\lambda _\theta ^2}|\theta |^2\) .
