During the forward pass, weights are sampled from the variational posterior \(q(\mathbf {w}|\mathbf {\theta })\)  but backpropagation can't flow through random samples, to avoid this reparameterization trick is used [1]}. Sampling of weights \(w\)  can be obtained by sampling a parameter free distribution i.e. unit Gaussian, and shifting it by a mean \(\mathbf {\mu }\)  and scaling by a standard deviation \(\mathbf {\sigma }\) . We parameterise the standard deviation point-wise as \(\mathbf {\sigma } = \log (1 + \text{exp}(\mathbf {\rho }))\)  to have \(\mathbf {\sigma }\)  always non-negative. Thus the sample is \(\mathbf {w} = \mathbf {\mu } + \log (1 + \text{exp}(\mathbf {\rho }))\odot \mathbf {\epsilon }\) , where \(\odot \)  is point-wise
multiplication, \(\mathbf {\epsilon }\)  is drawn from \(\mathbf {\mathcal {N}}(\mathbf {0},\mathbf {I})\)  and the variational posterior parameters are \(\mathbf {\theta } = (\mathbf {\mu },\mathbf {\rho })\)  [2]}.
