
where the minibatches \(\lbrace (x_{i_1}, y_{i_1}), \ldots , (x_{i_M}, y_{i_M})) \rbrace \) , the \(M\)  samples \(\theta _m \sim q_\phi (\theta _m)\)  and \(M\cdot L\)  samples \(\zeta _{i_m, l} \sim \pi (\zeta _{i_m, l}|x_i, \sigma _x^2)\)  will be drawn anew in each optimization step.
Note that the way \(\theta _m\)  is sampled in (REF ) differs slightly from the way this is done in approaches such as [1]}, [2]}, [3]} as \(\theta _m\)  will be identical for all inputs \(\zeta _{i_m,l}\)  with the same \(m\) .For the approach from [3]} this also means that Gaussian dropout of “type A” has to be used. For some algorithms on the other hand, such as Bayes by Backprop [5]}, taking in (REF ) a single Monte Carlo sample \(\theta _m = \theta \)  for all \(m\)  instead might allow for a more efficient implementation. The loss function in (REF ) and (REF ) is the backbone of the EiV algorithm.
To make the first term in (REF ) numerically stable, and suitable for backpropagation, the common "logsumexp" function can be used.
Before we can use (REF ) for training however, we still have to fix the prior distributions and the variational distribution \(q_\phi \) .
