BNN as Variational Inference (VI): Training binary neural networks can also be approached with VI; see among others [1]}, [2]}, [3]}.
Instead of optimizing binary weights, the parameters of Bernoulli distributions are learned using the VI Bayesian learning rule; see e.g. [4]}. Even if unbiased estimators of the ELBO are available, classical methods like MuProp [5]} or REINFORCE with variance-reduction baselines [6]} have a prohibitively high variance. The use of Gumbel-Softmax (GS) trick [7]}, [8]} has been advocated in [9]}, but as noted in [10]} there is an issue in the implementation which paradoxically enables the training. The connections between STE algorithms and their many variants - including MD - and VI methods are further discussed in [11]}.
