Stochastic neural networks were introduced by Wong [1]} to generalize Hopfield networks [2]}. The theoretical features of stochastic deep neural networks (SDNNs) have been the object of study of many recent papers. In [3]}, the authors study universal approximation properties of deep belief networks, a class of stochastic feedforward networks that were pivotal in the resurgence of deep learning. A key observation is that, for a finite number of inputs and outputs, the number of maps in the stochastic setting is infinite, unlike in the deterministic framework. This points to the massive increase in the approximation power of SDNNs. In [4]}, SDNNs are framed as learning measures (LMs); deep architectures are introduced to address issues that arise in LMs such as permutation invariance or equivariance and variation in weights. Training stochastic feedforward networks is significantly more challenging. To address this issue, [5]} proposed a stochastic feedforward network with hidden layers composed of both deterministic and stochastic variables with a novel generalized EM training procedure that allows the user to efficiently learn complicated conditional distributions.
