The passage from the hidden variables \({Z}\) , to their corresponding activation units \({A}\)  cannot be done analytically using non-linear activation functions. In order to work around this difficulty, we propose to employ functions that are linearized at \(\mathbb {E}[{Z}]={\mu _Z}\) . Linearized activation functions \(\tilde{\sigma }(\cdot )\)  allow calculating analytically the expected vector \(\mathbb {E}[{A}]\) , the covariance \(\text{cov}({A})\) , as well as the covariance between activation units and the weight and bias \(\text{cov}({A},{\theta })\) , where \({\theta }=\lbrace {W},{B}\rbrace \) . Figure REF  presents an example for the linearization of a softplus activation function. Note that a linearized activation function is not equivalent to having a linear activation function because for each instance of input covariates \({x}_i\) , the linearization is done at a different value \(\mu _z\)  which maintains the non-linear dependency between \({x}_i\)  and the output \({y}_i\) . It is known that this linearization procedure is an approximation of the change of variable rule that would be required to obtain true theoretical PDF for \(f(a)\) . It is also known that more advanced sampling-based approches exist for propagating uncertainty through non-linear function , e.g. [1]}. Nevertheless, we choose to employ the linearization procedure because of its minimal computational cost which still allows, as it will be shown in §, to match the state-of-the-art performance on equivalent neural networks architectures using backpropagation. Finally, note that the linearization procedure is compatible with all the common activation functions such as the relu, tanh, logistic sigmoid, etc.
<FIGURE>