The increasing demand for neural networks (NNs) being employed on embedded devices has led to plenty of research investigating methods for training low precision NNs. While most methods involve a quantization step, we propose a principled Bayesian approach where we first infer a distribution over a discrete weight space from which we subsequently derive hardware-friendly low precision NNs. To this end, we introduce a probabilistic forward pass to approximate the intractable variational objective that allows us to optimize over discrete-valued weight distributions for NNs with sign activation functions. In our experiments, we show that our model achieves state of the art performance on several real world data sets. In addition, the resulting models exhibit a substantial amount of sparsity that can be utilized to further reduce the computational costs for inference. With the advent of deep neural networks (NNs) impressive performances have been achieved in many applications such as computer vision BID13 , speech recognition , and machine translation , among others. However, the performance improvements are largely attributed to increasing hardware capabilities that enabled the training of ever-increasing network architectures. On the other side, there is also a growing interest in making NNs available for embedded devices with drastic memory and power limitations -a field with plenty of interesting applications that barely profit from the tendency towards larger and deeper network structures.Thus, there is an emerging trend in developing NN architectures that allow fast and energy-efficient inference and require little storage for the parameters. In this paper, we focus on reduced precision methods that restrict the number of bits per weight while keeping the network structures at a decent size. While this reduces the memory footprint for the parameters accordingly, it can also result in drastic improvements in computation speed if appropriate representations for the weight values are used. This direction of research has been pushed towards NNs that require in the extreme case only a single bit per weight. In this case, assuming weights w ∈ {−1, 1} and binary inputs x ∈ {−1, 1}, costly floating point multiplications can be replaced by cheap and hardware-friendly logical XNOR operations. However, training such NNs is inherently different as discrete valued NNs cannot be directly optimized using gradient based methods. Furthermore, NNs with binary weights exhibit their full computational benefits only in case the sign activation function is used whose derivative is zero almost everywhere, and, therefore, is not suitable for backpropagation.Most methods for training reduced precision NNs either quantize the weights of pre-trained full precision NNs BID3 or train reduced precision NNs by maintaining a set of full precision weights that are deterministically or stochastically quantized during forward or backward propagation. Gradient updates computed with the quantized weights are then applied to the full precision weights BID4 BID19 BID8 . This approach alone fails if the sign activation function is used. A promising approach is based on the straight through gradient estimator (STE) BID1 which replaces the zero gradient of hard threshold functions by a non-zero surrogate derivative. This allows information in computation graphs to flow backwards such that parameters can be updated using gradient based optimization methods. Encouraging results are presented in BID8 where the STE is applied to the weight binarization and to the sign activation function. These methods, although showing The aim is to obtain a single discrete-valued NN (top right) with a good performance. We achieve this by training a distribution over discrete-valued NNs (bottom right) and subsequently deriving a single discrete-valued NN from that distribution. (b) Probabilistic forward pass: The idea is to propagate distributions through the network by approximating a sum over random variables by a Gaussian and subsequently propagating that Gaussian through the sign activation function.convincing empirical performance, have in common that they appear rather heuristic and it is usually not clear whether they optimize any well defined objective.Therefore, it is desired to develop principled methods that support discrete weights in NNs. In this paper, we propose a Bayesian approach where we first infer a distribution q(W ) over a discrete weight space from which we subsequently derive discrete-valued NNs. Thus, we can optimize over real-valued distribution parameters using gradient-based optimization instead of optimizing directly over the intractable combinatorial space of discrete weights. The distribution q(W ) can be seen as an exponentially large ensemble of NNs where each NN is weighted by its probability q(W ).Rather than having a single value for each connection of the NN, we now maintain a whole distribution for each connection (see bottom right of FIG0 (a)). To obtain q(W ), we employ variational inference where we approximate the true posterior p(W |D) by minimizing the variational objective KL(q(W )||p(W |D)). Although the variational objective is intractable, this idea has recently received a lot of attention for real-valued NNs due to the reparameterization trick which expresses gradients of intractable expectations as expectations of tractable gradients BID20 BID12 BID2 . This allows us to efficiently compute unbiased gradient samples of the intractable variational objective that can subsequently be used for stochastic optimization. Unfortunately , the reparameterization trick is only suitable for real-valued distributions which renders it unusable for our case. The recently proposed Gumbel softmax distribution BID10 BID16 overcomes this issue by relaxing one-hot encoded discrete distributions with probability vectors. Subsequently , the reparameterization trick can again be applied. However, for the sign activation function one still has to rely on the STE or similar heuristics. The log-derivative trick offers an alternative for discrete distributions to express gradients of expectations with expectations of gradients BID18 . However, the resulting gradient samples are known to suffer from high variance. Therefore, the log-derivative trick is typically impractical unless suitable variance reduction techniques are used. This lack of practical methods has led to a limited amount of literature investigating Bayesian NNs with discrete weights BID23 .In this work, we approximate the intractable variational objective with a probabilistic forward pass (PFP) BID26 BID23 BID6 BID21 . The idea is to propagate probabilities through the network by successively approximating the distributions of activations with a Gaussian and propagating this Gaussian through the sign activation function FIG0 ). This results in a well-defined objective whose gradient with respect to the variational parameters can be computed analytically. This is true for discrete weight distributions as well as for the sign activation function with zero gradient almost everywhere. The method is very flexible in the sense that different weight distributions can be used in different layers. We utilize this flexibility to represent the weights in the first layer with 3 bits and we use ternary weights w ∈ {−1, 0, 1} in the remaining layers.In our experiments, we evaluate the performance of our model by reporting the error of (i) the most probable model of the approximate posterior q(W ) and (ii) approximated expected predictions using the PFP. We show that averaging over small ensembles of NNs sampled from W ∼ q(W ) can improve the performance while inference using the ensemble is still cheaper than inference using a single full precision NN. Furthermore, our method exhibits a substantial amount of sparsity that further reduces the computational overhead. Compared to BID8 , our method requires less precision for the first layer, and we do not introduce a computational overhead by using batch normalization which appears to be a crucial component of their method.The paper is outlined as follows. In Section 2, we introduce the notation and formally define the PFP. Section 3 shows details of our model. Section 4 shows experiments. In Section 5 we discuss important issues concerning our model and Section 6 concludes the paper. The presented model has many tunable parameters, especially the type of variational distributions for the individual layers, that heavily influence the behavior in terms of convergence at training time and performance at test time. The binomial distribution appears to be a natural choice for evenly spaced values with many desirable properties. It is fully specified by only a single parameter, and its mean, variance, and KL divergence with another binomial has nice analytic expressions. Furthermore, neighboring values have similar probabilities which rules out odd cases in which, for instance, there is a value with low probability in between of two values with high probability.Unfortunately, the binomial distribution is not suited for the first layer as here it is crucial to be able to set weights with high confidence to zero. However, when favoring zero weights by setting w p = 0.5, the variance of the binomial distribution takes on its largest possible value. This might not be a problem in case predictions are computed as the true expectations with respect to q(W ) as in the PFP, but it results in bad classification errors when deriving a single model from q(W ). We also observed that using the binomial distribution in deeper layers favor the weights −1 and 1 over 0 (cf. TAB2 ). This might indicate that binary weights w ∈ {−1, 1} using a Bernoulli distribution could be sufficient, but in our experiments we observed this to perform worse. We believe this to stem partly from the importance of the zero weight and partly from the larger variance of 4w p (1 − w p ) of the Bernoulli distribution compared to the variance of 2w p (1 − w p ) of the binomial distribution.Furthermore, there is a general issue with the sign activation functions if the activations are close to zero. In this case, a small change to the inputs can cause the corresponding neuron to take on a completely different value which might have a large impact on the following layers of the NN. We found dropout to be a very helpful tool to counteract this issue. FIG1 shows histograms of the activations of the second hidden layer for both a model trained with dropout and the same model trained without dropout. We can see that without dropout the activations are much closer to zero whereas dropout introduces a much larger spread of the activations and even causes the histogram to decrease slightly in the vicinity of zero. Thus, the activations are much more often in regions that are stable with respect to changes of their inputs which makes them more robust. We believe that such regularization techniques are crucial if the sign activation function is used. We introduced a method to infer NNs with low precision weights. As opposed to existing methods, our model neither quantizes the weights of existing full precision NNs nor does it rely on heuristics to compute "approximated" gradients of functions whose gradient is zero almost everywhere. We perform variational inference to obtain a distribution over a discrete weight space from which we subsequently derive a single discrete-valued NN or a small ensemble of discrete-valued NNs. Our method propagates probabilities through the network which results in a well defined function that allows us to optimize the discrete distribution even for the sign activation function. The weights in the first layer are modeled using fixed point values with 3 bits precision and the weights in the remaining layers have values w ∈ {−1, 0, 1}. This reduces costly floating point multiplications to cheaper multiplications with fixed point values of 3 bits precision in the first layer, and logical XNOR operations in the following layers. In general, our approach allows flexible bit-widths for each individual layer. We have shown that the performance of our model is on par with state of the art methods that use a higher precision for the weights. Furthermore, our model exhibits a large amount of sparsity that can be utilized to further reduce the computational overhead.A DATA SETS
Variational Inference for infering a discrete distribution from which a low-precision neural network is derived
{"src": "scitldr-AIC_train", "paper_id": "r1h2DllAW"}