Edge intelligence especially binary neural network (BNN) has attracted considerable attention of the artificial intelligence community recently. BNNs significantly reduce the computational cost, model size, and memory footprint.   However, there is still a performance gap between the successful full-precision neural network with ReLU activation and BNNs. We argue that the accuracy drop of BNNs is due to their geometry. 
 We analyze the behaviour of the full-precision neural network with ReLU activation and compare it with its binarized counterpart. This comparison suggests random bias initialization as a remedy to activation saturation in full-precision networks and  leads us towards an improved BNN training. Our numerical experiments confirm our geometric intuition. Convolutional neural network has become one of the most powerful tools for solving computer vision, natural language processing, speech recognition, machine translation, and many other complex tasks. The most successful and widely-used recipe for deep neural network is ReLU-style activation function with MSRA style weight initialization (He et al., 2015) . The standard sigmoid and the hyperbolic tangent were the most common activation functions, before the introductio of ReLU. ReLU-like activation functions are widely proved to be superior in terms of accuracy and convergence speed. It is more common to use low-bit quantized networks such as Binary Neural Networks (BNNs) (Courbariaux et al., 2016) to implement such deep neural networks on edge devices such as cell phones, smart wearables, etc. BNNs only keeps the sign of weights and compute the sign of activations {−1, +1} by applying Sign function in the forward pass. In backward propagation, BNN uses Straight-Through-Estimator (STE) to estimate the backward gradient through the Sign function and update on full-precision weights. The forward and backward loop of a BNN, therefore, becomes similar to the full-precision neural network with hard hyperbolic tangent htanh activation. The htanh function is a piece-wise linear version of the nonlinear hyper-bolic tangent, and is known to be inferior in terms of accuracy compared to ReLU-like activation function. We examine a full-precision network with htanh activation to provide a new look in improving BNN performance. We conclude that the bias initialization is the key to mimic ReLU geometric behavior in networks with htanh activation. This conclusion challenges the common practice of deterministic bias initialization for neural networks. Although the analysis is based on htanh function, this conclusion equally applies to BNNs that use STE, a htanh-like, back propagation scheme. Other saturating activations like hyperbolic tangent and sigmoid commonly applied in recurrent neural networks may benefit from this resolution as well. Our novelties can be summarized in four items i) we analyze the geometric properties of ReLU and htanh activation. This provides an insight into the training efficiency of the unbounded asymmetric activation functions such as ReLU. ii) we propose bias initialization strategy as a remedy to the bounded activations such as htanh. iii) We back up our findings with experiments on full-precision to reduce the performance gap between htanh and ReLU activations. iv) We show this strategy also improves BNNs, whose geometric behavior is similar to the full-precision neural network with htanh activation. There are very few works that focus on the initialization strategy of the bias term of the neural network. To the best of our knowledge, we are the first to propose random bias initialization as a remedy to the saturating full-precision neural network, also as a method to improve BNN training. 2 RELATED WORKS (Glorot et al., 2011) proposed training deep neurals network with ReLU activation, and argued that ReLU activation alleviates the vanishing gradient problem and encourages sparsity in the model. The hyperbolic tangent only allowed training of shallow neural networks. Since AlexNet (Krizhevsky et al., 2012) , almost every successful neural network architectures use ReLU activation or its variants, such as adaptive ReLU, leaky ReLU, etc. Although many works reported that ReLU activation outperforms the traditional saturating activation functions, the reason for its superior performance remains an open question. (Ramachandran et al., 2017) utilized automatic search techniques on searching different activation functions. Most top novel activation functions found by the searches have an asymmetric saturating regime, which is similar to ReLU. Farhadi et al. (2019) adapts ReLU and sigmoid while training. To improve the performance of saturating activations, Xu et al. (2016) proposed penalized tanh activation, which introduces asymmetric saturating regime to tanh by inserting leaky ReLU before tanh. The penalized tanh could achieve the same level of performance as ReLU activating CNN. It is worth to mention that similar ideas also appear in the related works of binarized neural network. Gulcehre et al. (2016) improved the performance of saturating activations by adding random noise when the neuron is saturated, so the backward signal can easily pass through the whole model, and the model becomes easier to optimize. In this works, we proposed to randomize the non-saturated regime by using random bias initialization. This initialization can guarantee all backward signals can pass through the whole model equally. The initial work on BNN appeared in Courbariaux et al. (2016) , which limits both weights and activations to −1 and +1, so the weighted sum can be computed by bit-wise XNOR and PopCount instructions. This solution reduces memory usage and computational cost up to 32X compared with its full-precision counterpart. In the original paper, BNN was tested on VGG-7 architecture. Although it is an over-parameterized architecture for CIFAR 10 dataset, there is a performance gap between BNN and full-precision with ReLU activation. We believe the different between the two activations, BNN using the sign and full-precision using ReLU, is partially responsible for this gap. XNOR-Net (Rastegari et al., 2016) developed the idea of BNN and proposed to approximate the full-precision neural network by using scaling factors. They suggest inserting non-Binary activation (like ReLU) after the binary convolution layer. This modification helps training considerably. Later, Tang et al. replaced replacing ReLU activation with PReLU in XNOR-Net to improve the accuracy. Note that XNOR-Net and many relaated works require to store the full-precision activation map during the inference stage, therefore their memory occupation is significantly larger than the pure 1-bit solution like the vanilla BNN.
Improve saturating activations (sigmoid, tanh, htanh etc.) and Binarized Neural Network with Bias Initialization
{"src": "scitldr-AIC_train", "paper_id": "SJx4Ogrtvr"}