We propose a new approach, known as the iterative regularized dual averaging (iRDA), to improve the efficiency of convolutional neural networks (CNN) by significantly reducing the redundancy of the model without reducing its accuracy.   The method has been tested for various data sets, and proven to be significantly more efficient than most existing compressing techniques in the deep learning literature.   For many popular data sets such as MNIST and CIFAR-10, more than 95% of the weights can be zeroed out without losing accuracy. In particular, we are able to make ResNet18 with 95% sparsity to have an accuracy that is comparable to that of a much larger model ResNet50 with the best 60% sparsity as reported in the literature. In recent decades, deep neural network models have achieved unprecedented success and state-ofthe-art performance in various tasks of machine learning or artificial intelligence, such as computer vision, natural language processing and reinforcement learning BID11 . Deep learning models usually involve a huge number of parameters to fit variant kinds of datasets, and the number of data may be much less than the amount of parameters BID9 . This may implicate that deep learning models have too much redundancy. This can be validated by the literatures from the general pruning methods BID18 to the compressing models BID6 .While compressed sensing techniques have been successfully applied in many other problems, few reports could be found in the literature for their application in deep learning. The idea of sparsifying machine learning models has attracted much attention in the last ten years in machine learning BID2 ; BID22 . When considering the memory and computing cost for some certain applications such as Apps in mobile, the sparsity of parameters plays a very important role in model compression BID6 ; BID0 . The topic of computing sparse neural networks can be included in the bigger topic on the compression of neural networks, which usually further involves the speedup of computing the compressed models.There are many sparse methods in machine learning models such as FOBOS method BID3 , also known as proximal stochastic gradient descent (prox-SGD) methods BID16 , proposed for general regularized convex optimization problem, where 1 is a common regularization term. One drawback of prox-SGD is that the thresholding parameters will decay in the training process, which results in unsatisfactory sparsity BID22 . Apart from that, the regularized dual averaging (RDA) method BID22 , proposed to obtain better sparsity, has been proven to be convergent with specific parameters in convex optimization problem, but has not been applied in deep learning fields.In this paper, we analyze the relation between simple dual averaging (SDA) method BID17 and the stochastic gradient descent (SGD) method BID19 , as well as the relation between SDA and RDA. It is well-known that SGD and its variants work quite well in deep learning problems. However, there are few literatures in applying pure training algorithms to deep CNNs for model sparsification. We propose an iterative RDA (iRDA) method for training sparse CNN models, and prove the convergence under convex conditions. Numerically, we compare prox-SGD with iRDA, where the latter can achieve better sparsity results while keeping satisfactory accuracy on MNIST, CIFAR-10 and CIFAR-100. We also show iRDA works for different CNN models such as VGG BID21 and BID9 . Finally, we compare the performance of iRDA with some other state-of-the-art compression methods. BID0 reviews the work on compressing neural network models, and categorizes the related methods into four schemes: parameter pruning and sharing, low-rank factorization, transfered/compact convolutional filters and knowledge distillation. Among them, BID14 uses sparse decomposition on the convolutional filters to get sparse neural networks, which could be classified to the second scheme. Apart from that, BID7 prunes redundant connections by learning only the important parts. BID15 starts from a Bayesian point of view, and removes large parts of the network through sparsity inducing priors. BID23 BID10 combines reinforcement learning methods to compression. BID13 considers deep learning as a discrete-time optimal control problem, and obtains sparse weights on ternary networks. Recently, BID4 applies RDA to fully-connected neural network models on MNIST. In comparison with many existing rule-based heuristic approaches, the new approach is based on a careful and iterative combination of 1 regularization and some specialized training algorithms. We find that the commonly used training algorithms such as SGD methods are not effective. We thus develop iRDA method that can be used to achieve much better sparsity. iRDA is a variant of RDA methods that have been used for some special types of online convex optimization problems in the literature. New elements in the iRDA mainly consist of judicious initialization and iterative retraining. In addition, iRDA method is carefully analyzed on its convergence for convex objective functions.Many deep neural networks trained by iRDA can achieve good sparsity while keeping the same validation accuracy as those trained by SGD with momentum on many popular datasets. This result shows iRDA is a powerful sparse optimization method for image classification problems in deep learning fields. One of the differences between RDA Xiao (2010) and iRDA is that the former one takes w 1 = arg min w h(w) whereas the latter one chooses w 1 randomly. In the following, we will prove the convergence of iRDA Step 1 for convex problem. The proofs use Lemma 9, Lemma 10, Lemma 11 directly and modify Theorem 1 and Theorem 2 in BID22 . For clarity, we have some general assumptions:• The regularization term Ψ(w) is a closed convex function with convexity parameter σ and domΨ is closed.• For each t ≥ 1, f t (w) is convex and subdifferentiable on domΨ.• h(w) is strongly convex on domΨ and subdifferentiable on rint(domΨ) and also satisfies DISPLAYFORM0 Without loss of generality, assume h(w) has convexity parameter 1 and min w h(w) = 0.• There exist a constant G such that DISPLAYFORM1 • Require {β} t≥1 be a nonnegative and nondecreasing sequence and DISPLAYFORM2 Moreover, we could always choose β 1 ≥ σ such that β 0 = β 1 .• For a random choosing w 1 , we assume DISPLAYFORM3 First of all, we define two functions: DISPLAYFORM4 DISPLAYFORM5 The maximum in (37) is always achieved because F D = {w ∈ domΨ|h(w) ≤ D 2 } is a nonempty compact set. Because of (35), we have σt+β t ≥ β 0 > 0 for all t ≥ 0, which means tΨ(w)+β t h(w) are all strongly convex, therefore the maximum in (38) is always achieved and unique. As a result , we have domU t = domV t = E * for all t ≥ 0. Moreover, by the assumption (33), both of the functions are nonnegative. Let s t denote the sum of the subgradients obtained up to time t in iRDA Step 1, that is DISPLAYFORM6 and π t (s) denotes the unique maximizer in the definition of V t (s) DISPLAYFORM7 which then gives DISPLAYFORM8 Lemma A.1 For any s ∈ E * and t ≥ 0, we have DISPLAYFORM9 For a proof, see Lemma 9 in Xiao (2010).Lemma A.2 The function V t is convex and differentiable. Its gradient is given by DISPLAYFORM10 and the gradient Lipschitz continuous with constant 1/(σt + β t ), that is DISPLAYFORM11 Moreover, the following inequality holds: DISPLAYFORM12 The results are from Lemma 10 in BID22 .Lemma A.3 For each t ≥ 1, we have DISPLAYFORM13 Since h(w t+1 ) ≥ 0 and the sequence {β t } t≥1 is nondecreasing, we have DISPLAYFORM14 DISPLAYFORM15 To prove this lemma, we refer to the Lemma 11 in Xiao (2010). What's more, from the assumption 35, we could always choose β 1 ≥ σ such that β 1 = β 0 and DISPLAYFORM16 The learner's regret of online learning is the difference between his cumulative loss and the cumulative loss of the optimal fixed hypothesis, which is defined by DISPLAYFORM17 and bounded by DISPLAYFORM18 Lemma A.4 Let the sequence {w t } t≥1 and {g t } t≥1 be generated by iRDA Step 1, and assume FORMULA2 and FORMULA2 hold. Then for any t ≥ 1 and any DISPLAYFORM19 Proof First, we define the following gap sequence which measures the quality of the solutions w 1 , .., w t : DISPLAYFORM20 and δ t is an upper bound on the regret R t (w) for all w ∈ F D , to see this, we use the convexity of f t (w) in the following: DISPLAYFORM21 Then, We are going to derive an upper bound on δ t . For this purpose, we subtract t τ =1 g τ , w 0 in (53), which leads to DISPLAYFORM22 the maximization term in (55) is in fact U t (−s t ), therefore, by applying Lemma A.1, we have DISPLAYFORM23 Next, we show that ∆ t is an upper bound for the right-hand side of inequality (56). We consider τ ≥ 2 and τ = 1 respectively. For any τ ≥ 2, we have DISPLAYFORM24 where FORMULA3 , FORMULA2 , FORMULA3 and FORMULA2 are used. Therefore, we have DISPLAYFORM25 , ∀τ ≥ 2.For τ = 1, we have a similar inequality by using (49) DISPLAYFORM26 Summing the above inequalities for τ = 1, ..., t and noting that V 0 (−s 0 ) = V 0 = 0, we arrive at DISPLAYFORM27 Since Ψ(w t+1 ) ≥ 0, we subtract it from the left hand side and add Ψ(w 1 ) to both sides of the above inequality yields DISPLAYFORM28 Combing FORMULA3 , FORMULA4 , (57) and using assumption (34) and (36)we conclude DISPLAYFORM29 Lemma A.5 Assume there exists an optimal solution w to the problem (3) that satisfies h(w ) ≤ D 2 for some D > 0, and let φ = φ(w ). Let the sequences {w t } t≥1 be generated by iRDA Step 1, and assume g t * ≤ G for some constant G. Then for any t ≥ 1, the expected cost associated with the random variablew t is bounded as DISPLAYFORM30 Proof First, from the definition (50), we have the regret at w DISPLAYFORM31 Let z[t] denote the collection of i.i.d. random variables (z , ..., z t ). We note that the random variable w τ , where 1 ≤ w ≥ t, is a function of (z 1 , ..., z τ −1 ) and is independent of (z τ , ..., z t ). Therefore DISPLAYFORM32 and DISPLAYFORM33 Since φ = φ(w ) = min w φ(w), we have the expected regret DISPLAYFORM34 Then, by convexity of φ, we have DISPLAYFORM35 Finally, from FORMULA4 and FORMULA4 , we have DISPLAYFORM36 Then the desired follows from that of Lemma A.4. Proof of Theorem 3.1 From Lemma A.5, the expected cost associated with the random variablew t is bounded as DISPLAYFORM37 Here, we consider 1 regularization function Ψ(w) = λ w 1 and it is a convex but not strongly convex function, which means σ = 0. Now, we consider how to choose β t for t ≥ 1 and β 0 = β 1 . First if β t = γt, we have 1 t · γtD 2 = γD 2 , which means the expected cost does not converge. Then assume β t = γt α , α > 0 and α = 1, the right hand side of the inequality (60) becomes DISPLAYFORM38 From above, we see that if 0 < α < 1, the expected cost converges and the optimal convergence rate O(t We have shown why prox-SGD will give poor sparsity, and although √ t-prox-SGD may introduce greater sparsity, it is not convergent. Finally, iRDA gives the best result , on both the top-1 accuracy and the sparsity. iRDA (
A sparse optimization algorithm for deep CNN models.
{"src": "scitldr-AIC_train", "paper_id": "HJMXus0ct7"}