Distillation is a method to transfer knowledge from one model to another and often achieves higher accuracy with the same capacity. In this paper, we aim to provide a theoretical understanding on what mainly helps with the distillation. Our answer is "early stopping". Assuming that the teacher network is overparameterized, we argue that the teacher network is essentially harvesting dark knowledge from the data via early stopping. This can be justified by a new concept, Anisotropic In- formation Retrieval (AIR), which means that the neural network tends to fit the informative information first and the non-informative information (including noise) later. Motivated by the recent development on theoretically analyzing overparame- terized neural networks, we can characterize AIR by the eigenspace of the Neural Tangent Kernel(NTK). AIR facilities a new understanding of distillation. With that, we further utilize distillation to refine noisy labels. We propose a self-distillation al- gorithm to sequentially distill knowledge from the network in the previous training epoch to avoid memorizing the wrong labels. We also demonstrate, both theoret- ically and empirically, that self-distillation can benefit from more than just early stopping. Theoretically, we prove convergence of the proposed algorithm to the ground truth labels for randomly initialized overparameterized neural networks in terms of l2 distance, while the previous result was on convergence in 0-1 loss. The theoretical result ensures the learned neural network enjoy a margin on the training data which leads to better generalization. Empirically, we achieve better testing accuracy and entirely avoid early stopping which makes the algorithm more user-friendly.
 Deep learning achieves state-of-the-art results in many tasks in computer vision and natural language processing LeCun et al. (2015) . Among these tasks, image classification is considered as one of the fundamental tasks since classification networks are commonly used as base networks for other problems. In order to achieve higher accuracy using a network with similar complexity as the base network, distillation has been proposed, which aims to utilize the prediction of one (teacher) network to guide the training of another (student) network. In Hinton et al. (2015) , the authors suggested to generate a soft target by a heavy-duty teacher network to guide the training of a light-weighted student network. More interestingly, Furlanello et al. (2018) ; Bagherinezhad et al. (2018) proposed to train a student network parameterized identically as the teacher network. Surprisingly, the student network significantly outperforms the teacher network. Later, it was suggested by Zagoruyko & Komodakis (2016a) ; Huang & Wang (2017) ; Czarnecki et al. (2017) to transfer knowledge of representations, such as attention maps and gradients of the classifier, to help with the training of the student network. In this work, we focus on the distillation utilizing the network outputs Hinton et al. (2015) ; Furlanello et al. (2018) ; Yang et al. (2018a) ; Bagherinezhad et al. (2018) ; Yang et al. (2018b) . To explain the effectiveness of distillation, Hinton et al. (2015) suggested that instead of the hard labels (i.e one-hot vectors), the soft labels generated by the pre-trained teacher network provide extra information, which is called the "Dark Knowledge". The "Dark knowledge" is the knowledge encoded by the relative probabilities of the incorrect outputs. In Hinton et al. (2015) ; Furlanello et al. (2018) ; Yang et al. (2018a) , the authors pointed out that secondary information, i.e the semantic similarity between different classes, is part of the "Dark Knowledge", and Bagherinezhad et al. (2018) observed that the "Dark Knowledge" can help to refine noisy labels. In this paper, we would like to answer the following question: can we theoretically explain how neural networks learn the Dark Knowledge? Answering this question will help us to understand the regularization effect of distillation. In this work, we assume that the teacher network is overparameterized, which means that it can memorize all the labels via gradient descent training Du et al. (2018b; a) ; Oymak & Soltanolkotabi (2018) ; Allen-Zhu et al. (2018) . In this case, if we train the overparameterized teacher network until convergence, the network's output coincides exactly with the ground truth hard labels. This is because the logits corresponding to the incorrect classes are all zero, and hence no "Dark knowledge" can be extracted. Thus, we claim that the core factor that enables an overparameterized network to learn "Dark knowledge" is early stopping. What's more, Arpit et al. (2017) ; Rahaman et al. (2018) ; Xu et al. (2019) observed that "Dark knowledge" represents the discrepancy of convergence speed of different types of information during the training of the neural network. Neural network tends to fit informative information, such as simple pattern, faster than non-informative and unwanted information such as noise. Similar phenomenon was observed in the inverse scale space theory for image restoration Scherzer & Groetsch (2001) ; Burger et al. (2006) ; Xu & Osher (2007) ; Shi & Osher (2008) . In our paper, we call this effect Anisotropic Information Retrieval (AIR). With the aforementioned interpretation of distillation, We further utilize AIR to refine noisy labels by introducing a new self-distillation algorithm. To extract anisotropic information, we sequentially extract knowledge from the output of the network in the previous epoch to supervise the training in the next epoch. By dynamically adjusting the strength of the supervision, we can theoretically prove that the proposed self-distillation algorithm can recover the correct labels, and empirically the algorithm achieves the state-of-the-art results on Fashion MNIST and CIFAR10. The benefit brought by our theoretical study is twofold. Firstly, the existing approach using large networks ; Zhang & Sabuncu (2018) often requires a validation set to early terminate the network training. However, our analysis shows that our algorithm can sustain long training without overfitting the noise which makes the proposed algorithm more user-friendly. Secondly, our analysis is based on an 2 -loss of the clean labels which enables the algorithm to generate a trained network with a bigger margin and hence generalize better. This paper provided an understanding of distillation using overparameterized neural networks. We observed that such neural networks posses the property of Anisotropic Information Retrieval (AIR), which means the neural network tends to fit the infomrative information (i.e. the eigenspaces associated with the largest few eigenvalues of NTK) first and the non-informative information later. Through AIR, we further observed that distillation of the Dark Knowledge is mainly due to early stopping. Based on this new understanding, we proposed a new self-distillation algorithm for noisy label refinery. Both theoretical and empirical justifications of the performance of the new algorithm were provided. Our analysis is based on the assumption that the teacher neural network is overparameterized. When the teacher network is not overparameterized, the network will be biased towards the label even without early stopping. It is still an interesting and unclear problem that whether the bias can provide us with more information. For label refinery, our analysis is mostly based on the symmetric noise setting. We are interested in extending our analysis to the asymmetric setting. A PROOF DETAILS A.1 NEURAL NETWORK PROPERTIES As preliminaries, we first discuss some properties of the neural network. We begin with the jacobian of the one layer neural network x → v φ(W x), the Jacobian matrix with respect to W takes the form First we borrow Lemma 6.6, 6.7, 6.8 from Oymak & Soltanolkotabi (2018) and Theorem 6.7, 6.8 from . T be a data matrix made up of data with unit Euclidean norm. Assuming that λ(X) > 0, the following properties hold. , at random Gaussian initialization W 0 ∼ N (0, 1) k×d , with probability at least 1 − δ, we have T in whichx i corresponds to the center of cluster including x i . What's more, we define the matrix of cluster center C = [c 1 , c 2 , . . . , c K ] T . Assuming that λ(C) > 0, the following properties hold. • , at random Gaussian initialization W 0 ∼ N (0, 1) k×d , with probability at least 1 − δ, we have • range(J(W,X)) ⊂ S + for any parameter matrix W . Then, we gives out the perturbation analysis of the Jacobian matrix. Lemma 3. Let X be a -clusterable data matrix with its center matrixX. For parameter matrices W,W , we have Proof. We bound J(W, X) − J(W ,X) by The first term is bounded by Lemma 1. As to the second term, we bound it by Combining the inequality above, we get Lemma 4. Let X be a -clusterable data matrix with its center matrixX. We assume W 1 , W 2 have a upper bound c √ k. Then for parameter matrices W 1 , W 2 ,W 1 ,W 2 , we have Proof. By the definition of average Jacobian, we have A.2 PROVE OF THE THEOREM First, we introduce the proof idea of our theorem. Our proof of the theorem divides the learning process into two stages. During the first stage, we aim to prove that the neural network will give out the right classification, i.e. the 0-1-loss converges to 0. The proof in this part is modified from . Furthermore, we proved that training 0-1-loss will keep 0 until the second stage starts and the margin at the first stage will larger than 1−2ρ 2 . During the second stage, we prove that the neural networks start to further enlarge the margin and finally the 2 loss starts to converge to zero. (2019) has shown that this dynamic can be illustrated by the average Jacobian. Definition 3. We define the average Jacobian for two parameters W 1 and W 2 and data matrix X as The residualr = f (θ) − y, r = f (θ) − y obey the following equation r = (I − ηC(θ))r In our proof, we project the residual to the following subspace Definition 4. Let {x i } n i=1 be a -clusterable dataset and {x i } n i=1 be the associated cluster centers, that is,x i = c l iff x i is from lth cluster. We define the support subspace S + as a subspace of dimension K, dictated by the cluster membership as follows. Let Λ l ⊂ {1, 2, · · · , n} be the set of coordinates i such that = c l . Then S + is characterized by Definition 5. We define the minimum eigenvalue of a matrix B on a subspace S σ min (B, S) = min where P S is the projection to the space S. Recall the generation process of the dataset Definition 6. (Clusterable Dataset Descriptions) • We assume that {x i } i∈[n] contains points with unit Euclidean norm and has K clusters. Let n l be the number of points in the lth cluster. Assume that number of data in each cluster is balanced in the sense that n l ≥ c low n K for constant c low > 0. • For each of the K clusters, we assume that all the input data lie within the Euclidean ball B(c l , ), where c l is the center with unit Euclidean norm and > 0 is the radius. • A dataset satisfying the above assumptions is called an -clusterable dataset.
theoretically understand the regularization effect of distillation. We show that early stopping is essential in this process. From this perspective, we developed a distillation method for learning with corrupted Label with theoretical guarantees.
{"src": "scitldr-AIC_train", "paper_id": "HJlF3h4FvB"}