We extend the recent results of (Arora et al., 2019) by a spectral analysis of representations corresponding to kernel and neural embeddings. They showed that in a simple single layer network, the alignment of the labels to the eigenvectors of the corresponding Gram matrix determines both the convergence of the optimization during training as well as the generalization properties. We generalize their result to kernel and neural representations and show that these extensions improve both optimization and generalization of the basic setup studied in (Arora et al., 2019). The well-known work of BID8 highlighted intriguing experimental phenomena about deep net trainingspecifically, optimization and generalization -and called for a rethinking of generalization in statistical learning theory. In particular, two fundamental questions that need understanding are: Optimization. Why do true labels give faster convergence rate than random labels for gradient descent? Generalization. What property of properly labeled data controls generalization? BID0 have recently tried to answer this question in a simple model by conducting a spectral analysis of the associated Gram matrix. They show that both training and generalization are better if the label vector aligns with the top eigenvectors.However, their analysis applies only to a simple two layer network. How could their insights be extended to deeper networks?A widely held intuitive view is that deep layers generate expressive representations of the raw input data. Adopting this view, one may consider a model where a representation generated by successive neural network layers is viewed as a kernel embedding which is then fed into the two-layer model of BID0 . The connection between neural networks and kernel machines has long been studied; BID2 ) introduced kernels that mimic deep networks and BID6 showed kernels equivalent to certain feed-forward neural networks. Recently, BID1 ) also make the case that progress on understanding deep learning is unlikely to move forward until similar phenomena in classical kernel machines are recognized and understood. Very recently, BID4 showed that the evolution of a neural network during training can be related to a new kernel, the Neural Tangent Kernel (NTK) which is central to describe the generalization properties of the network.Here we pursue this approach by studying the effect of incorporating embeddings in the simple two layer model and we perform a spectral analysis of these embeddings along the lines of BID0 . We can obtain embeddings in several ways: i. We can use an unbiased kernel such as Gaussian kernel. This choice is consistent with the maximum entropy principle and makes no prior assumption about the data. Or use a kernel which mimics or approximates deep networks ii. We could use data driven embeddings explicitly produced by the hidden layers in neural networks: either use a subset of the same training data to compute such an embedding, or transfer the inferred embedding from a different (but similar) domain.While a general transformation g(x) of the input data may have arbitrary effects, one would expect kernel and neural representations to improve performance. The interplay of kernels and data labellings has been addressed before, for example in the work of kernel-target alignment BID3 ).We do indeed observe a significant beneficial effect: Optimization. Using kernel methods such as random Fourier features (RFF) to approximate the Gaussian kernel embedding BID5 and neural embeddings, we obtain substantially better convergence in training. Generalization. We also achieve significantly lower test error and we confirm that the data dependent spectral measure introduced in BID0 significantly improves with kernel and neural embeddings.Thus this work shows empirically that kernel and neural embeddings improve the alignment of target labels to the eigenvectors of the Gram matrix and thus help training and generalization. This suggests a way to extend the insights of BID0 ) to deeper networks, and possible theoretical results in this direction. We extended the recent results of BID0 by a spectral analysis of the representations corresponding to kernel and neural embeddings and showed that such representations benefit both optimization and generalization. By combining recent results connecting kernel embeddings to neural networks such as BID6 BID4 , one may be able to extend the fine-grained theoretical results of BID0 for two layer networks to deeper networks.
Spectral analysis for understanding how different representations can improve optimization and generalization.
{"src": "scitldr-AIC_train", "paper_id": "rkgcikhcT4"}