This paper explores the simplicity of learned neural networks under various settings: learned on real vs random data, varying size/architecture and using large minibatch size vs small minibatch size. The notion of simplicity used here is that of learnability i.e., how accurately can the prediction function of a neural network be learned from labeled samples from it. While learnability is different from (in fact often higher than) test accuracy, the results herein suggest that there is a strong correlation between small generalization errors and high learnability.
 This work also shows that there exist significant qualitative differences in shallow networks as compared to popular deep networks. More broadly, this paper extends in a new direction, previous work on understanding the properties of learned neural networks. Our hope is that such an empirical study of understanding learned neural networks might shed light on the right assumptions that can be made for a theoretical study of deep learning. Over the last few years neural networks have significantly advanced state of the art on several tasks such as image classification BID23 ), machine translation BID32 ), structured prediction BID2 ) and so on, and have transformed the areas of computer vision and natural language processing. Despite the success of neural networks in making these advances, the reasons for their success are not well understood. Understanding the performance of neural networks and reasons for their success are major open problems at the moment. Questions about the performance of neural networks can be broadly classified into two groups: i) optimization i.e., how are we able to train large neural networks well even though it is NP-hard to do so in the worst case, and ii) generalization i.e., how is it that the training error and test error are close to each other for large neural networks where the number of parameters in the network is much larger than the number of training examples (highly overparametrized). This paper explores three aspects of generalization in neural networks.The first aspect is the performance of neural networks on random training labels. While neural networks generalize well (i.e., training and test error are close to each other) on real datasets even in highly overparametrized settings, BID33 shows that neural networks are nevertheless capable of achieving zero training error on random training labels. Since any given network will have large error on random test labels, BID33 concludes that neural networks are indeed capable of poor generalization. However since the labels of the test set are random and completely independent of the training data, this leaves open the question of whether neural networks learn simple patterns even on random training data. Indeed the results of BID22 establish that even in the presence of massive label noise in the training data, neural networks obtain good test accuracy on real data. This suggests that neural networks might learn some simple patterns even with random training labels. The first question this paper asks is (Q1): Do neural networks learn simple patterns on random training data?A second, very curious, aspect about the generalization of neural networks is the observation that increasing the size of a neural network helps in achieving better test error even if a training error of zero has already been achieved (see, e.g., BID21 ) i.e., larger neural networks have better generalization error. This is contrary to traditional wisdom in statistical learning theory which holds that larger models give better training error but at the cost of higher generalization error. A recent line of work proposes that the reason for better generalization of larger neural networks is implicit regularization, or in other words larger learned models are simpler than smaller learned models. See Neyshabur (2017) for references. The second question this paper asks is (Q2): Do larger neural networks learn simpler patterns compared to smaller neural networks when trained on real data?The third aspect about generalization that this paper considers is the widely observed phenomenon that using large minibatches for stochastic gradient descent (SGD) leads to poor generalization LeCun et al..(Q3): Are neural networks learned with small minibatch sizes simpler compared to those learned with large minibatch sizes?All the above questions have been looked at from the point of view of flat/sharp minimizers BID11 . Here flat/sharp corresponds to the curvature of the loss function around the learned neural network. BID18 for true vs random data , BID24 for large vs small neural networks and BID16 for small vs large minibatch training, all look at the sharpness of minimizers in various settings and connect it to the generalization performance of neural networks. While there certainly seems to be a connection between the sharpness of the learned neural network, there is as yet no unambiguous notion of this sharpness to quantify it. See BID4 for more details. This paper takes a complementary approach: it looks at the above questions through the lens of learnability. Let us say we are considering a multi-class classification problem with c classes and let D denote a distribution over the inputs x ∼ R d . Given a neural network N , draw n independent samples x tr 1 , · · · , x tr n from D and train a neural network N on training data DISPLAYFORM0 The learnability of a neural network N is defined to be DISPLAYFORM1 Note that L(N ) implicitly depends on D, the architecture and learning algorithm used to learn N as well as n. This dependence is suppressed in the notation but will be clear from context. Intuitively, larger the L(N ), easier it is to learn N from data. This notion of learnability is not new and is very closely related to probably approximately correct (PAC) learnability Valiant (1984); BID15 . In the context of neural networks, learnability has been well studied from a theoretical point as we discuss briefly in Sec.2. There we also discuss some related empirical results but to the best of our knowledge there has been no work investigating the learnability of neural networks that are encountered in practice.This paper empirically investigates the learnability of neural networks of varying sizes/architectures and minibatch sizes, learned on real/random data in order to answer (Q1) and (Q2) and (Q3). The main contributions of this paper are as follows: DISPLAYFORM2 The results in this paper suggest that there is a strong correlation between generalizability and learnability of neural networks i.e., neural networks that generalize well are more learnable compared to those that do not generalize well. Our experiments suggest that• Neural networks do not learn simple patterns on random data.• Learned neural networks of large size/architectures that achieve higher accuracies are more learnable.• Neural networks learned using small minibatch sizes are more learnable compared to those learned using large minibatch sizes.Experiments also suggest that there are qualitative differences between learned shallow networks and deep networks and further investigation is warranted.Paper organization: The paper is organized as follows. Section 2 gives an overview of related work. Section 3 presents the experimental setup and results . Section 5 concludes the paper with some discussion of results and future directions. This paper explores the learnability of learned neural networks under various scenarios. The results herein suggest that while learnability is often higher than test accuracy, there is a strong correlation between low generalization error and high learnability of the learned neural networks. This paper also shows that there are some qualitative differences between shallow and popular deep neural networks. Some questions that this paper raises are the effect of optimization algorithms, hyperparameter selection and initialization schemes on learnability. On the theoretical front, it would be interesting to characterize neural networks that can be learned efficiently via backprop. Given the strong correlation between learnability and generalization, driving the network to converge to learnable networks might help achieve better generalization.
Exploring the Learnability of Learned Neural Networks
{"src": "scitldr-AIC_train", "paper_id": "rJ1RPJWAW"}