Understanding theoretical properties of deep and locally connected nonlinear network, such as deep convolutional neural network (DCNN), is still a hard problem despite its empirical success. In this paper, we propose a novel theoretical framework for such networks with ReLU nonlinearity. The framework bridges data distribution with gradient descent rules, favors disentangled representations and is compatible with common regularization techniques such as Batch Norm, after a novel discovery of its projection nature. The framework is built upon teacher-student setting, by projecting the student's forward/backward pass onto the teacher's computational graph. We do not impose unrealistic assumptions (e.g., Gaussian inputs, independence of activation, etc). Our framework could help facilitate theoretical analysis of many practical issues, e.g. disentangled representations in deep networks. Deep Convolutional Neural Network (DCNN) has achieved a huge empirical success in multiple disciplines (e.g., computer vision BID0 BID10 He et al., 2016) , Computer Go BID8 BID12 BID13 , and so on). On the other hand, its theoretical properties remain an open problem and an active research topic.Learning deep models are often treated as non-convex optimization in a high-dimensional space. From this perspective, many properties in deep models have been analyzed: landscapes of loss functions (Choromanska et al., 2015b; BID1 BID3 , saddle points (Du et al., 2017; Dauphin et al., 2014) , relationships between local minima and global minimum (Kawaguchi, 2016; Hardt & Ma, 2017; BID5 , trajectories of gradient descent (Goodfellow et al., 2014) , path between local minima BID15 , etc.However, such a modeling misses two components: neither specific network structures nor input data distribution is considered. Both are critical in practice. Empirically, deep models work particular well for certain forms of data (e.g., images); theoretically, for certain data distribution, popular methods like gradient descent is shown to fail to recover network parameters (Brutzkus & Globerson, 2017) .Along this direction, previous theoretical works assume specific data distributions like spherical Gaussian and focus on shallow nonlinear networks BID12 Brutzkus & Globerson, 2017; Du et al., 2018) . These assumptions yield nice gradient forms and enable analysis of many properties such as global convergence. However , it is also nontrivial to extend such approaches to deep nonlinear neural networks that yield strong empirical performance.In this paper, we propose a novel theoretical framework for deep and locally connected ReLU network that is applicable to general data distributions. Specifically , we embrace a teacher-student setting. The teacher computes classification labels via a computational graph that has local structures (e.g., CNN): intermediate variables in the graph, (called summarization variables), are computed from a subset of the input dimensions. The student network, with similar local structures, updates the weights to fit teacher's labels with gradient descent, without knowing the summarization variables.One ultimate goal is to show that after training, each node in the student network is highly selective with respect to the summarization variable in the teacher. Achieving this goal will shed light to how the training of practically effective methods like CNN works, which remains a grand challenge. As a first step , we reformulate the forward/backward pass in gradient descent by marginalizing out the input data conditioned on the graph variables of the teacher at each layer. The reformulation has nice properties: (1) it relates data distribution with gradient update rules, (2) it is compatible with existing Receptive fields form a hierarchy. The entire input is denoted as x (or x ω ). A local region of an input x is denoted as x α . (b) For each region α, we have a latent multinomial discrete variable z α which is computed from its immediate children {z β } β∈ch (α) . Given the input x, z α = z α (x α ) is a function of the image content x α at α. Finally, z ω at the top level is the class label. (c) A locally connected neural network is trained with pairs (x, z ω (x)), where z ω (x) is the class label generated from the teacher. (d) For each node j, f j (x) is the activation while g j (x) is the back-propagated gradient, both as function of input x (and weights at different layers).state-of-the-art regularization techniques such as Batch Normalization (Ioffe & Szegedy, 2015) , and (3) it favors disentangled representation when data distributions have factorizable structures. To our best knowledge, our work is the first theoretical framework to achieve these properties for deep and locally connected nonlinear networks.Previous works have also proposed framework to explain deep networks, e.g., renormalization group for restricted Boltzmann machines BID2 , spin-glass models (Amit et al., 1985; Choromanska et al., 2015a) , transient chaos models BID4 , differential equations BID11 BID6 , information bottleneck (Achille & Soatto, 2017; BID14 BID7 , etc. In comparison, our framework (1 ) imposes mild assumptions rather than unrealistic ones (e.g., independence of activations), (2) explicitly deals with back-propagation which is the dominant approach used for training in practice, and relates it with data distribution, and (3) considers spatial locality of neurons, an important component in practical deep models. In this paper, we propose a novel theoretical framework for deep (multi-layered) nonlinear network with ReLU activation and local receptive fields. The framework utilizes the specific structure of neural networks, and formulates input data distributions explicitly. Compared to modeling deep models as non-convex problems, our framework reveals more structures of the network; compared to recent works that also take data distribution into considerations, our theoretical framework can model deep networks without imposing idealistic analytic distribution of data like Gaussian inputs or independent activations. Besides, we also analyze regularization techniques like Batch Norm, depicts its underlying geometrical intuition, and shows that BN is compatible with our framework.Using this novel framework, we have made an initial attempt to analyze many important and practical issues in deep models, and provides a novel perspective on overfitting, generalization, disentangled representation, etc. We emphasize that in this work, we barely touch the surface of these core issues in deep learning. As a future work, we aim to explore them in a deeper and more thorough manner, by using the powerful theoretical framework proposed in this paper.
This paper presents a theoretical framework that models data distribution explicitly for deep and locally connected ReLU network
{"src": "scitldr-AIC_train", "paper_id": "SyeKf30cFQ"}