One can substitute each neuron in any neural network with a kernel machine and obtain a counterpart powered by kernel machines. The new network inherits the expressive power and architecture of the original but works in a more intuitive way since each node enjoys the simple interpretation as a hyperplane (in a reproducing kernel Hilbert space). Further, using the kernel multilayer perceptron as an example, we prove that in classification, an optimal representation that minimizes the risk of the network can be characterized for each hidden layer. This result removes the need of backpropagation in learning the model and can be generalized to any feedforward kernel network. Moreover, unlike backpropagation, which turns models into black boxes, the optimal hidden representation enjoys an intuitive geometric interpretation, making the dynamics of learning in a deep kernel network simple to understand. Empirical results are provided to validate our theory. Any neural network (NN) can be turned into a kernel network (KN) by replacing each artificial neuron (McCulloch & Pitts, 1943) , i.e., learning machine of the form f (x) = σ(w x + b), with a kernel machine, i.e., learning machine of the form f (x) = w, φ(x ) + b with kernel function k(x, y ) = φ(x ), φ(y ) . This combination of connectionism and kernel method enables the learning of hierarchical, distributed representations with kernels.In terms of training, similar to NN, KN can be trained with backpropagation (BP) (Rumelhart et al., 1986) . In the context of supervised learning, the need for BP in learning a deep architecture is caused by the fact that there is no explicit target information to tune the hidden layers (Rumelhart et al., 1986) . Moreover , BP is usually computationally intensive and can suffer from vanishing gradient. And most importantly, BP results in hidden representations that are notoriously difficult to interpret or assess, turning deep architectures into "black boxes".The main theoretical contribution of this paper is the following: Employing the simplest feedforward, fully-connected KN as an example, we prove that in classification and under certain losses, the optimal representation for each hidden layer that minimizes the risk of the network can be explicitly characterized. This result removes the need for BP and makes it possible to train the network in a feedforward, layer-wise fashion. And the same idea can be generalized to other feedforward KNs.The layer-wise learning algorithm gives the same optimality guarantee as BP in the sense that it minimizes the risk. But the former is much faster and evidently less susceptible to vanishing gradient. Moreover, the quality of learning in the hidden layers can be directly assessed during or after training, providing more information about the model to the user. For practitioners , this enables completely new model selection paradigms. For example, the bad performance of the network can now be traced to a certain layer, allowing the user to debug the layers individually. Most importantly , the optimal representation for each hidden layer enjoys an intuitive geometric interpretation, making the learning dynamics in a deep KN more transparent than that in a deep NN. A simple acceleration method that utilizes the "sparse" nature of the optimal hidden representations is proposed to further reduce computational complexity.Empirical results on several computer vision benchmarks are provided to demonstrate the competence of the model and the effectiveness of the greedy learning algorithm. Figure 1: (a) Any NN (left, presented in the usual weight-nonlinearity abstraction) can be abstracted as a "graph" (right) with each node representing a neuron and each edge the input-output relationship between neurons. If a node receives multiple inputs, we view its input as a vector in some Euclidean space, as indicated by the colored rectangles. Under this abstraction, each neuron (f (x) = σ(w x + b)) can be directly replaced by a kernel machine (f (x) = w, φ(x) + b with kernel k(x , y) = φ(x ), φ(y) ) mapping from the same Euclidean space into the real line without altering the architecture and functionality of the model. (b) Illustration for layer-wise optimality drifting away from network-optimality. Consider a two-layer network and let T 1 , T 2 be the target function of the first and second layer, respectively. If the first layer creates error, which is illustrated by F (1) (x) being far away from T 1 (x), the composed solution F (2) • F (1) on the right is better than that on the left and hence the F (2) on the right corresponds to the network-wise optimality of the second layer. But the F (2) on the left is clearly a better estimate to the layer-wise optimality T 2 if the quality of estimation is measured by the supremum distance.
We combine kernel method with connectionist models and show that the resulting deep architectures can be trained layer-wise and have more transparent learning dynamics. 
{"src": "scitldr-AIC_train", "paper_id": "H1GLm2R9Km"}