Traditionally, deep learning algorithms update the network weights whereas the network architecture is chosen manually, using a process of trial and error. In this work, we propose two novel approaches that automatically update the network structure while also learning its weights. The novelty of our approach lies in our parameterization where the depth, or additional complexity, is encapsulated continuously in the parameter space through control parameters that add additional complexity. We propose two methods: In tunnel networks, this selection is done at the level of a hidden unit, and in budding perceptrons, this is done at the level of a network layer; updating this control parameter introduces either another hidden unit or another hidden layer. We show the effectiveness of our methods on the synthetic two-spirals data and on two real data sets of MNIST and MIRFLICKR, where we see that our proposed methods, with the same set of hyperparameters, can correctly adjust the network complexity to the task complexity.
Early work on searching for the optimal network focused on incremental methods. Most such methods fix the depth to one and so the problem reduces to a search over the number of hidden units @cite_9 @cite_6 @cite_8 @cite_32 . Among these, there are those that add hidden units one by one; such as dynamic node creation @cite_9 , upstart algorithm @cite_10 , and feedforward neural network construction using cross-validation @cite_7 . A related method is cascade-correlation, where a new hidden unit that has the rest of the network as input, is trained while the rest of the network weights are fixed @cite_34 @cite_16 @cite_35 @cite_26 . This process technically increases the depth of the network for each addition of a unit. A recent related work applies a similar methodology to convolutional networks where a pre-trained network is widened or deepened on a new task to achieve better knowledge transfer @cite_11 .
@cite_35 @cite_26 @cite_7 @cite_8 @cite_9 @cite_32 @cite_6 @cite_16 @cite_34 @cite_10 @cite_11
2020663993 2788295695 2964055354 191220328
This article presents an algorithm that constructs feedforward neural networks with a single hidden layer for pattern classification. The algorithm starts with a small number of hidden units in the network and adds more hidden units as needed to improve the network's predictive accuracy. To determine when to stop adding new hidden units, the algorithm makes use of a subset of the available training samples for cross validation. New hidden units are added to the network only if they improve the classification accuracy of the network on the training samples and on the cross-validation samples. Extensive experimental results show that the algorithm is effective in obtaining networks with predictive accuracy rates that are better than those obtained by state-of-the-art decision tree methods. In this paper, we propose a novel approach for efficient training of deep neural networks in a bottom-up fashion using a layered structure. Our algorithm, which we refer to as deep cascade learning, is motivated by the cascade correlation approach of Fahlman and Lebiere, who introduced it in the context of perceptrons. We demonstrate our algorithm on networks of convolutional layers, though its applicability is more general. Such training of deep networks in a cascade directly circumvents the well-known vanishing gradient problem by ensuring that the output is always adjacent to the layer being trained. We present empirical evaluations comparing our deep cascade training with standard end–end training using back propagation of two convolutional neural network architectures on benchmark image classification tasks (CIFAR-10 and CIFAR-100). We then investigate the features learned by the approach and find that better, domain-specific, representations are learned in early layers when compared to what is learned in end–end training. This is partially attributable to the vanishing gradient problem that inhibits early layer filters to change significantly from their initial settings. While both networks perform similarly overall, recognition accuracy increases progressively with each added layer, with discriminative features learned in every stage of the network, whereas in end–end training, no such systematic feature representation was observed. We also show that such cascade training has significant computational and memory advantages over end–end training, and can be used as a pretraining algorithm to obtain a better performance. In this paper, we make two contributions to unsupervised domain adaptation (UDA) using the convolutional neural network (CNN). First, our approach transfers knowledge in all the convolutional layers through attention alignment. Most previous methods align high-level representations, e.g., activations of the fully connected (FC) layers. In these methods, however, the convolutional layers which underpin critical low-level domain knowledge cannot be updated directly towards reducing domain discrepancy. Specifically, we assume that the discriminative regions in an image are relatively invariant to image style changes. Based on this assumption, we propose an attention alignment scheme on all the target convolutional layers to uncover the knowledge shared by the source domain. Second, we estimate the posterior label distribution of the unlabeled data for target network training. Previous methods, which iteratively update the pseudo labels by the target network and refine the target network by the updated pseudo labels, are vulnerable to label estimation errors. Instead, our approach uses category distribution to calculate the cross-entropy loss for training, thereby ameliorating the error accumulation of the estimated labels. The two contributions allow our approach to outperform the state-of-the-art methods by +2.6 on the Office-31 dataset. The optimal selection of neural network architecture is a problem of significant theoretical and practical importance. The functional capacity of the network architecture determines the generalization properties and the computational complexity of the algorithm. For supervised training of feed-forward networks, current generalization theories tell us to minimize the functional capacity, i.e., the number of weights subject to the constraints imposed by the rule implicit in the training examples see e.g., (, 91). Several schemes have been proposed for pruning of the network architecture. The optimal brain damage method for network pruning proposed in (, 90) has been successful for pruning of classifiers as well as for pruning of regression networks for time series processing (, 93). The optimal brain surgeon (OBS) advanced in (Hassihi and Stork, 92) provides a refined estimate of the weight saliency, by incorporating the effects of quadratic retraining; however, at the expense of Irignificant additional computation. Likewise, growth algorithms are legio. Among these, Fahlman’s cascade correlation stands out for solving hard classification problems (Fahlman and Lebiere, 89). In the standard implementation cascade correlation adds hidden units that are fully connected to all input units and to all previously added hidden units, thereby rapidly increasing the number of weights in the network with the obvious danger of overfitting.