Multi-label classification is an approach which allows a datapoint to be labelled with more than one class at the same time. A common but trivial approach is to train individual binary classifiers per label, but the performance can be improved by considering associations within the labels. Like with any machine learning algorithm, hyperparameter tuning is important to train a good multi-label classifier model. The task of selecting the best hyperparameter settings for an algorithm is an optimisation problem. Very limited work has been done on automatic hyperparameter tuning and AutoML in the multi-label domain. This paper attempts to fill this gap by proposing a neural network algorithm, CascadeML, to train multi-label neural network based on cascade neural networks. This method requires minimal or no hyperparameter tuning and also considers pairwise label associations. The cascade algorithm grows the network architecture incrementally in a two phase process as it learns the weights using adaptive first order gradient algorithm, therefore omitting the requirement of preselecting the number of hidden layers, nodes and the learning rate. The method was tested on 10 multi-label datasets and compared with other multi-label classification algorithms. Results show that CascadeML performs very well without hyperparameter tuning.
The cascade correlation neural network approach @cite_10 was an early AutoML method. In cascade correlation neural networks training starts with a simple perceptron network, which is grown incrementally by adding new cascaded layers with skip-level connections as long as performance on a validation dataset improves. Since the proposal of the original cascade correlation algorithm in @cite_10 , various improvements that follow a similar overall process to the original method have been proposed, for example in @cite_16 @cite_18 @cite_14 @cite_36 , as well as Cascade2 @cite_8 . Active research in this field, however, is fairly limited. As it is the basis for CascadeML, the Cascade2 algorithm is described in detail in the next section.
@cite_18 @cite_14 @cite_8 @cite_36 @cite_16 @cite_10
191220328 1543146175 2133640977 2099790329 1527646298 2109779438
The optimal selection of neural network architecture is a problem of significant theoretical and practical importance. The functional capacity of the network architecture determines the generalization properties and the computational complexity of the algorithm. For supervised training of feed-forward networks, current generalization theories tell us to minimize the functional capacity, i.e., the number of weights subject to the constraints imposed by the rule implicit in the training examples see e.g., (, 91). Several schemes have been proposed for pruning of the network architecture. The optimal brain damage method for network pruning proposed in (, 90) has been successful for pruning of classifiers as well as for pruning of regression networks for time series processing (, 93). The optimal brain surgeon (OBS) advanced in (Hassihi and Stork, 92) provides a refined estimate of the weight saliency, by incorporating the effects of quadratic retraining; however, at the expense of Irignificant additional computation. Likewise, growth algorithms are legio. Among these, Fahlman’s cascade correlation stands out for solving hard classification problems (Fahlman and Lebiere, 89). In the standard implementation cascade correlation adds hidden units that are fully connected to all input units and to all previously added hidden units, thereby rapidly increasing the number of weights in the network with the obvious danger of overfitting. Abstract Six learning algorithms are investigated and compared empirically. All of them are based on variants of the candidate training idea of the Cascade Correlation method. The comparison was performed using 42 different datasets from the PROBEN1 benchmark collection. The results indicate: (1) for these problems it is slightly better not to cascade the hidden units; (2) error minimization candidate training is better than covariance maximization for regression problems but may be a little worse for classification problems; (3) for most learning tasks, considering validation set errors during the selection of the best candidate will not lead to improved networks, but for a few tasks it will. © 1997 Elsevier Science Ltd. The cascade correlation is a very flexible, efficient and fast algorithm for supervised learning. It incrementally builds the network by adding hidden units one at a time, until the desired input output mapping is achieved. It connects all the previously installed units to the new unit being added. Consequently, each new unit in effect adds a new layer and the fan-in of the hidden and output units keeps on increasing as more units get added. The resulting structure could be hard to implement in VLSI, because the connections are irregular and the fan-in is unbounded. Moreover, the depth or the propagation delay through the resulting network is directly proportional to the number of units and can be excessive. We have modified the algorithm to generate networks with restricted fan-in and small depth (propagation delay) by controlling the connectivity. Our results reveal that there is a tradeoff between connectivity and other performance attributes like depth, total number of independent parameters, and learning time. > Abstract : The Cascade-Correlation learning algorithm constructs a multi-layer artificial neural network as it learns to perform a given task. The resulting network's size and topology are chosen specifically for this task. In the resulting 'cascade' networks, each new hidden unit receives incoming connections from all input and pre-existing hidden units. In effect, each new unit adds a new layer to the network. This allows Cascade-Correlation to create complex feature detectors, but it typically results in a network that is deeper, in terms of the longest path from input to output, than is necessary to solve the problem efficiently. In this paper we investigate a simple variation of Cascade-Correlation that will build deep nets if necessary, but that is biased toward minimizing network depth. We demonstrate empirically, across a range of problems, that this simple technique can reduce network depth, often dramatically. However, we show that this technique does not, in general, reduce the total number of weights or improve the generalization ability of the resulting networks. Cascade-Correlation is a new architecture and supervised learning algorithm for artificial neural networks. Instead of just adjusting the weights in a network of fixed topology. Cascade-Correlation begins with a minimal network, then automatically trains and adds new hidden units one by one, creating a multi-layer structure. Once a new hidden unit has been added to the network, its input-side weights are frozen. This unit then becomes a permanent feature-detector in the network, available for producing outputs or for creating other, more complex feature detectors. The Cascade-Correlation architecture has several advantages over existing algorithms: it learns very quickly, the network determines its own size and topology, it retains the structures it has built even if the training set changes, and it requires no back-propagation of error signals through the connections of the network.