In recent years, Convolutional Neural Networks (CNNs) have shown remarkable performance in many computer vision tasks such as object recognition and detection. However, complex training issues, such as catastrophic forgetting' and hyper-parameter tuning, make incremental learning in CNNs a difficult challenge. In this paper, we propose a hierarchical deep neural network, with CNNs at multiple levels, and a corresponding training method for incremental learning. The network grows in a tree-like manner to accommodate the new classes of data without losing the ability to identify the previously trained classes. The proposed network was tested on CIFAR-100 and reported 60.46 accuracy and 20 reduction in training effort as compared to retraining final layers of a deep network. The network organizes the incoming classes of data into feature-driven super-classes and improves upon existing hierarchical CNN models by adding the capability of self-growth.
In this work, the starts out as a single root node and generates new hierarchies to accommodate the new classes. Images belonging to the older dataset are required during retraining, but by localizing the change to a small section of the whole network, our method tries to reduce the training effort and complexity. In @cite_28 , a similar approach is applied, where the new classes are added to the old classes, and divided into two super-classes, by using an error-based model. The initial network is cloned to form two new networks which are fine tuned over the two new super-classes. While their motivation was a "divide-and-conquer" approach for large datasets, this work tries to incrementally grow with new data. And, we sequentially add new data over multiple learning stages. In the next section, we lay out in detail our design principle, network topology and the algorithm used to grow the network.
@cite_28
2160684493
Supervised learning using deep convolutional neural network has shown its promise in large-scale image classification task. As a building block, it is now well positioned to be part of a larger system that tackles real-life multimedia tasks. An unresolved issue is that such model is trained on a static snapshot of data. Instead, this paper positions the training as a continuous learning process as new classes of data arrive. A system with such capability is useful in practical scenarios, as it gradually expands its capacity to predict increasing number of new classes. It is also our attempt to address the more fundamental issue: a good learning system must deal with new knowledge that it is exposed to, much as how human do. We developed a training algorithm that grows a network not only incrementally but also hierarchically. Classes are grouped according to similarities, and self-organized into levels. The newly added capacities are divided into component models that predict coarse-grained superclasses and those return final prediction within a superclass. Importantly, all models are cloned from existing ones and can be trained in parallel. These models inherit features from existing ones and thus further speed up the learning. Our experiment points out advantages of this approach, and also yields a few important open questions.