Conventional deep learning classifiers are static in the sense that they are trained on
 a predefined set of classes and learning to classify a novel class typically requires
 re-training. In this work, we address the problem of Low-shot network-expansion
 learning. We introduce a learning framework which enables expanding a pre-trained
 (base) deep network to classify novel classes when the number of examples for the
 novel classes is particularly small. We present a simple yet powerful distillation
 method where the base network is augmented with additional weights to classify
 the novel classes, while keeping the weights of the base network unchanged. We
 term this learning hard distillation, since we preserve the response of the network
 on the old classes to be equal in both the base and the expanded network. We
 show that since only a small number of weights needs to be trained, the hard
 distillation excels for low-shot training scenarios. Furthermore, hard distillation
 avoids detriment to classification performance on the base classes. Finally, we
 show that low-shot network expansion can be done with a very small memory
 footprint by using a compact generative model of the base classes training data
 with only a negligible degradation relative to learning with the full training set. In many real life scenarios, a fast and simple classifier expansion is required to extend the set of classes that a deep network can classify. For example, consider a cleaning robot trained to recognize a number of objects in a certain environment. If the environment is modified with an additional novel object, it is desired to be able to update the classifier by taking only a few images of that object and expand the robot classifier. In such a scenario, the update should be a simple procedure, based on a small collection of images captured in a non-controlled setting. Furthermore, such a low-shot network update should be fast and without access the entire training set of previously learned data. A common solution to classifier expansion is fine-tuning the network BID6 . However fine-tuning requires keeping a large amount of base training data in memory, in addition to collecting sufficient examples of the novel classes. Otherwise, fine-tuning can lead to degradation of the network accuracy on the base classes, also known as catastrophic forgetting BID0 . In striking contrast, for some tasks, humans are capable of instantly learning novel categories. Using one or only a few training examples humans are able to learn a novel class, without compromising previously learned abilities or having access to training examples from all previously learned classes.We consider the classifier expansion problem under the following constraints:1. Low-shot: very few samples of the novel classes are available. 2. No forgetting: preserving classification performance on the base classes. 3. Small memory footprint: no access to the base classes training data.In this work we introduce a low-shot network expansion technique, augmenting the capability of an existing (base) network trained on base classes by training additional parameters that enables to classify novel classes. The expansion of the base network with additional parameters is performed in the last layers of the network.To satisfy low-shot along with no-forgetting constraints, we present a hard distillation framework. Distillation in neural networks BID5 is a process for training a target network to imitate another network. A loss function is added to the target network so that its output matches the output of the mimicked network. In standard soft distillation the trained network is allowed to deviate from the mimicked network. Whereas hard distillation enforces that the output of the trained network for base classes matches the output of the mimicked network as a hard constraint. We achieve hard distillation by keeping the weights of the base network intact, and learn only the newly added weights. Network expansion with hard distillation yields a larger network, distilling the knowledge of the base network in addition to augmented capacity to classify novel classes. We show that in the case of low-shot (only 1-15 examples of a novel class), hard distillation outperforms soft distillation. Moreover, since the number of additional parameters in the expanded network is small, the inference time of the new network is nearly identical to the base network.To maintain a small memory footprint, we refrain from saving the entire training set. Instead, we present a compact generative model, consisting of a collection of generative models fitted in the feature space to each of the base classes. We use a Gaussian Mixture Model (GMM) with small number of mixtures, and show it inflicts a minimal degradation in classification accuracy. Sampling from the generative GMM model is fast, reducing the low-shot training time and allowing fast expansion of the network.We define a benchmark for low-shot network expansion. The benchmark is composed of a series of tests of increasing complexity, ranging from simple tasks where base and novel classes are from different domains and to difficult tasks where base and novel classes are from the same domain and shares objective visual similarities. We perform a comprehensive set of experiments on this challenging benchmark, comparing the performance of the proposed to alternative methods.To summarize, the main contributions of the paper are:1. A novel hard-distillation solution to a low-shot classifier expansion problem 2. GMM as a sufficient generative model to represent base classes in a feature space 3. A new benchmark for the low-shot classifier expansion problem 2 RELATED WORKS A common solution to the class-incremental learning problem is to use a Nearest-Neighbors (NN) based classifier in feature space. A significant advantage of a NN-based classifier is that it can be easily extended to classify a novel class, even when only a single example of the class is available (one-shot learning). However NN-based classifiers require keeping in the memory significant amount of training data from the base classes. BID7 proposed to use Nearest Class Mean (NCM) classifier, where each class is represented by a single prototype example which is the mean feature vector of all class examples. One major disadvantage of NCM and NN-based methods is that they are based on a fixed feature representation of the data. To overcome this problem BID7 proposed to learn a new distance function in the feature space using metric learning.The ideas of metric learning combined with the NN classifier resonate with recent work by on Matching Networks for one-shot learning, where both feature representation and the distance function are learned end-to-end with attention and memory augmented networks. The problem we consider in this paper is different from the one discussed by . We aim to expand existing deep classifier trained on large dataset to classify novel classes, rather than to create a general mechanism for one-shot learning. BID3 presented an innovative low-shot learning mechanism, where they proposed a Squared Gradient Magnitude regularization technique for an improved fixed feature representation learning designed for low-shot scenarios. They also introduced techniques to hallucinate additional training examples for novel data classes. In contrast, we present a method which aims to maximize performance in low-shot network expansion given a fixed representation, allowing expanding the representation based on novel low-shot data. Furthermore, in our work, we demonstrate the ability to expand the network without storing the entire base classes training data.Recently, BID9 proposed iCaRL -(Incremental Classifier and Representation Learning), to solve the class-incremental learning problem. iCaRL is based on Nearest-Mean-of-Exemplars classifier, similar to the NCM classifier of BID7 . In the iCaRL method, the feature representation is updated and the class means are recomputed from a small stored number of representative examples of the base classes. During the feature representation update, the network parameters are updated by minimizing a combined classification and distillation loss. The iCaRL method was introduced as a class-incremental learning method for large training sets. In Section 4 we discuss its adaptation to low-shot network expansion and compare it to our method. BID11 proposed the Progressive Network for adding new tasks without affecting the performance of old tasks. They propose freezing the parameters that were trained on old tasks and expand the network with a additional layers when training a new task. BID15 proposed the Progressive learning technique which solves the problem of online sequential learning in extreme learning machines paradigm (OS-ELM). The purpose of their work is to incrementally learn the last fully-connected layer of the network. When a sample from a novel class arrives, the last layer is expanded with additional parameters. The Progressive learning solution updates the last layer only sequentially and only works in the ELM framework (does not update internal layers of the network). In another work BID14 proposed an incremental learning technique which augments the base network with additional parameters in last fully connected layer to classify novel classes. Similar to iCaRL, they perform soft distillation by learning all parameters of the network. Instead of keeping historical training data, they propose phantom sampling -hallucinating data from past distribution modeled with Generative Adversarial Networks.In this work we propose a solution that borrows ideas from freeze-and-expand paradigm, improved feature representation learning, network distillation and modeling past data with a generative model. We propose to apply expansion to the last fully connected layer of a base network to enable classification on novel classes, and to deeper layers to extend and improve the feature representation. However, in contrast to other methods BID9 ; BID15 , we do not retrain the base network parameters, but only the newly introduced weights of the expansion.Moreover, the extended feature representation is learned from samples of base and novel classes. In contrast to BID3 , where the improved feature representation is learned from simulating low-shot scenarios on the base classes only, before the actual novel data is available. Finally, in order to avoid keeping all historical training data, we use Gaussian Mixture Model of the feature space as a generative model for base classes.
 In this paper, we address the problem of Low-shot network-expansion learning
{"src": "scitldr-AIC_train", "paper_id": "SJw03ceRW"}