The initial strategies to address catastrophic forgetting phenomena were based on distillation [1]}, in which the knowledge of the previously trained instance of the model is compressed and transferred to the new instance. The new instance then retains these representations by minimizing the distillation loss function explicitly (through the training examples which correspond to the previously added classes) [1]}. Apart from this, the new model instance is also penalized (during training) to learn the new classes (from the provided small-scale set of training examples) [3]}. Here, the work of Li et al. [4]} is notable where they proposed the learning without forgetting (LwF) scheme. LwF [4]} optimizes the distillation and cross-entropy loss functions in each training increment to enable the model to retain its prior knowledge while learning new representation simultaneously.
Moreover, Rebuffi et al. [6]} improved the LwF [4]} for the class-incremental learning tasks by indefinitely learning the feature representations related to the distilled and newly added categories via joint distillation and classification loss function optimization. Aljundi et al. [8]} introduced gating auto-encoders that can incrementally learn the feature representations for the task at hand, and based upon the nature of the test sample, the processing request is automatically forwarded to the relevant gate to perform the appropriate classification task. Castro et al. [9]} presented an end-to-end incremental learning scheme in which they proposed a cross-distilled loss function to retain prior learned knowledge while learning new classification tasks by penalizing the candidate network via cross-entropy loss function. Similarly, Roy et al. [10]} presented hierarchically-fashioned CNN architecture that is incrementally trained to perform various classification tasks with minimal training efforts. Tian et al. [11]} presented a contrastive learning strategy that outperformed knowledge distillation and other cutting-edge distillers for various knowledge transfer tasks, including single model compression, ensemble distillation, and cross-modal transfer. They proposed an objective function that minimizes the Kullback–Leibler (KL) divergence between a teacher and student network's outputs.
Mirzadeh et al. [12]} addressed the gap in teacher-student learning network through a multi-step knowledge distillation process.
Lee et al. [13]} introduced a global distillation strategy to reduce catastrophic forgetting on the massive unlabeled data. Lopez-Paz et al. [14]} proposed utilizing episodic memories in their framework, dubbed Gradient Episodic Memory (GEM), to resist catastrophic forgetting during incremental training. Chaudhry et al. [15]} proposed averaged GEM (A-GEM) that replaces the quadratic programming (in GEM [14]}) with dot products through which the gradient calculation is performed only once. This modification significantly reduces the computational resources which were required in the original GEM [14]} framework. Apart from this, Hegde et al. [18]} presented a compact and sparse student network that replicates the compressed representation of the teacher model to learn various classification via knowledge distillation incrementally.
