
Structure-based approach: One reason for catastrophic forgetting is that a neural network's parameters are optimized for new tasks and no longer for previous ones. This suggests that not optimizing the entire network or expanding the internal model structure to deal with the new tasks while isolating old network parameters could attenuate catastrophic forgetting. PNN [1]} pioneered this approach by adding new components to the network and freezing old task parameters during training. Context-dependent gating (XdG) [2]} is a simple but popular approach to randomly assigning nodes to tasks. However, these approaches are limited to the task-incremental learning scenario by design, as task identity is required to select the correct task-specific components during training.

Regularization-based approach: When task knowledge is only available during training time, training a different part of the network for each task can still happen, but then the whole network is used through inference. Standard methods in this approach estimate the importance of the network parameters for the previously learned tasks and penalize future changes accordingly. Elastic Weight Consolidation (EWC) [3]} and its online counterpart (EWCo) [4]} adopt the Fisher information matrix to estimate the importance of the network synapses. Synaptic intelligence (SI) [5]} utilizes an accumulated gradient to quantify the significance of the network parameters.

Memory-based approach: This strategy replays old, or augmented, samples stored in memory when learning a new task. Learning without forgetting (LWF) [6]} uses a pseudo-data strategy where it labels the samples of the current tasks using the model trained on the previous tasks, resulting in training that mixes hard-target (likely category according to previous tasks) with soft-target (predicted probabilities within all classes). Gradient episodic memory (GEM) [7]} and Averaged GEM (A-GEM) [8]} successfully boost continual learning performance by the usage of exact samples stored in memory to estimate the forgetting case and to constraint the parameter updates accordingly. Gradient-based Sample Selection (GSS) [9]} focuses on optimizing the selection of samples to be replayed. Dark Experience Replay (DER/DER++) [10]} and Function Distance Regularization (FDR) [11]} use past samples and soft outputs to align past and current outputs. Hindsight Anchor Learning (HAL) [12]} adds additional objectives into replaying, aiming to reduce forgetting of key learned data points.
Alternatively, methods can also take advantage of generative models for pseudo-rehearsal. For example, Deep Generative Replay (DGR) [13]} utilizes a separated generative model sequentially trained on all tasks to generate samples from their data distribution. Additionally, knowledge distillation can be combined with DGR (DGR+distill) [14]} to pair generated samples with soft target knowledge.

