Despite advances in deep learning, artificial neural networks do not learn the same way as humans do. Today, neural networks can learn multiple tasks when trained on them jointly, but cannot maintain performance on learnt tasks when tasks are presented one at a time -- this phenomenon called catastrophic forgetting is a fundamental challenge to overcome before neural networks can learn continually from incoming data. In this work, we derive inspiration from human memory to develop an architecture capable of learning continuously from sequentially incoming tasks, while averting catastrophic forgetting. Specifically, our model consists of a dual memory architecture to emulate the complementary learning systems (hippocampus and the neocortex) in the human brain and maintains a consolidated long-term memory via generative replay of past experiences. We (i) substantiate our claim that replay should be generative, (ii) show the benefits of generative replay and dual memory via experiments, and (iii) demonstrate improved performance retention even for small models with low capacity. Our architecture displays many important characteristics of the human memory and provides insights on the connection between sleep and learning in humans. Many machine learning models, when trained sequentially on tasks, forget how to perform the previously learnt tasks. This phenomenon called catastrophic forgetting is prominent in neural networks BID23 . Without a way to avert catastrophic forgetting, a learning system needs to store all training data and relearn on it along with new incoming data, when retraining. Hence, it is an important challenge to overcome in order to enable systems to learn continuously. BID23 first suggested that the underlying cause of forgetting was the distributed shared representation of tasks via network weights. Subsequent works attempted to remedy the issue by reducing representational overlap between input representations via activation sharpening algorithms BID17 , orthogonal recoding of inputs BID19 or orthogonal activations at all hidden layers BID24 BID5 . More recent works have explored activations like dropout BID9 and local winner-takes-all BID36 to create sparse, less correlated feature representations. But such sparse encodings can be task specific at times and in general act as heuristics to mildly pacify the underlying problem.Further, natural cognitive systems are also connectionist in nature and yet they forget gradually but not 'catastrophically'. For instance, humans demonstrate gradual systematic forgetting. Frequently and recently encountered tasks tend to survive much longer in the human memory, while those rarely encountered are slowly forgotten. Some of the earlier tasks may be seen again, but it is not necessary for them to be retained in memory BID7 . Hence only sparsifying representations does not solve the problem. Instead, neuroscientific evidence suggests that humans have evolved mechanisms to separately learn new incoming tasks and consolidate the learning with previous knowledge to avert catastrophic forgetting BID22 BID29 BID7 .Complementary learning systems: BID22 suggested that this separation has been achieved in the human brain via evolution of two separate areas of the brain, the hippocampus and the neocortex. The neocortex is a long term memory which specializes in consolidating new information with previous knowledge and gradually learns the joint structure of all tasks and experiences; whereas the hippocampus acts as a temporary memory to rapidly learn new tasks and then slowly transfer the knowledge to neocortex after acquisition.Experience replay: Another factor deemed essential for sequential learning is experience replay. BID22 ; BID29 have emphasized the importance of replayed data patterns in the human brain during sleep and waking rest. BID31 BID32 proposed several replay techniques (a.k.a. pseudopattern rehearsal) to achieve replay, but they involved generating replay data without storing input representations and our experiments show that they lack the accuracy required for consolidation.Weight consolidation or freezing: Recent evidence from neuroscience also suggests that mammalian brain protects knowledge in the neocortex via task-specific consolidation of neural synapses over long periods of time BID37 BID0 . Such techniques have recently been employed in progressive neural networks BID34 and Pathnets BID4 both of which freeze neural network weights after learning tasks. BID16 have used the fisher information matrix (FIM) to slow down learning on network weights which correlate with previously acquired knowledge.In this paper, we address the catastrophic forgetting problem by drawing inspiration from the above neuroscientific insights and present a method to overcome catastrophic forgetting. More specifically, we propose a dual-memory architecture for learning tasks sequentially while averting catastrophic forgetting. Our model comprises of two generative models: a short-term memory (STM) to emulate the human hippocampal system and a long term memory (LTM) to emulate the neocortical learning system. The STM learns new tasks without interfering with previously learnt tasks in the LTM. The LTM stores all previously learnt tasks and aids the STM in learning tasks similar to previous tasks. During sleep/down-time , the STM generates and transfers samples of learnt tasks to the LTM. These are gradually consolidated with the LTM's knowledge base of previous tasks via generative replay.Our approach is inspired from the strengths of deep generative models, experience replay and the complementary learning systems literature. We demonstrate our method's effectiveness in averting catastrophic forgetting by sequentially learning multiple tasks. Moreover, our experiments shed light on some characteristics of human memory as observed in the psychology and neuroscience literature. In this section we show that DGDMN shares some more remarkable characteristics with the human memory and present a discussion of some more related ideas. Due to space constraints, visualizations of the learnt latent structures when training jointly vs. sequentially have been deferred to appendix A. The hyperparameters of DGDMN (κ and n ST M ) have intuitive interpretations and we have provided simple heuristics to choose them without any complex searches (in appendix B).Resilience to noise and occlusion: We use a VAE to be able to reconstruct representations of samples. Reconstructed images are less noisy and can recover from partial occlusion, which gives our model human-like abilities to recognize objects in noisy, distorted or occluded images. We test our LTM model and a NN model by jointly training on uncorrupted Digits data and testing on noisy and occluded images. We see that the LTM is more robust to noisy and occluded images and exhibits smoother degradation in classification accuracy because of its denoising reconstructive properties (see FIG7 ). The choice of underlying generative model: Our consolidation ability and retention performance relies heavily on the generation and reconstruction ability of the underlying generative model. We chose a VAE for its reconstructive capabilities but our architecture is agnostic to the choice of the underlying generative model as long as the generator can generate reliable samples and reconstruct incoming samples accurately. Hence, variants of Generative Adversarial Networks (GAN) Goodfellow et al. (2014) like BiGANs BID2 , ALI (Dumoulin et al., 2017) and AVB BID25 can also be used for the generative model depending on the modeled domain.Why use short-term memory?: Our LTM always learns from STTMs and never from real data, and the STTMs' errors slowly propagate into the LTM and contribute to forgetting. An alternative could be to directly store data from new incoming tasks, consolidate it into the LTM after periodic intervals, and then discard the data. We show the accuracy curves on Digits dataset for this approach in FIG8 . This results in higher retention compared to DGDMN in FIG3 because LTM now learns from real data. However, this approach is not truly online since recently learnt tasks cannot be used immediately until after a sleep phase. Since the STM's error can be made smaller by using high capacity generators and classifiers, we suggest using a STM for true online continual learning.Connections to knowledge distillation: Previous works on (joint) multitask learning have also proposed approaches to learn individual tasks with small networks and then "distilling" them jointly into a larger neural network . Such distillation can sometimes improve performance on individual tasks if they share structure and at other times mitigate inter-task interference due to refinement of learnt functions while distilling BID30 . Though we do not use temperature-controlled soft-labels while consolidating tasks into the LTM (unlike distillation), we surmise that due to refinement and compression during consolidation phase, DGDMN is also able to learn joint task structure effectively while mitigating interference between tasks.Approaches based on synaptic consolidation: Though our architecture draws inspiration from complementary learning systems and experience replay in the human brain, there is also considerable neuroscientific evidence for synaptic consolidation in the human brain (like in EWC). It might be interesting to explore how synaptic consolidation can be incorporated in our dual memory architecture without causing stagnation and we leave this to future work. We also plan to extend our architecture to learning optimal policies over time via reinforcement learning without explicit replay memories. In this work, we have developed a model capable of learning continuously on sequentially incoming tasks, while averting catastrophic forgetting. Our model employs a dual memory architecture to emulate the complementary learning systems (hippocampus and the neocortex) in the human brain and maintains a consolidated long-term memory via generative replay of past experiences. We have shown that generative replay performs the best for long-term performance retention even for neural networks with small capacity, while demonstrating the benefits of using generative replay and a dual memory architecture via our experiments. Our model hyperparameters have simple interpretations and can be set without much tuning. Moreover, our architecture displays remarkable parallels with the human memory system and provides useful insights about the connection between sleep and learning in humans. Deep Generative Replay (algorithm 1), as described in section 3.1, consolidates new tasks for a DGM with previously learnt tasks. It first computes sampling fractions for new tasks (η tasks ) and previously learnt tasks (η gen ) and ensures a minimum fraction (κ) per new task (lines 3-6). Then it computes the number of samples to generate from previous tasks and whether to subsample the incoming task samples to satisfy the memory capacity N max (lines 7-12). Finally, it generates the required number of samples from previous tasks, reconstructs all data and trains the DGM on resulting data (lines 13-16). For a dictionary D, D is the total number of tasks in D counting repetitions, while |D| is the total number of tasks without repetitions. |X| is the number of samples in set X. BID35 have recently proposed a similar idea independently and BID27 have also employed a generative replay in two-layer restricted boltzmann machines, but they do not describe balancing new and generated samples and cannot recognize repeated tasks (section 4.2). Their generative replay without a dual memory architecture is costly to train (section 4.3) and a lack of reconstruction for new samples makes their representations less robust to noise and occlusions (section 5).
A dual memory architecture inspired from human brain to learn sequentially incoming tasks, while averting catastrophic forgetting.
{"src": "scitldr-AIC_train", "paper_id": "BkVsWbbAW"}