Effects of Auxiliary Knowledge on Continual Learning
In Continual Learning (CL), a neural network is trained on a stream of data
whose distribution changes over time. In this context, the main problem is how
to learn new information without forgetting old knowledge (i.e., Catastrophic
Forgetting). Most existing CL approaches focus on finding solutions to preserve
acquired knowledge, so working on the past of the model. However, we argue that
as the model has to continually learn new tasks, it is also important to put
focus on the present knowledge that could improve following tasks learning. In
this paper we propose a new, simple, CL algorithm that focuses on solving the
current task in a way that might facilitate the learning of the next ones. More
specifically, our approach combines the main data stream with a secondary,
diverse and uncorrelated stream, from which the network can draw auxiliary
knowledge. This helps the model from different perspectives, since auxiliary
data may contain useful features for the current and the next tasks and
incoming task classes can be mapped onto auxiliary classes. Furthermore, the
addition of data to the current task is implicitly making the classifier more
robust as we are forcing the extraction of more discriminative features. Our
method can outperform existing state-of-the-art models on the most common CL
Image Classification benchmarks.