Continual Learning with Transformers for Image Classification
In many real-world scenarios, data to train machine learning models become
available over time. However, neural network models struggle to continually
learn new concepts without forgetting what has been learnt in the past. This
phenomenon is known as catastrophic forgetting and it is often difficult to
prevent due to practical constraints, such as the amount of data that can be
stored or the limited computation sources that can be used. Moreover, training
large neural networks, such as Transformers, from scratch is very costly and
requires a vast amount of training data, which might not be available in the
application domain of interest. A recent trend indicates that dynamic
architectures based on an expansion of the parameters can reduce catastrophic
forgetting efficiently in continual learning, but this needs complex tuning to
balance the growing number of parameters and barely share any information
across tasks. As a result, they struggle to scale to a large number of tasks
without significant overhead. In this paper, we validate in the computer vision
domain a recent solution called Adaptive Distillation of Adapters (ADA), which
is developed to perform continual learning using pre-trained Transformers and
Adapters on text classification tasks. We empirically demonstrate on different
classification tasks that this method maintains a good predictive performance
without retraining the model or increasing the number of model parameters over
the time. Besides it is significantly faster at inference time compared to the
state-of-the-art methods.