@inproceedings{
chitale2023task,
title={Task Arithmetic with Lo{RA} for Continual Learning},
author={Rajas Chitale and Ankit Vaidya and Aditya Kane and Archana Santosh Ghotkar},
booktitle={R0-FoMo:Robustness of Few-shot and Zero-shot Learning in Large Foundation Models},
year={2023},
url={https://openreview.net/forum?id=UcsXamgPtT}
}
Continual learning refers to the problem where the training data is available in sequential chunks, termed "tasks". The majority of progress in continual learning has been stunted by the problem of catastrophic forgetting, which is caused by sequential training of the model on streams of data. Moreover, it becomes computationally expensive to sequentially train large models multiple times. To mitigate both of these problems at once, we propose a novel method to continually train transformer-based vision models using low-rank adaptation and task arithmetic. Our method completely bypasses the problem of catastrophic forgetting, as well as reducing the computational requirement for training models on each task. When aided with a small memory of 10 samples per class, our method achieves performance close to full-set finetuning. We present rigorous ablations to support the prowess of our method.
Task Arithmetic with LoRA for Continual Learning
Rajas Chitale Ankit Vaidya Aditya Kane Archana Santosh Ghotkar