TKIL: Tangent Kernel Approach for Class Balanced Incremental Learning
When learning new tasks in a sequential manner, deep neural networks tend to
forget tasks that they previously learned, a phenomenon called catastrophic
forgetting. Class incremental learning methods aim to address this problem by
keeping a memory of a few exemplars from previously learned tasks, and
distilling knowledge from them. However, existing methods struggle to balance
the performance across classes since they typically overfit the model to the
latest task. In our work, we propose to address these challenges with the
introduction of a novel methodology of Tangent Kernel for Incremental Learning
(TKIL) that achieves class-balanced performance. The approach preserves the
representations across classes and balances the accuracy for each class, and as
such achieves better overall accuracy and variance. TKIL approach is based on
Neural Tangent Kernel (NTK), which describes the convergence behavior of neural
networks as a kernel function in the limit of infinite width. In TKIL, the
gradients between feature layers are treated as the distance between the
representations of these layers and can be defined as Gradients Tangent Kernel
loss (GTK loss) such that it is minimized along with averaging weights. This
allows TKIL to automatically identify the task and to quickly adapt to it
during inference. Experiments on CIFAR-100 and ImageNet datasets with various
incremental learning settings show that these strategies allow TKIL to
outperform existing state-of-the-art methods.