One way to evaluate the generalisability of a learnt representation is to testify its performance on the out-of-distribution (OOD) detection task (i.e. how likely we are able to detect the unknown samples from novel classes based on the feature representation being used). The basic motivation behind such experiments is that: Since the continual learning agent aims to sequentially extend its recognition ability on the unseen classes (i.e. unseen classes with respect to the current model, which can also be treated as out-of-distribution samples), the generalisable feature representation, which ideally would make the unseen classes more separable from the seen classes thus requires less adjustment while learning the unseen/new classes, hence is able to alleviate the catastrophic forgetting phenomenon in continual learning as the feature extractor of the classification model only needs to be minorly updated.
To this end, we take the training set of CIFAR-10 as inliers and treat the other datasets (e.g. MNIST [1]}, Omniglot [2]}, and SVHN [3]}) as outliers, ideally the test samples from the test set of CIFAR-10 should be closer to the inliers than the samples from the outlier dataset (where such distance is computed via the Mahalanobis distance from the test sample to the nearest centre of inlier classes, averaged over various augmentations of the test sample). The performance of OOD detection is then evaluated by the area under ROC curve (AUC) based on the aforementioned distance, and the results are shown in Table REF . We can see that the representation learnt by our proposed SDAF provides a better OOD detection performance than SCL and SCR thus being more generalised (with following a clear logic behind: improvement in the contrastive learning leads to better generalizability as shown by better OOD detection performance, and the better generalizability leads to better alleviation of the catastrophic forgetting thus boosting the continual learning performance).
<TABLE>