The performance difference versus training stages caused by the aforementioned randomness between SCR and SCL motivates us to conduct further research on the impact of the random transformations upon the representation learning. As found by the work of SimCLR [1]}, adding some particular transformations (e.g. rotation or blur) into the transformation module would instead hurt the quality of learnt representations as these transformations cause more significant distortion to the input image (thus having the semantic shift). Following such an empirical observation, we propose the mechanism named Semantically Distinct Augmentation (SDA) which is applied on the input image \(x\)  before the transformation module \(\mathcal {H}\) .
The SDA consists of multiple deterministic augmentations, and every augmentation would cause a distinct semantics change of the input images. In results, if SDA are adopted during the representation learning, the feature space tends to have higher intrinsic dimension (which leads to more generalised features) for handling the diverse semantics produced by SDA.
Later in experiments, we demonstrate that using such an SDA mechanism benefits the online continual learning to learn more generalised representations, thus leading to superior performance.
