However, [1]} observed that some augmentations (e.g. rotation, noise) are too strong to deteriorate the representation quality if they involve in the view generation, and [2]} discovered that the semantic shift caused by these augmentations is too large to align the corresponding random views well.
Despite this, it does not imply that strong augmentations [3]}, [4]} cannot provide meaningful semantics for representation learning. For example, [4]} propose learning image representation by predicting rotation.
Hence, those augmentations can still be utilised for contrastive representation learning, but we should be aware of not encouraging the model to align the views with those strong augmentations which could cause the large semantic shift.
In turn, if the diverse views produced by the strong augmentation are treated as belonging to the other classes which are distinct from their original samples, additionally considering them in contrastive learning could help the model learn to extract rich features and represent the unseen classes better (according to the observation made by [6]} where the diverse views together are similar to the auxiliary dataset and the unseen classes are analogous to the out-of-distribution samples).
In this work, we base on such a line of thought and propose a technique named semantically distinct augmentation (SDA): given a mini-batch composed of the training samples, the strong transformations are first applied to these training samples to produce diverse views which are treated as from different/novel classes (cf. Figure REF ) and are added back to the mini-batch, then both contrastive learning and softmax classifier are applied to the extended mini-batch (with having both the original samples and their corresponding diverse views).
The two-pronged benefits are introduced by such SDA technique: First, learning on a diverse dataset allows the model to get a more generalisable representation and mitigates catastrophic forgetting; Second, as data comes in a stream for the OCL setting such that each data sample ideally can only be adopted once for training, our SDA leverages strong augmentations for the attempt on making best use of every sample.
