PIRL [1]} conducted experiments to show that learning to be invariant, instead of covariant, to augmentation set \(\mathcal {T}\)  typically would result in better representations, as illustrated in Fig REF  (b). In previous transformation-covariant networks, a common observation is that the representation in the final layer of the network is worse than the representation in the penultimate layer, while PIRL does not have this issue. It is reasonable to surmise that since the last layer representation would interact with the loss function directly, the loss function would discourage the last layer to contain general information (e.g. a loss function that predicts rotation angle would encourage the last layer to contain rotation-only information). Thus, invariant networks typically would learn a better representation than the covariant networks. Due to this observation, works later on typically choose to learn representation that is invariant with augmentations.
