Most works in self-supervised learning evaluate the representations learned at each layer of the network by training a linear classifier, e.g., ImageNet classification. Usually, the fully supervised model performs better than the self-supervised model, but the gap shrinks as we go to the earlier layers. This is expected as the late layers are specifically tuned for the pseudo-task rather than general visual recognition. For instance in Table 3 of [1]}, the first layer of the self-supervised model performs even slightly better than the fully supervised model on the linear classification task.
