During transfer, the ResNet-50 is extracted from the visual backbone, and a fully connected linear layer is added as a decision head for supervised classification tasks. While we only report downstream classification results here, the linear decision head could be replaced with networks suited to other downstream tasks. We note that non-convolutional encoders, such as Vision Transformers[1]}, [2]}, could be incorporated into the visual backbone in the place of ResNet.
