We use three different image encoders with increasing levels of linguistic supervision in pretraining: BEIT [1]}, Normalizer Free Resnet50 (NFRN50) [2]}, and CLIP [3]} to train different projections into the LM. By linguistic supervision, we refer to the extent to which the image encoder was exposed to language data during its pretraining, thus influencing the expected representational similarity between it and an LM. While CLIP was pretrained to align images with full natural language captions in a shared image-text representation space, BEIT had no exposure to language and was trained by predicting the contents of masked out sections of images. NFRN50 falls somewhere between these extremes: having been pretrained on an image classification task for identifying the subject of an image over the set of classes in ImageNet1k [4]}. Although there is no natural language in this task, the pretraining objective encourages the model to map visual features along lexical categorical concepts (the image classes) derived from the WordNet hierarchy [5]}.
