From \(E\)  we extract an image encoding of dimensionality \(h_I\)  representing the image. We then project that encoding to a \(e_L*k\)  sequence of soft prompts, which we hereafter refer to as image prompts. \(k\)  is determined by the architecture of the \(E\) . For example, for consistency with the MAGMA model, we use the 12x12x3072d feature map before pooling from CLIP, which we flatten to \(k=12*12=144\) . The encoders we experiment with are (1) CLIP RN50x16 [1]}, \(k=144\) , \(h_I=3072\) . Because CLIP is trained to learn multimodal image-text embeddings, we expect that it will be easier for the model to learn a projection into language space than a vision only encoder.
(2) NFRN50 [2]}, \(k=2\) , \(h_I=2048\) . We train three variants using NF-Resnet50: one pretrained and frozen during caption training (NFRN50), one tuned during caption training (NFRN50 Tuned; note that the LM is still frozen), and one randomly initialized (NFRN50 Random). The NFRN50 models are pretrained on an image classification task on data that is labeled according to WordNet hypo/hypernym structure. This signal trains the model to separate object classes according to these words. For this reason, we consider it to have indirect access to linguistic supervision.
(3) BEIT-Large [3]}, \(k=196\) , \(h_I=1024\) . BEIT is pretrained using a self-supervised masked visual token modeling task and does not have access to any labeled data which may give the model an inductive bias towards a linguistic structure. We use the 16-pixel patch version that was pretrained only on ImageNet22k.
