Our approach takes inspiration from recent work in adapting pretrained language models for accepting representations of images as inputs. Particularly, the Frozen and MAGMA models [1]}, [2]}, as well as [3]}, [4]}, [5]}, [6]}, which show that pretrained image and text networks can be tuned together on an image captioning task and applied to downstream vision-language (VL) tasks. These approaches either fine-tune the pretrained models, or train non-linear MLP projection/fusion networks between modalities, making interpretation of the representations difficult compared to our approach. Pretrained/from scratch LMs have typically been used in the past for image captioning applications in which an image representation is fed into the LM as input [7]}, [8]}, [9]}. [10]}, [11]} pretrain vision-language models from scratch using image-caption data. The success of aforementioned models on VL tasks indicates there is a representational similarity learned by text and image models independently, which we investigate in this paper.
