Opposite to our setup, a current trend is to pretrain models on vast amounts of bimodal data [1]}, [2]}, [3]}, and perform zero/few-shot evaluation. While remarkable results were achieved, performance is lower than finetuned models pretrained on less bimodal data, which is relatively cheap to obtain.
