We then compare the effectiveness of vision encoder.
With the same input image size 224x224 and the same transformer architecture, we found ViT-B/16 [1]} (L6) brings a consistent gain over ResNet50 (L5) on each of the 7 downstream tasks, showing a better visual encoder is beneficial to the multimodal applications.
