The development of the vision-language pretraining model [1]}, [2]}, [3]}, [4]}, [5]}, [6]}, [7]}, [8]}, [9]}, [10]}, [11]}, [12]}, [13]}, [14]} has advanced tremendously, with joint representations often constructed using a multimodal fusion transformer network topology.
Existing pre-trained models for vision-language often learn image-text semantic alignment using a multi-layer Transformer architecture, such as Bert [15]}, on multi-modal input in a shared cross-modal space.
They may be loosely divided into two categories based on their network structural differences:
1) Single-stream models [16]}, [17]}, [18]}, [3]}, [1]}, [2]}, [7]} in a unified architecture encode the integrated multi-modal characteristics. 2) Two-stream models [23]}, [24]}, [4]}, [5]}, [6]}, [8]}, [14]}, [13]}, [12]}, on the other hand, use distinct encoders for inputs with varying modalities.
These approaches are not optimized for instance-level retrieval, and we demonstrate how they may be defective as a result of network architecture design flaws and incorrect pretext tasks.
Additionally, since real-world semantic information cannot be effectively learned during the network training phase, performance increases are restricted.
