We directly utilize the bottom-up-attention model pre-trained [1]} on the VG dataset [2]} to extract proposal-wise features for the visual data and employ the BERT [3]} model to initialize the linguistic transformer of our EGE-CMP.
We limit the number of input areas to between 10 and 36 by picking regions with predicted confidence greater than a predefined threshold as in [4]}.
The region's features are then flattened and input into EGE-CMP using RoIAlign.
The number of the Text/Visual/Entity Transformer, Text/Visual and Text/entity Cross-Transformer, and Co-Transformer is set to \(L=4, K=4,\)  and \(H=4\) , respectively, for a total of 12 transformer layers.
We set the hidden state size of EGE-CMP and other baselines to 768 for a fair comparison.
The transformer blocks in EGE-CMP have hidden state size of 768 with 8 attention heads.
Meanwhile, we separately attach a 512 dimensional fully connected layer after Co-Transformer and Text/Visual/Entity Transformer for task-agnostic masked learning and cross-modal contrastive learning.
By concatenating the features from the final Co-transformer layer, a 1024 dimensional feature vector is obtained for retrieval, which is also the same for other baselines.
The maximum sequence length for the sentence is set to 36.
Our EGE-CMP model is trained with a total batch size of 128 for 10 epochs on 8 RTX 3090 GPUs.
And an Adam [5]} optimizer is used with an initial learning rate of 1e-4 and a linear learning rate decay schedule is adopted in the EGE-CMP training process.
Temperature parameter \(\tau \)  is set to 0.07.
The entity numbers are limited to 12,000 for faster training speed.
EGE-CMP generates instance features from texts and proposal-wise features during inference.
Before retrieval, the instance features of the image/text are all normalized and then concatenated as the overall representations.
During retrieval, we compute cosine similarities between each box query with single-product samples using the overall representations mentioned above and score all retrieval single-product results based on their similarity.
The top-\(N\)  samples are returned as the retrieval results.
Fig REF  shows more retrieval results by EGE-CMP.
To provide a fair comparison with other baselines, we conduct all experiments using the same training settings and assessment procedures.
<TABLE><TABLE>