As mentioned above, our EGE-CMP contains SMT, CMT and COMT three kinds of transformer structures.
EGE-CMP takes an image-text pair \((I, C)\)  as input where \(I\)  is the product image and \(C\)  is the caption data. Image \(I\)  are first processed by bottom-up-attention model [1]} to obtain the proposal features \(p=\lbrace p_1, p_2,..,p_m\rbrace \)  where objects are most likely to be detected and their corresponding position information \(q=\lbrace q_1,q_2,..,q_m\rbrace \) .
Caption \(C\)  are parsed by NER analysis tools to obtain the noun entities \(E_{C}=\lbrace E_1, E_2,...\rbrace \) .
Based on the above preparation, SMT extracts the embeddings \(v, t, e\)  for three different inputs (image, caption and entity).
Then CMT performs cross-modal fusion respectively on \(v\)  and \(t\)  by exchanging key-value pairs to generate the new representations \(h_v\)  and \(h_{t_1}\) .
Similarly, we utilize the same operation to produce the new representations \(h_e\)  and \(h_{t_2}\) .
To keep the representation consistent, we take the average of \(h_{t_1}\)  and \(h_{t_2}\)  to obtain the new representations of text \(h_t\) .
Finally, \(h_v\) , \(h_t\)  and \(h_e\) , \(h_t\)  are separately concatenated in the feature dimension and fed into two COMTs to produce joint image-text-entity representations \(f\)  for multi-product retrieval and identical-product retrieval tasks.
We give the detailed explanations of each transformer structure in below.
