Implementation details. We set the dimension \(d\)  in all transformer layers as 768. We adopt the first three layers of BERT [1]} as the text encoder and a four layers transformer decoder [2]} for multi-modal feature fusion, which makes the number of parameters comparable to SAT [3]}. The text encoder is initialized from the first three layers of \(\text{BERT}_{\text{BASE}}\)  [2]}. The fusion decoder is trained from scratch. The number of views \(N\)  is set to 4.
Model optimization is conducted using Adam [5]} optimizer with a batch size of 24. We set an initial learning rate of \(0.0005\)  for the model, and the learning rate of the transformer layer is further multiplied by \(0.1\) . We reduce the learning rate by a multiplicative factor of \(0.65\)  every 10 epochs after 40 epochs for a total of 100 epochs.
