In Table REF , we first show the results using the off-the-shelf GrokNet [1]} Hash feature as the image embedding.
We feed the text tokens and image embedding directly into the multimodal transformer as [2]}, [3]}, i.e., the multimodal model is an early-fusion model without text encoder.
We apply MLM, MIM-kl, MIM-fr and ITM for the multimodal pre-training, each of which is same as in Sec.  REFWe cannot apply image-text contrastive learning or omni retrieval as there is no text encoder in the early-fusion architecture..
The mere difference lies in the masking strategy of MIM, where we randomly shuffle the 0/1 bits of the input Hash for masking.
For fine-tuning, we focused on the multimodal categorization tasks of CC and MPC.
We observe that even with only 4 pre-training tasks, there is a significant improvement on CC and MPC tasks from L1 over L2, indicating that our pre-training also works well with a fixed vision encoder of GrokNet.
We launched L2 as our current production model.
More details of the deployment and product impact are provided in Sec. REF .
We then show the advantage of end-to-end training.
Comparing L2 and L3, we observe the end-to-end trained model with ResNet50 already outperforms GrokNet's Hash [1]}, which is from a ResNeXt101 model.
