During the MTL pre-train stage, all the encoder backbones will be pre-trained on ImageNet and the decoder will be randomly initialized. In the M-ViT models, we use the pre-trained weights provided by DeiT [1]} to initialize all the transformer layers and the input linear projection layer in the encoder. In the MoE ViT models, we pre-train our encoder on ImageNet following the same strategy as its counterpart DeiT ViT encoder in [1]}.
