For the MTL encoder, we evaluate our model based on several variants of ViT following DeiT [1]}, including ViT-tiny, ViT-small, and ViT-base. The final ViT block's output feature will be fed into decoders for multi-task predictions. We embed MoE expert layers once in every two ViT blocks. The router is a single-layer MLP which maps token embedding to experts' selection probability. In task-conditioned MoE ViT, the task embedding network \(\mathcal {T}\)  is a two-layer MLP of dimensions 64 and 64. As for MLP decoder, the previous SoTA works  [2]}, [3]}, [4]} uses Deeplab [5]} as the decoder for a ResNet backbone. However, Deeplab is defined for Conv backbone and not suitable for ViT encoder output. Therefore, we follow the prior work [6]} and use a PUP  [6]} as decoder, which is a progressive upsampling strategy that alternates conv layers and upsampling operations. Each decoder contains five conv layers (the first four of dimension 256 and the final one of dimension corresponding to task prediction) and four upsampling layers. This decoder is of lighter weight and consumes fewer FLOPs than Deeplab. The output feature of last and second last conv layers will also be used in a multi-tasks feature distillation module. The distillation module will only be used during train stage and deactivated during inference stage, thus adding no extra FLOPs to the whole network.
