Vision Transformers (ViTs) [1]}, [2]}, [3]}, [4]}, as the latest performant deep models, have achieved impressive performance on various computer vision tasks  [5]}, [6]}, [7]}. These models are specially trained or tested for only one or a few tasks; however, many real-world applications require one compact system that can handle many different tasks efficiently, and often need to swiftly switch between tasks per demand. For example, an autonomous driving system [8]} needs to perform and switch between many tasks such as drivable area estimation, lane detection, pedestrian detection, and scene classification: apparently both single task inference and cross-task switching need to happen at ultra-low latency. As another example, smart-home indoor robots [9]} are expected to address semantic segmentation, navigation, tracking, or other tasks in varying contexts, with very limited on-board resources. Multi-task learning (MTL) [10]}, [11]}, [12]} solves multiple tasks simultaneously within a single model and learns improved feature representations [13]} shared by related tasks [14]}, [15]}. Therefore, accomplishing realistic efficient MTL is becoming a key knob for building real-time sophisticated AI systems.
