To address this question, we propose a single generic model that jointly learns classification and segmentation tasks in skin images. Our framework, called Multi-Task TransUNet (MT-TransUNet), inherits the merits of both Convolutional Neural Networks and Transformers to capture both local details (skin lesion color, texture) and long-range context (skin lesion shape, physical size) in a multi-task learning pipeline. What makes our framework most distinct from the latest medical transformers ([1]}) is that instead of tailoring for different downstream tasks separately, we jointly learn the two complementary tasks with the new design of classification and segmentation tokens. To mediate those two types of tokens, consistencies are further enforced in different levels, the intermediate activated attention map as well as the final segmentation outputs, which we find is crucial for successful multi-task learning in skin images.
