We propose a new architecture to jointly segment the lesion, classify the pathology, reconstruct the image and predict the outcome. The proposed network is shown in figure REF . We use U-Net as the backbone due to its great performance in 3D medical image segmentation. The architecture consists of four parts: (i) a common encoding part, (ii) a decoding part for reconstruction, iii) a decoding part for the segmentation and (iii) skip connections between them, which form a W, see figure REF . To that we add a multi-layer perceptron (MLP) for the classification task, and a convolutional neural network (CNN) for the outcome prediction based on the segmentation result. Finally, we use multi-scale approach to feed global features to the CNN, to make a prediction on both global features and tumor ones. To summarize, many classic image classification networks use transfer learning [1]} to extract high level features from CNN models, such as VGG16 [2]} or ResNET [3]}. Motivated by this, we use the same encoder for lesion segmentation and pathology classification to extract common features. We add reconstruction task as a secondary task so that the neural network can extract meaningful features about PET images.
