Multimodal registration is a challenging problem in medical imaging due the high variability of tissue appearance under different imaging modalities. The crucial component here is the choice of the right similarity measure. We make a step towards a general learning-based solution that can be adapted to specific situations and present a metric based on a convolutional neural network. Our network can be trained from scratch even from a few aligned image pairs. The metric is validated on intersubject deformable registration on a dataset different from the one used for training, demonstrating good generalization. In this task, we outperform mutual information by a significant margin.
The power of CNNs to capture complex relationships between multimodal medical images has been shown in the problem of modality synthesis @cite_7 , where CNNs are used to map MRI-T2 images to MRI-T1 images using jointly the appearance of a small patch together with its localization. Our work is arguably most similar to the approach of Cheng al @cite_9 who train a multilayer fully-connected network pretrained with autoencoder for estimating similarity of 2D CT-MR patch pairs. Our network is a CNN, which enables us to scale to 3D due to weight sharing and train from scratch. Moreover, we evaluate our metric on the actual task of registration, unlike Cheng al
@cite_9 @cite_7
2334328509 2397287600
The present embodiments relate to machine learning for multimodal image data. By way of introduction, the present embodiments described below include apparatuses and methods for learning a similarity metric using deep learning based techniques for multimodal medical images. A novel similarity metric for multi-modal images is provided using the corresponding states of pairs of image patches to generate a classification setting for each pair. The classification settings are used to train a deep neural network via supervised learning. A multi-modal stacked denoising auto encoder (SDAE) is used to pre-train the neural network. A continuous and smooth similarity metric is constructed based on the output of the neural network before activation in the last layer. The trained similarity metric may be used to improve the results of image fusion. Cross-modality image synthesis has recently gained significant interest in the medical imaging community. In this paper, we propose a novel architecture called location-sensitive deep network LSDN for synthesizing images across domains. Our network integrates intensity feature from image voxels and spatial information in a principled manner. Specifically, LSDN models hidden nodes as products of features and spatial responses. We then propose a novel method, called ShrinkConnect, for reducing the computations of LSDN without sacrificing synthesis accuracy. ShrinkConnect enforces simultaneous sparsity to find a compact set of functions that accurately approximates the responses of all hidden nodes. Experimental results demonstrate that LSDN+ShrinkConnect outperforms the state of the art in cross-domain synthesis of MRI brain scans by a significant margin. Our approach is also computationally efficient, e.g. 26× faster than other sparse representation based methods.