Several different DL algorithms have been proposed for multimodal learning. The most common are: (1) variants of the deep Boltzmann machine, which have been proposed to model the joint distribution from different modalities' data [1]}, [2]}; (2) extensions of classical autoencoders to discover correlations between hidden representations of two modalities [3]}, [4]}; and (3) non-linear extension of the Canonical Correlation Analysis (CCA) algorithm using deep neural networks [5]}. However, in their originally proposed forms these models are not scalable to the number of features (pixels/voxels) typically present in medical images as they were based on fully-connected neural networks. Recently, some convolutional neural network (CNN)-based architectures that combine information from multiple sources for image and shape recognition have been proposed [6]}, [7]}, [8]}.
With the aim of directly utilising the CMR and echocardiography data,
in this paper we employ a CNN-based architecture for deep multimodal classification.
