Recent work in multimodal deep learning applied to neuroimaging has focused on information maximization between representations extracted from two modalities [1]}, [2]} or by translating between modalities [3]}. This work aims to learn a continuous manifold of multiple modalities so that they are represented in a locally Euclidean space. The model architecture that is used is a variational autoencoder (VAE) [4]}, which maximizes a lower bound on the log-likelihood of the data's marginal distribution. Other work on multimodal VAEs focuses on a factorization of shared and private subspaces [5]} and uses a separate encoder for each modality. In this work, we intentionally force all of the modalities to inhabit the same shared subspace by using a single encoder-decoder pair for all modalities. This allows us to interpolate between modalities, similar to how VAEs have previously been used to interpolate between different digits in the MNIST dataset [4]}.
