Let \(\lbrace M_i = \lbrace x_{i,j},...,x_{i, N}\rbrace \rbrace _{i=1,...,n}\)  be a set of modalities with \(N\)  samples per modality and \(n\)  modalities. Sample index \(j\)  corresponds to a subject and each subject will have a sample for each modality. Instead of learning each modality with a separate decoder, we enforce a shared subspace. Further, to make this approach scalable to a large number of modalities and because we already use a shared decoder, we also only use one encoder for all modalities. This forces the features that are learned for each modality to be similar and makes sure the model scales well in terms of memory usage. Further, given that neuroimaging datasets are considered small compared to more commonly used deep learning datasets, using multiple encoders may lead to overfitting. The encoder decoder couple is optimized with respect to the log-likelihood of the marginal distribution of \(M\)  and each volume is treated as an independent sample. The integral over the marginal distribution of \(M\) , \(p_{\theta }(x) = \int p_{\theta }(z) p_{\theta }(x|z) \mathit {d}z\)  is intractable. We therefore optimize the following evidence lower bound (ELBO) using a variational autoencoder (VAE) [1]}:
\(\begin{aligned}\log p_{\theta }(x_{i, j}) &\ge \mathcal {L}(\theta , \phi ; x_{i, j}) \\&= \mathbb {E}_{q_{\phi }(z|x)} \left[- \log q_{\phi } (z|x) + \log p_{\theta }(x, z) \right] \\&= -D_{\text{KL}}(q_{\phi }(z|x_{i,j}) \ || \ p_{\theta }(z)) \\ & \quad + \mathbb {E}_{q_{\phi }(z|x_{i,j)}} \left[\log p_{\theta }(x_{i,j}|z) \right]\end{aligned}\) 
