Despite promising progress on unimodal data imputation (e.g. image inpainting), models for multimodal data imputation are far from satisfactory. In this work, we propose variational selective autoencoder (VSAE) for this task. Learning only from partially-observed data, VSAE can model the joint distribution of observed/unobserved modalities and the imputation mask, resulting in a unified model for various down-stream tasks including data generation and imputation. Evaluation on synthetic high-dimensional and challenging low-dimensional multimodal datasets shows significant improvement over state-of-the-art imputation models. Modern deep learning techniques rely heavily on extracting information from large scale datasets of clean and complete training data, such as labeled data or images with all pixels. Practically these data is costly due to the limited resources or privacy concerns. Having a model that learns and extracts information from partially-observed data will largely increase the application spectrum of deep learning models and provide benefit to down-stream tasks, e.g. data imputation, which has been an active research area. Despite promising progress, there are still challenges in learning effective imputation models: 1) Some prior works focus on learning from fully-observed data and then performing imputation on partially-observed data (Suzuki et al., 2016; Ivanov et al., 2019) ; 2) They usually have strong assumptions on missingness mechanism (see Appendix A.1) such as data is missing completely at random (MCAR) (Yoon et al., 2018) ; 3) Some other works explore only unimodal imputation such as image in-painting for high-dimensional data (Ivanov et al., 2019; Mattei and Frellsen, 2019) . Modeling any combination of data modalities has not been well-established yet. This can limit the potential of such models since raw data in real-life is usually acquired in a multimodal manner (Ngiam et al., 2011) . A class of prior works focus on learning the conditional likelihood of the modalities (Sohn et al., 2015; Pandey and Dukkipati, 2017) . However, they require complete data during training and cannot handle arbitrary conditioning. In practice, one or more of the modalities maybe be missing, leading to a challenging multimodal data imputation task. For more on related works, see Appendix A.2. The unimodal/multimodal proposal networks are employed by selection indicated by the arrows. Standard normal prior is ignored for simplicity. φ, ψ, θ and are the parameters of each modules. All components are trained jointly. We propose Variational Selective Autoencoder (VSAE) for multimodal data generation and imputation. It can model the joint distribution of data and mask and avoid the limited assumptions such as MCAR. VSAE is optimized efficiently with a single variational objective. The contributions are summarized as: (1) A novel variational framework to learn from partially-observed multimodal data; (2) VSAE can learn the joint distribution of observed/unobserved modalities and the mask, resulting in a unified model for various down-stream tasks including data generation/imputation with relaxed assumptions on missigness mechanism; (3) Evaluation on both synthetic high-dimensional and challenging low-dimensional multimodal datasets shows improvement over the state-of-the-art data imputation models.
We propose a novel VAE-based framework learning from partially-observed data for imputation and generation. 
{"src": "scitldr-AIC_train", "paper_id": "Byx-FknVtS"}