While generative models have shown great success in generating high-dimensional samples conditional on low-dimensional descriptors (learning e.g. stroke thickness in MNIST, hair color in CelebA, or speaker identity in Wavenet), their generation out-of-sample poses fundamental problems. The conditional variational autoencoder (CVAE) as a simple conditional generative model does not explicitly relate conditions during training and, hence, has no incentive of learning a compact joint distribution across conditions. We overcome this limitation by matching their distributions using maximum mean discrepancy (MMD) in the decoder layer that follows the bottleneck. This introduces a strong regularization both for reconstructing samples within the same condition and for transforming samples across conditions, resulting in much improved generalization. We refer to the architecture as transformer VAE (trVAE). Benchmarking trVAE on high-dimensional image and tabular data, we demonstrate higher robustness and higher accuracy than existing approaches. In particular, we show qualitatively improved predictions for cellular perturbation response to treatment and disease based on high-dimensional single-cell gene expression data, by tackling previously problematic minority classes and multiple conditions. For generic tasks, we improve Pearson correlations of high-dimensional estimated means and variances with their ground truths from 0.89 to 0.97 and 0.75 to 0.87, respectively.
 The task of generating high-dimensional samples x conditional on a latent random vector z and a categorical variable s has established solutions (Mirza & Osindero, 2014; Ren et al., 2016) . The situation becomes more complicated if the support of z is divided into different domains d with different semantic meanings: say d ∈ {men, women} and one is interested in out-of-sample generation of samples x in a domain and condition (d, s) that is not part of the training data. If one predicts how a given black-haired man would look with blonde hair, which we refer to as transforming x men, black-hair → x men, blonde-hair , this becomes an out-of-sample problem if the training data does not have instances of blonde-haired men, but merely of blonde-and black-haired woman and blacked haired men. In an application with higher relevance, there is strong interest in how untreated (s = 0) humans (d = 0) respond to drug treatment (s = 1) based on training data from in vitro (d = 1) and mice (d = 2) experiments. Hence, the target domain of interest (d = 0) does not offer training data for s = 1, but only for s = 0. In the present paper, we suggest to address the challenge of transforming out-of-sample by regularizing the joint distribution across the categorical variable s using maximum mean discrepancy (MMD) in the framework of a conditional variational autoencoder (CVAE) (Sohn et al., 2015) . This produces a more compact representation of a distribution that displays high variance in the vanilla CVAE, which incentivizes learning of features across s and results in more accurate out-of-sample prediction. MMD has proven successful in a variety of tasks. In particular, matching distributions with MMD in variational autoencoders (Kingma & Welling, 2013) has been put forward for unsupervised domain adaptation (Louizos et al., 2015) or for learning statistically independent latent dimensions (Lopez et al., 2018b) . In supervised domain adaptation approaches, MMD-based regularization has been shown to be a viable strategy of learning label-predictive features with domain-specific information removed (Long et al., 2015; Tzeng et al., 2014) . In further related work, the out-of-sample transformation problem was addressed via hard-coded latent space vector arithmetics (Lotfollahi et al., 2019) and histogram matching (Amodio et al., 2018) . The approach of the present paper, however, introduce a data-driven end-to-end approach, which does not involve hard-coded elements and generalizes to more than one condition. By arguing that the vanilla CVAE yields representations in the first layer following the bottleneck that vary strongly across categorical conditions, we introduced an MMD regularization that forces these representations to be similar across conditions. The resulting model (trVAE) outperforms existing modeling approaches on benchmark and real-world data sets. Within the bottleneck layer, CVAEs already display a well-controlled behavior, and regularization does not improve performance. Further regularization at later layers might be beneficial but is numerically costly and unstable as representations become high-dimensional. However, we have not yet systematically investigated this and leave it for future studies. Further future work will concern the application of trVAE on larger and more data, focusing on interaction effects among conditions. For this, an important application is the study of drug interaction effects, as previously noted by Amodio et al. (2018) . Future conceptual investigations concern establishing connections to causal-inference-inspired models such as CEVAE (Louizos et al., 2017) : faithful modeling of an interventional distribution might possibly be re-framed as successful perturbation effect prediction across domains. A HYPER-PARAMETERS
Generates never seen data during training from a desired condition 
{"src": "scitldr-AIC_train", "paper_id": "Hkl-7C4FvH"}