Generative adversarial networks (GANs) have been shown to provide an effective way to model complex distributions and have obtained impressive results on various challenging tasks. However, typical GANs require fully-observed data during training. In this paper, we present a GAN-based framework for learning from complex, high-dimensional incomplete data. The proposed framework learns a complete data generator along with a mask generator that models the missing data distribution. We further demonstrate how to impute missing data by equipping our framework with an adversarially trained imputer. We evaluate the proposed framework using a series of experiments with several types of missing data processes under the missing completely at random assumption. Generative adversarial networks (GANs) BID0 provide a powerful modeling framework for learning complex high-dimensional distributions. Unlike likelihood-based methods, GANs are referred to as implicit probabilistic models BID8 . They represent a probability distribution through a generator that learns to directly produce samples from the desired distribution. The generator is trained adversarially by optimizing a minimax objective together with a discriminator. In practice, GANs have been shown to be very successful in a range of applications including generating photorealistic images BID3 . Other than generating samples, many downstream tasks require a good generative model, such as image inpainting BID9 BID15 .Training GANs normally requires access to a large collection of fully-observed data. However, it is not always possible to obtain a large amount of fully-observed data. Missing data is well-known to be prevalent in many real-world application domains where different data cases might have different missing entries. This arbitrary missingness poses a significant challenge to many existing machine learning models.Following BID6 , the generative process for incompletely observed data can be described as shown below where x ∈ R n is a complete data vector and m ∈ {0, 1} n is a binary mask 2 that determines which entries in x to reveal: DISPLAYFORM0 Let x obs denote the observed elements of x, and x mis denote the missing elements according to the mask m. In addition, let θ denote the unknown parameters of the data distribution, and φ denote the unknown parameters for the mask distribution, which are usually assumed to be independent of θ. In the standard maximum likelihood setting, the unknown parameters are estimated by maximizing the 1 Our implementation is available at https://github.com/steveli/misgan 2 The complementm is usually referred to as the missing data indicator in the literature.following marginal likelihood, integrating over the unknown missing data values:p(x obs , m) = p θ (x obs , x mis )p φ (m|x obs , x mis )dx mis .Little & Rubin ( 2014) characterize the missing data mechanism p φ (m|x obs , x mis ) in terms of independence relations between the complete data x = [x obs , x mis ] and the masks m:• Missing completely at random (MCAR): p φ (m|x) = p φ (m),• Missing at random (MAR): p φ (m|x) = p φ (m|x obs ),• Not missing at random (NMAR): m depends on x mis and possibly also x obs .Most work on incomplete data assumes MCAR or MAR since under these assumptions p(x obs , m) can be factorized into p θ (x obs )p φ (m|x obs ). With such decoupling, the missing data mechanism can be ignored when learning the data generating model while yielding correct estimates for θ. When p θ (x) does not admit efficient marginalization over x mis , estimation of θ is usually performed by maximizing a variational lower bound, as shown below, using the EM algorithm or a more general approach BID6 Ghahramani & Jordan, 1994) :log p θ (x obs ) ≥ E q(xmis|xobs) [log p θ (x obs , x mis ) − log q(x mis |x obs )] .The primary contribution of this paper is the development of a GAN-based framework for learning high-dimensional data distributions in the presence of incomplete observations. Our framework introduces an auxiliary GAN for learning a mask distribution to model the missingness. The masks are used to "mask " generated complete data by filling the indicated missing entries with a constant value. The complete data generator is trained so that the resulting masked data are indistinguishable from real incomplete data that are masked similarly.Our framework builds on the ideas of AmbientGAN (Bora et al., 2018) . AmbientGAN modifies the discriminator of a GAN to distinguish corrupted real samples from corrupted generated samples under a range of corruption processes (or measurement processes). For images, examples of the measurement processes include random dropout, blur, block-patch, and so on. Missing data can be seen as a special type of corruption, except that we have access to the missing pattern in addition to the corrupted measurements. Moreover, AmbientGAN assumes the measurement process is known or parameterized only by a few parameters, which is not the case in general missing data problems.We provide empirical evidence that the proposed framework is able to effectively learn complex, highdimensional data distributions from highly incomplete data when the GAN generator incorporates suitable priors on the data generating process. We further show how the architecture can be used to generate high-quality imputations. This work presents and evaluates a highly flexible framework for learning standard GAN data generators in the presence of missing data. Although we only focus on the MCAR case in this work, MisGAN can be easily extended to cases where the output of the data generator is provided to the mask generator. These modifications can capture both MAR and NMAR mechanisms. The question of learnability requires further investigation as the analysis in Section 3 no longer holds due to dependence between the transition matrix and the data distribution under MAR and NMAR. We have tried this modified architecture in our experiments and it showed similar results as to the original MisGAN. This suggests that the extra dependencies may not adversely affect learnability. We leave the formal evaluation of this modified framework for future work.A PROOF OF THEOREM 1 AND THEOREM 2Let P be the finite set of feature values. For the n-dimensional case, let M = {0, 1} n be the set of masks and I = P n be the set of all possible feature vectors. Also let D M be the set of probability distributions on M, which implies m 0 and v∈I m(v) = 1 for all m ∈ M, where m(v) denotes the entry of m indexed by v.Given τ ∈ P and q ∈ D M , define the transformation DISPLAYFORM0 where is the entry-wise multiplication and 1{·} is the indicator function.Given m ∈ M, define an equivalent relation ∼ m on I by v ∼ m u iff v m = u m, and denote by [v] m the equivalence class containing v.Given q ∈ D M , let S q ⊂ M be the support of q, that is, DISPLAYFORM1 Given τ ∈ P and v ∈ I, let M τ,v denote the set of masks consistent with v in the sense that q(m) > 0 and v m = τm, that is, DISPLAYFORM2 Proof. This is clear from the following equation DISPLAYFORM3 which can be obtained from (13) as follows, DISPLAYFORM4 Proposition 2. For any τ ∈ P, q ∈ D M and x ∈ R I , the vector T q,τ x determines the collection of marginals {x ([v] DISPLAYFORM5 Proof. Fix τ ∈ P, q ∈ D M and x ∈ R I . Since v m + τm ∈ [v] m , it suffices to show that we can solve for x ([v] m ) in terms of T q,τ x for m ∈ M τ,v = ∅. We use induction on the size of M τ,v .First consider the base case |M τ,v | = 1. Consider v 0 ∈ I with M τ,v0 = {m 0 }. By FORMULA0 , DISPLAYFORM6 , which proves the base case. Now assume we can solve for x ([v] m ) in terms of T q,τ x for m ∈ S q and v ∈ I with |M τ,v | ≤ k. Consider v 0 ∈ I with |M τ,v0 | = k + 1; if no such v 0 exists, the conclusion holds trivially. Let M τ,v0 = {m 0 , m 1 , . . . , m k }. We need to show that T q,τ x determines x([v 0 ] m ) for = 0, 1, . . . , k. By (14) again , DISPLAYFORM7 Let m = k =0 m , which may or may not belong to S q . Note that DISPLAYFORM8 and hence DISPLAYFORM9 Plugging FORMULA0 into FORMULA0 yields DISPLAYFORM10 Note that DISPLAYFORM11 It follows from FORMULA0 and FORMULA0 Theorem 1 is a direct consequence of Proposition 1 and Proposition 2 as the collection of marginals {x ([v] m ) : v ∈ I, m ∈ S q } is independent of τ . Therefore, if x 1 , x 2 ∈ R I satisfy T q,τ0 x 1 = T q,τ0 x 2 for some τ 0 ∈ P, then T q,τ x 1 = T q,τ x 2 for all τ ∈ P. Theorem 1 is a special case when x 1 = 0.Moreover, Proposition 2 also shows that MisGAN overall learns the distribution p(x obs , m), as x([v] m ) is equivalent to p(x obs |m) and T q,τ x is essentially the distribution of f τ (x, m) under the optimally learned missingness q = p(m). Theorem 2 basically restates Proposition 1 and Proposition 2. This is also true when τ / ∈ P according to Appendix B.
This paper presents a GAN-based framework for learning the distribution from high-dimensional incomplete data.
{"src": "scitldr-AIC_train", "paper_id": "S1lDV3RcKm"}