We introduce CGNN, a framework to learn functional causal models as generative neural networks. These networks are trained using backpropagation to minimize the maximum mean discrepancy to the observed data. Unlike previous approaches, CGNN leverages both conditional independences and distributional asymmetries to seamlessly discover bivariate and multivariate 
 causal structures, with or without hidden variables. CGNN does not only estimate the causal structure, but a full and differentiable generative model of the data. Throughout an extensive variety of experiments, we illustrate the competitive  esults of CGNN w.r.t state-of-the-art alternatives in observational causal discovery on both simulated and real data, in the tasks of cause-effect inference, v-structure identification, and multivariate causal discovery. Deep learning models have shown extraordinary predictive abilities, breaking records in image classification BID19 , speech recognition , language translation BID1 , and reinforcement learning BID33 . However, the predictive focus of black-box deep learning models leaves little room for explanatory power. In particular, current machine learning paradigms offer no protection to avoid mistaking correlation by causation. For example, consider that we are interested in predicting a target variable Y given a feature vector (X 1 , X 2 ). Assume that the generative process underlying (X 1 , X 2 , Y ) is described by the equations: DISPLAYFORM0 , where (E X2 , E y ) are additive noise variables. These equations tell that the values of Y are computed as a function of the values of X 1 , and that the values of X 2 are computed as a function of the values of Y . The "assignment arrows" emphasize the asymmetric relations between the three random variables: we say that "X 1 causes Y ", and that "Y causes X 2 ". However, since X 2 provides a stronger signal-to-noise ratio for the prediction of Y , the least-squares solution to this problem iŝ Y = 0.25X 1 + 0.5X 2 , a typical case of inverse regression BID7 . Such least-squares prediction would explain some changes in Y as a function of changes in X 2 . This is a wrong explanation, since X 2 does not cause the computation of Y . Even though there exists the necessary machinery to detect all the cause-effect relations in this example BID15 , common machine learning solutions will misunderstand how manipulating the values and distributions of (X 1 , X 2 ), or how changing the mapping from Y to X 2 , affect the values of Y . Mistaking correlation by causation can be catastrophic for agents who must plan, reason, and decide based on observation. Thus, discovering causal structures is of crucial importance.The gold standard to discover causal relations is to perform experiments BID27 . However, experiments are in many cases expensive, unethical, or impossible to realize. In these situations, there is a need for observational causal discovery, that is, the estimation of causal relations from observation alone BID35 BID28 . The literature in observational causal discovery is vast (see Appendix B for a brief survey), but lacks a unified solution. For instance, some approaches rely on distributional asymmetries to discover bivariate causal relations BID15 BID40 BID4 BID37 BID6 , while others rely on conditional independence to discover structures on three or more variables BID35 BID0 . Furthermore, different algorithms X 5 DISPLAYFORM1 Figure 1: Example of causal graph and associated functional model for X = (X 1 , . . . , X 5 ).FCMs are generative models. We can draw a sample x = (x 1 , . . . , x d ) from the distribution P := P (X) by observing the FCM at play. First, draw e i ∼ Q for all i = 1, . . . , d. Second , construct Pa(i;G) , e i ) in the topological order of G. Since this process observes but does not manipulate the equations of the FCM, we call x one observational sample from P , the observational distribution of X. However, one FCM contains more information than the observational distribution alone, since we can decide to manipulate any of its equations and obtain a new distribution. For instance , we could decide to set and hold constant X j = 0.1, hereby removing all the causal influences X k → X j , for all k ∈ Pa(j; G). We denote by P do(Xj =0.1) (X) the corresponding interventional distribution. Importantly, intervening is different from conditioning (correlation does not imply causation). Understanding the effect of interventions requires the (partial) knowledge of the FCM. This is why this work focuses on discovering such causal structures from data. DISPLAYFORM2 Formal definitions and assumptions Two random variables (X, Y ) are conditionally independent given Z if P (X, Y |Z) = P (X|Z)P (Y |Z). Three of random variables (X, Y, Z) form a v-structure iff their causal structure is X → Z ← Y . The random variable Z is a confounder (or common cause) of the pair of random variables (X, Y ) if (X, Y, Z) have causal structure X ← Z → Y . The skeleton U of a DAG G is obtained by replacing all the directed edges in G by undirected edges.Discovering the causal structure of a random vector is a difficult task when considered in full generality. Because of this reason, the literature in causal inference relies on a set of common assumptions BID27 . The causal sufficiency assumption states that there are no unobserved confounders. The causal Markov assumption states that all the d-separations in the causal graph G imply conditional independences in the observational distribution P . The causal faithfulness assumption states that all the conditional independences in the observational distribution P imply d-separations in the causal graph G. We call Markov equivalence class to the set of graphs containing the same set of d-separations. When using the causal faithfulness assumption and conditional independence information, we are able to recover the Markov equivalence class of the causal structure underlying a random vector -which, in some cases contains one graph, the causal structure itself. Markov equivalence classes are DAGs where some of the edges remain undirected.Learning FCMs from data using score methods Consider a random vector X = (X 1 , . . . , X d ) following the FCM C = (G, f, Q) with associated observational distribution P . Furthermore, assume access to n samples drawn from P , denoted by DISPLAYFORM3 , where DISPLAYFORM4 for all i = 1, . . . , n. Given these data, the goal of observational causal discovery is to estimate the underlying causal DAG G and the causal mechanisms f .One family of methods for observational causal discovery are score-based methods BID0 . In essence, score-based methods rely on some score-function S(G, D) to measure the fit between a candidate set {G, f } and the observed data D. Then, we select the DAG on d variables achieving the maximum score as measured by S. As an example of score-function, consider the Bayesian Information Criterion (BIC): DISPLAYFORM5 where pθ j the maximum-likelihood estimate of a simple parametric family of conditional distributions p θ∈Θ allowing efficient density evaluation. The term λ ∈ [0, ∞) penalizes the number of edges (that is, the model complexity assuming equal number of parameters per edge) in the graph. Finally, we may associate each edge X i → X j in G to an importance or confidence score proportional to its contribution to the overal loss: as DISPLAYFORM6 A naïve score-based method would enumerate all the DAGs of d variables and select the one maximizing S. Unfortunately, the number of DAGs over d nodes is super-exponential in d. Thus, the brute-force search of the best DAG is intractable, even for moderate d. Inspired by BID38 ; BID25 , we assume in this paper known graph skeletons . Such a skeleton may arise from expert knowledge or a feature selection algorithm algorithm BID39 under standard assumptions such as causal Markov, faithfulness, and sufficiency. Given a skeleton with k edges, causal discovery reduces to selecting one out of the O(2 k ) possible edge orientations. We introduced a new framework to learn functional causal models based on generative neural networks. We train these networks by minimizing the discrepancy between their generated samples and the observed data. Such models are instances of the bigger family of FCMs for which each function is a shallow neural network with n h hidden units.We believe that our approach opens new avenues of research, both from the point of view of leveraging the power of deep learning in causal discovery and from the point of view of building deep networks with better structure interpretability. Once the model is learned, the CGNNs present the advantage to be fully parametrized and may be used to simulate interventions on one or more variables of the model and evaluate their impact on a set of target variables. This usage is relevant in a wide variety of domains, typically among medical and sociological domains.Five directions for future work are to i) lower the computational cost of CGNN, ii) extend CGNN to deal with categorical data, iii) explore better heuristics for causal graph search, iv) adapt our methods for temporal data and v) obtain theoretical guarantees for basic use cases.
Discover the structure of functional causal models with generative neural networks
{"src": "scitldr-AIC_train", "paper_id": "ry5wc1bCW"}