The relation between Granger causality and directed information theory: a review <s> V. CONCLUSIONS <s> We present some new results on the nonparametric estimation of entropy and mutual information. First, we use an exact local expansion of the entropy function to prove almost sure consistency and central limit theorems for three of the most commonly used discretized information estimators. The setup is related to Grenander's method of sieves and places no assumptions on the underlying probability measure generating the data. Second, we prove a converse to these consistency theorems, demonstrating that a misapplication of the most common estimation techniques leads to an arbitrarily poor estimate of the true information, even given unlimited data. This "inconsistency" theorem leads to an analytical approximation of the bias, valid in surprisingly small sample regimes and more accurate than the usual 1/N formula of Miller and Madow over a large region of parameter space. The two most practical implications of these results are negative: (1) information estimates in a certain data regime are likely contaminated by bias, even if "bias-corrected" estimators are used, and (2) confidence intervals calculated by standard techniques drastically underestimate the error of the most common estimation methods.Finally, we note a very useful connection between the bias of entropy estimators and a certain polynomial approximation problem. By casting bias calculation problems in this approximation theory framework, we obtain the best possible generalization of known asymptotic bias results. More interesting, this framework leads to an estimator with some nice properties: the estimator comes equipped with rigorous bounds on the maximum error over all possible underlying probability distributions, and this maximum error turns out to be surprisingly small. We demonstrate the application of this new estimator on both real and simulated data. <s> BIB001 </s> The relation between Granger causality and directed information theory: a review <s> V. CONCLUSIONS <s> We present two classes of improved estimators for mutual information M(X,Y), from samples of random points distributed according to some joint probability density mu(x,y). In contrast to conventional estimators based on binnings, they are based on entropy estimates from k -nearest neighbor distances. This means that they are data efficient (with k=1 we resolve structures down to the smallest possible scales), adaptive (the resolution is higher where data are more numerous), and have minimal bias. Indeed, the bias of the underlying entropy estimates is mainly due to nonuniformity of the density at the smallest resolved scale, giving typically systematic errors which scale as functions of k/N for N points. Numerically, we find that both families become exact for independent distributions, i.e. the estimator M(X,Y) vanishes (up to statistical fluctuations) if mu(x,y)=mu(x)mu(y). This holds for all tested marginal distributions and for all dimensions of x and y. In addition, we give estimators for redundancies between more than two random variables. We compare our algorithms in detail with existing algorithms. Finally, we demonstrate the usefulness of our estimators for assessing the actual independence of components obtained from independent component analysis (ICA), for improving ICA, and for estimating the reliability of blind source separation. <s> BIB002 </s> The relation between Granger causality and directed information theory: a review <s> V. CONCLUSIONS <s> The identification of effective connectivity from time-series data such as electroencephalogram (EEG) or time-resolved function magnetic resonance imaging (fMRI) recordings is an important problem in brain imaging. One commonly used approach to inference effective connectivity is based on vector autoregressive models and the concept of Granger causality. However, this probabilistic concept of causality can lead to spurious causalities in the presence of latent variables. Recently, graphical models have been used to discuss problems of causal inference for multivariate data. In this paper, we extend these concepts to the case of time-series and present a graphical approach for discussing Granger-causal relationships among multiple time-series. In particular, we propose a new graphical representation that allows the characterization of spurious causality and, thus, can be used to investigate spurious causality. The method is demonstrated with concurrent EEG and fMRI recordings which are used to investigate the interrelations between the alpha rhythm in the EEG and blood oxygenation level dependent (BOLD) responses in the fMRI. The results confirm previous findings on the location of the source of the EEG alpha rhythm. <s> BIB003 </s> The relation between Granger causality and directed information theory: a review <s> V. CONCLUSIONS <s> In this paper, we will present and review the most usual methods to detect linear and nonlinear causality between signals: linear Granger causality test (Geweke in J Am Stat Assoc 77:304–313, 1982) extended to direct causality in multivariate case (LGC), directed coherence (DCOH, Saito and Harashima in Recent advances in EEG and EMG data processing, Elsevier, Amsterdam, 1981), partial directed coherence (PDC, Sameshima and Baccala 1999) and nonlinear Granger causality test of Baek and Brock (in Working Paper University of Iowa, 1992) extended to direct causality in multivariate case (partial nonlinear Granger causality, PNGC). All these methods are tested and compared on several ARX, Poisson and nonlinear models, and on neurophysiological data (depth EEG). The results show that LGC, DCOH and PDC are not very robust in relation to nonlinear linkages but they seem to correctly find linear linkages if only the autoregressive parts are nonlinear. PNGC is extremely dependent on the choice of parameters. Moreover, LGC and PNGC may give misleading results in the case of causality on a spectral band, which is illustrated by our neurophysiological database. <s> BIB004 </s> The relation between Granger causality and directed information theory: a review <s> V. CONCLUSIONS <s> The directed transfer function (DTF) has been proposed as a measure of information flow between the components of multivariate time series. In this paper, we discuss the interpretation of the DTF and compare it with other measures for directed relationships. In particular, we show that the DTF does not indicate multivariate or bivariate Granger causality, but that it is closely related to the concept of impulse response function and can be viewed as a spectral measure for the total causal influence from one component to another. Furthermore, we investigate the statistical properties of the DTF and establish a simple significance level for testing for the null hypothesis of no information flow. <s> BIB005 </s> The relation between Granger causality and directed information theory: a review <s> V. CONCLUSIONS <s> We introduce graphical time series models for the analysis of dynamic relationships among variables in multivariate time series. The modelling approach is based on the notion of strong Granger causality and can be applied to time series with non-linear dependences. The models are derived from ordinary time series models by imposing constraints that are encoded by mixed graphs. In these graphs each component series is represented by a single vertex and directed edges indicate possible Granger-causal relationships between variables while undirected edges are used to map the contemporaneous dependence structure. We introduce various notions of Granger-causal Markov properties and discuss the relationships among them and to other Markov properties that can be applied in this context. Examples for graphical time series models include nonlinear autoregressive models and multivariate ARCH models. <s> BIB006 </s> The relation between Granger causality and directed information theory: a review <s> V. CONCLUSIONS <s> Synchronization, a basic nonlinear phenomenon, is widely observed in diverse complex systems studied in physical, biological and other natural sciences, as well as in social sciences, economy and finance. While studying such complex systems, it is important not only to detect synchronized states, but also to identify causal relationships (i.e. who drives whom) between concerned (sub) systems. The knowledge of information-theoretic measures (i.e. mutual information, conditional entropy) is essential for the analysis of information flow between two systems or between constituent subsystems of a complex system. However, the estimation of these measures from a set of finite samples is not trivial. The current extensive literatures on entropy and mutual information estimation provides a wide variety of approaches, from approximation-statistical, studying rate of convergence or consistency of an estimator for a general distribution, over learning algorithms operating on partitioned data space to heuristical approaches. The aim of this paper is to provide a detailed overview of information theoretic approaches for measuring causal influence in multivariate time series and to focus on diverse approaches to the entropy and mutual information estimation. <s> BIB007 </s> The relation between Granger causality and directed information theory: a review <s> V. CONCLUSIONS <s> We propose a new measure of conditional dependence of random variables, based on normalized cross-covariance operators on reproducing kernel Hilbert spaces. Unlike previous kernel dependence measures, the proposed criterion does not depend on the choice of kernel in the limit of infinite data, for a wide class of kernels. At the same time, it has a straightforward empirical estimate with good convergence behaviour. We discuss the theoretical properties of the measure, and demonstrate its application in experiments. <s> BIB008 </s> The relation between Granger causality and directed information theory: a review <s> V. CONCLUSIONS <s> A new universal estimator of divergence is presented for multidimensional continuous densities based on k-nearest-neighbor (k-NN) distances. Assuming independent and identically distributed (i.i.d.) samples, the new estimator is proved to be asymptotically unbiased and mean-square consistent. In experiments with high-dimensional data, the k-NN approach generally exhibits faster convergence than previous algorithms. It is also shown that the speed of convergence of the k-NN method can be further improved by an adaptive choice of k. <s> BIB009 </s> The relation between Granger causality and directed information theory: a review <s> V. CONCLUSIONS <s> Understanding causal relationships, or effective connectivity, between parts of the brain is of utmost importance because a large part of the brain’s activity is thought to be internally generated and, hence, quantifying stimulus response relationships alone does not fully describe brain dynamics. Past efforts to determine effective connectivity mostly relied on model based approaches such as Granger causality or dynamic causal modeling. Transfer entropy (TE) is an alternative measure of effective connectivity based on information theory. TE does not require a model of the interaction and is inherently non-linear. We investigated the applicability of TE as a metric in a test for effective connectivity to electrophysiological data based on simulations and magnetoencephalography (MEG) recordings in a simple motor task. In particular, we demonstrate that TE improved the detectability of effective connectivity for non-linear interactions, and for sensor level MEG signals where linear methods are hampered by signal-cross-talk due to volume conduction. <s> BIB010 </s> The relation between Granger causality and directed information theory: a review <s> V. CONCLUSIONS <s> The ability to identify directional interactions that occur among multiple neurons in the brain is crucial to an understanding of how groups of neurons cooperate in order to generate specific brain functions. However, an optimal method of assessing these interactions has not been established. Granger causality has proven to be an effective method for the analysis of the directional interactions between multiple sets of continuous-valued data, but cannot be applied to neural spike train recordings due to their discrete nature. This paper proposes a point process framework that enables Granger causality to be applied to point process data such as neural spike trains. The proposed framework uses the point process likelihood function to relate a neuron's spiking probability to possible covariates, such as its own spiking history and the concurrent activity of simultaneously recorded neurons. Granger causality is assessed based on the relative reduction of the point process likelihood of one neuron obtained excluding one of its covariates compared to the likelihood obtained using all of its covariates. The method was tested on simulated data, and then applied to neural activity recorded from the primary motor cortex (MI) of a Felis catus subject. The interactions present in the simulated data were predicted with a high degree of accuracy, and when applied to the real neural data, the proposed method identified causal relationships between many of the recorded neurons. This paper proposes a novel method that successfully applies Granger causality to point process data, and has the potential to provide unique physiological insights when applied to neural spike trains. <s> BIB011 </s> The relation between Granger causality and directed information theory: a review <s> V. CONCLUSIONS <s> We propose a new type of probabilistic graphical model, based on directed information, to represent the causal dynamics between processes in a stochastic system. We show the practical significance of such graphs by proving their equivalence to generative model graphs which succinctly summarize interdependencies for causal dynamical systems under mild assumptions. This equivalence means that directed information graphs may be used for causal inference and learning tasks in the same manner Bayesian networks are used for correlative statistical inference and learning. <s> BIB012 </s> The relation between Granger causality and directed information theory: a review <s> V. CONCLUSIONS <s> In recent years, Granger causality has become a popular method in a variety of research areas including engineering, neuroscience, and economics. However, despite its simplicity and wide applicability, the linear Granger causality is an insufficient tool for analyzing exotic stochastic processes such as processes involving non-linear dynamics or processes involving causality in higher order statistics. In order to analyze such processes more reliably, a different approach toward Granger causality has become increasingly popular. This new approach employs conditional independence as a tool to discover Granger non-causality without any assumption on the underlying stochastic process. This paper discusses the concept of discovering Granger non-causality using measures of conditional independence, and proposes a novel measure of conditional independence. In brief, the proposed approach estimates the conditional distribution function through a kernel based least square regression approach. This paper also explores the strengths and weaknesses of the proposed method compared to other available methods, and provides a detailed comparison of these methods using a variety of synthetic data sets. <s> BIB013
Granger causality was developed originally in econometrics, and it is now transdisciplinary, with the literature on the subject is widely dispersed. We have tried here to sum up the profound links that exist between Granger causality and directed information theory. The key ingredients to build these links are conditional independence and the recently introduced causal conditioning. We have eluded the important question of how to practically use the definitions and measures presented here. Some of the measures can be used and implemented easily, especially in the linear Gaussian case. In a more general case, different approaches can be taken. The information-theoretic measures can be estimated, or the prediction can be explicitly carried out and the residuals used to assess causality. Many studies have been carried out over the last 20 years on the problem of estimation of information-theoretic measures. We refer to  ,  , BIB001 , BIB002 ,  for information on the different ways to estimate information measures. Recent studies into the estimation of entropy and/or information measures are  , BIB009 ,  . The recent report by BIB010 extensively details and applies transfer entropy in neuroscience using k-nearest neighbors type of estimators. Concerning the applications, important reviews include BIB007 ,  , where some of the ideas discussed here are also mentioned, and where practicalities such as the use of surrogate data, for example, are extensively discussed. Applications for neuroscience are discussed in  , BIB004 , BIB011 , BIB003 , BIB005 . Information-theoretic measures of conditional independence based on Kullback divergence were chosen here to illustrate the links between Granger causality and (usual) directed information theory. Other type of divergence could have been chosen; metrics in probability space could also be useful in the assessing of conditional independence. As an illustration, we refer to the study of Fukumizu and co-workers BIB008 , where conditional independence was evaluated using the Hilbert-Schmidt norm of an operator between reproducing kernel Hilbert spaces. The operator generalizes the partial covariance between two random vectors given a third one, and is called the conditional covariance operator. Furthermore, the Hilbert-Schmidt norm of conditional covariance operator can be efficiently estimated from data. A related approach is also detailed in BIB013 . Many important directions can be followed. Causality between nonstationary processes has rarely been considered (see however BIB010 for an ad-hoc approach in neuroscience). A very promising methodology is to adopt a graphical modeling way of thinking. The result of BIB006 on the structural properties of Markov-Granger causality graphs can be used to identify such graphs from real datasets. A first step in this direction was proposed by BIB012 . Assuming that the network under study is a network of sparsely connected nodes and that some Markov properties hold, efficient estimation procedures can be designed, as is the case in usual graphical modeling.