The relation between Granger causality and directed information theory: a review <s> B. Granger causality inference <s> In this paper we propose causality measures based on the Kullback Information Criterion. These causality measures are applicable in a general context which contains, as special cases, the stationary autoregressive case, considered by GEWEKE, and qualitative models. Estimators of these measures and test procedures are proposed. The nesting of the hypotheses and the asymptotic independence of the test statistics are carefully studied. <s> BIB001 </s> The relation between Granger causality and directed information theory: a review <s> B. Granger causality inference <s> New measures are proposed for mutual and causal dependence between two time series, based on information theoretical ideas. The measure of mutual dependence is shown to be the sum of the measure of unidirectional causal dependence from the first time series to the second, the measure of unidirectional causal dependence from the second to the first, and the measure of instantaneous causal dependence. The measures are applicable to any kind of time series: continuous, discrete, or categorical. <s> BIB002 </s> The relation between Granger causality and directed information theory: a review <s> B. Granger causality inference <s> We discuss a recently proposed quantity, called transfer entropy, which uses time series data to measure the amount of information transferred from one process to another. In order to understand its foundation, merits, and limitations, we review some aspects of information theoretic functionals. While for symbol sequences these measures have an intuitive interpretation, their application to continuous state processes and, in particular, their estimation from finite data sets is problematic. For mutual information, finite length scale estimates converge from below and can thus be used to reject the assumption that the observed processes are independent. However, mutual information does not provide any directional information. Conversely, transfer entropy does resolve the directionality of information exchange but no similar monotonic convergence seems to hold. Thus, only in the case of zero transfer entropy in one direction we can reliably infer an asymmetry of the information exchange. <s> BIB003 </s> The relation between Granger causality and directed information theory: a review <s> B. Granger causality inference <s> Synchronization, a basic nonlinear phenomenon, is widely observed in diverse complex systems studied in physical, biological and other natural sciences, as well as in social sciences, economy and finance. While studying such complex systems, it is important not only to detect synchronized states, but also to identify causal relationships (i.e. who drives whom) between concerned (sub) systems. The knowledge of information-theoretic measures (i.e. mutual information, conditional entropy) is essential for the analysis of information flow between two systems or between constituent subsystems of a complex system. However, the estimation of these measures from a set of finite samples is not trivial. The current extensive literatures on entropy and mutual information estimation provides a wide variety of approaches, from approximation-statistical, studying rate of convergence or consistency of an estimator for a general distribution, over learning algorithms operating on partitioned data space to heuristical approaches. The aim of this paper is to provide a detailed overview of information theoretic approaches for measuring causal influence in multivariate time series and to focus on diverse approaches to the entropy and mutual information estimation. <s> BIB004 </s> The relation between Granger causality and directed information theory: a review <s> B. Granger causality inference <s> Advances in recording technologies have given neuroscience researchers access to large amounts of data, in particular, simultaneous, individual recordings of large groups of neurons in different parts of the brain. A variety of quantitative techniques have been utilized to analyze the spiking activities of the neurons to elucidate the functional connectivity of the recorded neurons. In the past, researchers have used correlative measures. More recently, to better capture the dynamic, complex relationships present in the data, neuroscientists have employed causal measures-most of which are variants of Granger causality-with limited success. This paper motivates the directed information, an information and control theoretic concept, as a modality-independent embodiment of Granger's original notion of causality. Key properties include: (a) it is nonzero if and only if one process causally influences another, and (b) its specific value can be interpreted as the strength of a causal relationship. We next describe how the causally conditioned directed information between two processes given knowledge of others provides a network version of causality: it is nonzero if and only if, in the presence of the present and past of other processes, one process causally influences another. This notion is shown to be able to differentiate between true direct causal influences, common inputs, and cascade effects in more two processes. We next describe a procedure to estimate the directed information on neural spike trains using point process generalized linear models, maximum likelihood estimation and information-theoretic model order selection. We demonstrate that on a simulated network of neurons, it (a) correctly identifies all pairwise causal relationships and (b) correctly identifies network causal relationships. This procedure is then used to analyze ensemble spike train recordings in primary motor cortex of an awake monkey while performing target reaching tasks, uncovering causal relationships whose directionality are consistent with predictions made from the wave propagation of simultaneously recorded local field potentials. <s> BIB005 </s> The relation between Granger causality and directed information theory: a review <s> B. Granger causality inference <s> Understanding causal relationships, or effective connectivity, between parts of the brain is of utmost importance because a large part of the brain’s activity is thought to be internally generated and, hence, quantifying stimulus response relationships alone does not fully describe brain dynamics. Past efforts to determine effective connectivity mostly relied on model based approaches such as Granger causality or dynamic causal modeling. Transfer entropy (TE) is an alternative measure of effective connectivity based on information theory. TE does not require a model of the interaction and is inherently non-linear. We investigated the applicability of TE as a metric in a test for effective connectivity to electrophysiological data based on simulations and magnetoencephalography (MEG) recordings in a simple motor task. In particular, we demonstrate that TE improved the detectability of effective connectivity for non-linear interactions, and for sensor level MEG signals where linear methods are hampered by signal-cross-talk due to volume conduction. <s> BIB006 </s> The relation between Granger causality and directed information theory: a review <s> B. Granger causality inference <s> Assessing the causal relationship among multivariate time series is a crucial problem in many fields. Granger causality has been widely used to identify the causal interactions between continuous-valued time series based on multivariate autoregressive models in the Gaussian case. In order to extend the application of the Granger causality concept to non-Gaussian time series, we propose a general statistical framework for assessing the causal interactions. In this study, the Granger causality from a time series x 2 to a time series x 1 is assessed based on the relative reduction of the likelihood of x 1 by the exclusion of x 2 compared to the likelihood obtained using all the time series. Simulation results indicated that the proposed algorithm accurately predicted nature of interactions between discrete-valued time series as well as between continuousvalued time series. <s> BIB007 </s> The relation between Granger causality and directed information theory: a review <s> B. Granger causality inference <s> The ability to identify directional interactions that occur among multiple neurons in the brain is crucial to an understanding of how groups of neurons cooperate in order to generate specific brain functions. However, an optimal method of assessing these interactions has not been established. Granger causality has proven to be an effective method for the analysis of the directional interactions between multiple sets of continuous-valued data, but cannot be applied to neural spike train recordings due to their discrete nature. This paper proposes a point process framework that enables Granger causality to be applied to point process data such as neural spike trains. The proposed framework uses the point process likelihood function to relate a neuron's spiking probability to possible covariates, such as its own spiking history and the concurrent activity of simultaneously recorded neurons. Granger causality is assessed based on the relative reduction of the point process likelihood of one neuron obtained excluding one of its covariates compared to the likelihood obtained using all of its covariates. The method was tested on simulated data, and then applied to neural activity recorded from the primary motor cortex (MI) of a Felis catus subject. The interactions present in the simulated data were predicted with a high degree of accuracy, and when applied to the real neural data, the proposed method identified causal relationships between many of the recorded neurons. This paper proposes a novel method that successfully applies Granger causality to point process data, and has the potential to provide unique physiological insights when applied to neural spike trains. <s> BIB008
We consider the practical problem of inferring the graph of dependence between the components of a multivariate process. Let us assume that we have measured a multivariate process x V (n) for n ≤ T . We want to study the DRAFT dependence between each pair of components (Granger causality and instantaneous coupling between any pair of components relative to V ). We can use the result of the preceding section to evaluate the directed information measures on the data. When studying the influence from any subset A to any subset B, if the measures are zero, then there is no causality (or no coupling); if they are strictly positive, then A Granger causes B relative to V (or A and B are coupled relative to V ). This point of view has been adopted in many of the studies that we have already referred to (e.g. BIB003 , BIB004 ,  , BIB005 , BIB006 ), and it relies on estimating the measures from the data. We will not review the estimation problem here. However, it is interesting to examine more traditional frameworks for testing Granger causality, and to examine how directed information theory naturally emerges from these frameworks. To begin with, we show how the measures defined emerge from a binary hypothesis-testing view of Granger causality inference. We then turn to prediction and model-based approaches. We will review how Geweke's measures of Granger causality in the Gaussian case are equivalent to directed information measures. We will then present a more general case adopted by BIB001 , BIB002 , BIB007 , BIB008 , BIB005 and based on a model of the data.