The relation between Granger causality and directed information theory: a review <s> D. Transfer entropy and instantaneous information exchange <s> An information theoretic measure is derived that quantifies the statistical coherence between systems evolving in time. The standard time delayed mutual information fails to distinguish information that is actually exchanged from shared information due to common history and input signals. In our new approach, these influences are excluded by appropriate conditioning of transition probabilities. The resulting transfer entropy is able to distinguish effectively driving and responding elements and to detect asymmetry in the interaction of subsystems. <s> BIB001 </s> The relation between Granger causality and directed information theory: a review <s> D. Transfer entropy and instantaneous information exchange <s> We discuss a recently proposed quantity, called transfer entropy, which uses time series data to measure the amount of information transferred from one process to another. In order to understand its foundation, merits, and limitations, we review some aspects of information theoretic functionals. While for symbol sequences these measures have an intuitive interpretation, their application to continuous state processes and, in particular, their estimation from finite data sets is problematic. For mutual information, finite length scale estimates converge from below and can thus be used to reject the assumption that the observed processes are independent. However, mutual information does not provide any directional information. Conversely, transfer entropy does resolve the directionality of information exchange but no similar monotonic convergence seems to hold. Thus, only in the case of zero transfer entropy in one direction we can reliably infer an asymmetry of the information exchange. <s> BIB002 </s> The relation between Granger causality and directed information theory: a review <s> D. Transfer entropy and instantaneous information exchange <s> We discuss some problems encountered in inference of directionality of coupling, or, in the case of two interacting systems, in inference of causality from bivariate time series. We identify factors and influences that can lead to either decreased test sensitivity or false detections and propose ways to cope with them in order to perform tests with high sensitivity and a low rate of false positive results. <s> BIB003
As introduced by Schreiber in BIB001 , BIB002 , transfer entropy evaluates the deviation of the observed data from a model, assuming the following joint Markov property: This leads to the following definition: (28) is satisfied. Although in the original definition, the past of x in the conditioning might begin at a different time m = n, for practical reasons m = n is considered. Actually, no a priori information is available about possible delays, and setting m = n allows the transfer entropy to be compared with the directed information. By expressing the transfer entropy as a difference of conditional entropies, we get: For l = n = k and choosing 1 as the time origin, the identity I(x, y; z w) = I(x; z w) + I(y; z x, w) leads to: For stationary processes, letting n → ∞ and provided the limits exist, for the rates, we obtain: Transfer entropy is the part of the directed information that measures the influence of the past of x A on the present of x B . However it does not take into account the possible instantaneous dependence of one time series on another, which is handled by directed information. Moreover, as defined by Schreiber in BIB001 , BIB002 , only I(x is considered in T , instead of its sum over i in the directed information. Thus stationarity is implicitly assumed and the transfer entropy has the same meaning as a rate. A sum over delays was considered by Paluš as a means of reducing errors when estimating the measure BIB003 . Summing over n in Eq. (31), the following decomposition of the directed information is obtained: Eq. (33) establishes that the influence of one process on another can be decomposed into two terms that account for the past and for the instantaneous contributions. Moreover, this explains the presence of the term I(x  which is very appealing, as it shows how dependence as measured by mutual information decomposes as the sum of the measures of directional dependences and the measure of instantaneous coupling.