The relation between Granger causality and directed information theory: a review <s>  <s> A generalization of information theory is presented with the aim of distinguishing the direction of information flow for mutually coupled statistical systems. The bidirectional communication theory refers to two systems. Two directed transinformations are defined which are a measure of the statistical coupling between the systems. Their sum equals Shannon's transinformation. An information flow diagram explains the relation between the directed transinformations and the entropies of the sources. An extension to a group of such systems has also been proposed. The theory is able to describe the informational relationships between living beings and other multivariate complex systems as encountered in economy. An application example referring to group behavior with monkeys is given. <s> BIB001 </s> The relation between Granger causality and directed information theory: a review <s>  <s> It is shown that the "usual definition" of a discrete memoryless channel (DMC) in fact prohibits the use of feedback. The difficulty stems from the confusion of causality and statistical dependence. An adequate definition of a DMC is given, as well as a definition of using a channel without feedback. A definition, closely based on an old idea of Marko, is given for the directed information flowing from one sequence to another. This directed information is used to give a simple proof of the well-known fact that the use of feedback cannot increase the capacity of a DMC. It is shown that, when feedback is present, directed information is a more useful quantity than the traditional mutual information. <s> BIB002
solar physics, these can be solar indices measured by sensors onboard some satellite; In the study of turbulent fluids, these can be the velocity measured at different scales in the fluid (or can be as in the figure, the wavelet analysis of the velocity at different scales). For these different examples, the aim is to find dependencies between the different measurements, and if possible, to give a direction to the dependence. In neuroscience, this will allow to understand how information flows between different areas of the brain; In solar physics, this will allow to understand the links between indices and their influence on the total solar irradiance received on Earth; In the study of turbulence, this can confirm the directional cascade of energy from large down to small scales. In a graphical modeling approach, each signal is associated to a particular node of a graph, and dependence are represented by edges, directed if a directional dependence exists. The questions addressed in this paper concern the assessment of directional dependence between signals, and thus concern the inference problem of estimating the edge set in the graph of signals considered. Climatology and neuroscience were already given as examples by Norbert Wiener in 1956  , a paper which inspired econometrist Clive Granger to develop what is now termed Granger causality  . Wiener proposed in this paper that a signal x causes another time series y, if the past of x has a strictly positive influence on the quality of prediction of y. Let us quote Wiener  : "As an application of this, let us consider the case where f 1 (α) represents the temperature at 9 A.M. May 5, 2014 DRAFT in Boston and f 2 (α) represents the temperature at the same time in Albany. We generally suppose that weather moves from west to east with the rotation of the earth; the two quantities 1 − C and its correlate in the other direction will enable us to make a precise statement containing some if this content and then verify whether this statement is true or not. Or again, in the study of brain waves we may be able to obtain electroencephalograms more or less corresponding to electrical activity in different part of the brain. Here the study of coefficients of causality running both ways and of their analogues for sets of more than two functions f may be useful in determining what part of the brain is driving what other part of the brain in its normal activity." In a wide sense, Granger causality can be summed up as a theoretical framework based on conditional independence to assess directional dependencies between time series. It is interesting to note that Norbert Wiener influenced Granger causality, as well as another field dedicated to the analysis of dependencies: information theory. Information theory has led to the definition of quantities that measure the uncertainty on variables using probabilistic concepts. Furthermore, this has led to the definition of measures of dependence based on the decrease in uncertainty relating to one variable after observing another one. Usual information theory is, however, symmetrical. For example, the well-known mutual information rate between two stationary time series is symmetrical under an exchange of the two signals: the mutual information assesses the undirectional dependence. Directional dependence analysis viewed as an information-theoretic problem requires the breaking of the usual symmetry of information theory. This was realized in the 1960's and early 1970's by Hans Marko, a German professor of communication. He developed the bidirectional information theory in the Markov case BIB001 . This theory was later generalized by James Massey and Gerhard Kramer , to what we may now call directed information theory BIB002 ,  .