The relation between Granger causality and directed information theory: a review <s> Several interesting simple cases occur: <s> It is shown that the "usual definition" of a discrete memoryless channel (DMC) in fact prohibits the use of feedback. The difficulty stems from the confusion of causality and statistical dependence. An adequate definition of a DMC is given, as well as a definition of using a channel without feedback. A definition, closely based on an old idea of Marko, is given for the directed information flowing from one sequence to another. This directed information is used to give a simple proof of the well-known fact that the use of feedback cannot increase the capacity of a DMC. It is shown that, when feedback is present, directed information is a more useful quantity than the traditional mutual information. <s> BIB001 </s> The relation between Granger causality and directed information theory: a review <s> Several interesting simple cases occur: <s> There is an increasing interest in studying control systems employing multiple sensors and actuators that are geographically distributed. Communication is an important component of these distributed and networked control systems. Hence, there is a need to understand the interactions between the control components and the communication components of the distributed system. In this paper, we formulate a control problem with a communication channel connecting the sensor to the controller. Our task involves designing the channel encoder and channel decoder along with the controller to achieve different control objectives. We provide upper and lower bounds on the channel rate required to achieve these different control objectives. In many cases, these bounds are tight. In doing so, we characterize the "information complexity" of different control objectives. <s> BIB002 </s> The relation between Granger causality and directed information theory: a review <s> Several interesting simple cases occur: <s> Two conservation laws for mutual information in terms of directed informations between two synchronized sequences of random variables are derived, the first for the case of no conditioning and the second for the case of causal conditioning on a third synchronized sequence. As a byproduct of the derivation of the first conservation law, the directed information specifying the feedback flowing from the second sequence to the first sequence is identified, which leads to a simple proof that a previously known sufficient condition for equality of mutual and directed information is also a necessary condition <s> BIB003 </s> The relation between Granger causality and directed information theory: a review <s> Several interesting simple cases occur: <s> In this work, we consider a source coding model with feed-forward. We analyze a system with a noiseless, feed-forward link where the decoder has knowledge of all previous source samples while reconstructing the present sample. The rate-distortion function for an arbitrary source with feed-forward is derived in terms of directed information, a variant of mutual information. We further investigate the nature of the rate-distortion function with feed-forward for two common types of sources- discrete memory- less sources and Gaussian sources. We then characterize the error exponent for a general source with feed-forward. The results are then extended to feed-forward with an arbitrary delay larger than the block length. <s> BIB004 </s> The relation between Granger causality and directed information theory: a review <s> Several interesting simple cases occur: <s> In this paper, we introduce a general framework for treating channels with memory and feedback. First, we prove a general feedback channel coding theorem based on Massey's concept of directed information. Second, we present coding results for Markov channels. This requires determining appropriate sufficient statistics at the encoder and decoder. We give a recursive characterization of these sufficient statistics. Third, a dynamic programming framework for computing the capacity of Markov channels is presented. Fourth, it is shown that the average cost optimality equation (ACOE) can be viewed as an implicit single-letter characterization of the capacity. Fifth, scenarios with simple sufficient statistics are described. Sixth, error exponents for channels with feedback are presented. <s> BIB005
• In the absence of feedback in the link from A to B, there is the following: or equivalently, in terms of entropies, and as a consequence: • Likewise, if there is only a feedback term, then p( and then: • If the link is memoryless, i.e., the output x B does not depend on the past, then: These results allow the question of whether  This quantity measures the loss of information when it is incorrectly assumed that x A does not influence x B . This was called directed information by Massey BIB001 . Expanding the Kullback divergence allows different forms for the directed information to be obtained: where we define the 'causal conditional entropy': Note that causal conditioning might involve more than one process. This leads to the defining of the causal conditional directed information as: The basic properties of the directed information were studied by Massey and Kramer BIB001 , BIB003 ,  , and some are recalled below. As a Kullback divergence, the directed information is always positive or zero. Then, simple algebraic manipulation allows the decomposition to be obtained: Eq. (20) is fundamental, as it shows how mutual information splits into the sum of a feedforward information flow  or equivalently: This situation corresponds to the absence of feedback in the link A → B, whence the fundamental result that the directed information and the mutual information are equal if the channel is free of feedback. This result implies that mutual information over-estimates the directed information between two processes in the presence of feedback. This was thoroughly studied in  , BIB002 , BIB004 , BIB005 , in a communication-theoretic framework. The decomposition of Eq. (20) is surprising, as it shows that the mutual information is not the sum of the directed information flowing in both directions. Instead, the following decomposition holds: where:  which quantifies an instantaneous information exchange between A and B causally conditionally to C.