The relation between Granger causality and directed information theory: a review <s> III. DIRECTED INFORMATION THEORY AND DIRECTIONAL DEPENDENCE <s> A generalization of information theory is presented with the aim of distinguishing the direction of information flow for mutually coupled statistical systems. The bidirectional communication theory refers to two systems. Two directed transinformations are defined which are a measure of the statistical coupling between the systems. Their sum equals Shannon's transinformation. An information flow diagram explains the relation between the directed transinformations and the entropies of the sources. An extension to a group of such systems has also been proposed. The theory is able to describe the informational relationships between living beings and other multivariate complex systems as encountered in economy. An application example referring to group behavior with monkeys is given. <s> BIB001 </s> The relation between Granger causality and directed information theory: a review <s> III. DIRECTED INFORMATION THEORY AND DIRECTIONAL DEPENDENCE <s> It is shown that the "usual definition" of a discrete memoryless channel (DMC) in fact prohibits the use of feedback. The difficulty stems from the confusion of causality and statistical dependence. An adequate definition of a DMC is given, as well as a definition of using a channel without feedback. A definition, closely based on an old idea of Marko, is given for the directed information flowing from one sequence to another. This directed information is used to give a simple proof of the well-known fact that the use of feedback cannot increase the capacity of a DMC. It is shown that, when feedback is present, directed information is a more useful quantity than the traditional mutual information. <s> BIB002 </s> The relation between Granger causality and directed information theory: a review <s> III. DIRECTED INFORMATION THEORY AND DIRECTIONAL DEPENDENCE <s> There is an increasing interest in studying control systems employing multiple sensors and actuators that are geographically distributed. Communication is an important component of these distributed and networked control systems. Hence, there is a need to understand the interactions between the control components and the communication components of the distributed system. In this paper, we formulate a control problem with a communication channel connecting the sensor to the controller. Our task involves designing the channel encoder and channel decoder along with the controller to achieve different control objectives. We provide upper and lower bounds on the channel rate required to achieve these different control objectives. In many cases, these bounds are tight. In doing so, we characterize the "information complexity" of different control objectives. <s> BIB003 </s> The relation between Granger causality and directed information theory: a review <s> III. DIRECTED INFORMATION THEORY AND DIRECTIONAL DEPENDENCE <s> In this work, we consider a source coding model with feed-forward. We analyze a system with a noiseless, feed-forward link where the decoder has knowledge of all previous source samples while reconstructing the present sample. The rate-distortion function for an arbitrary source with feed-forward is derived in terms of directed information, a variant of mutual information. We further investigate the nature of the rate-distortion function with feed-forward for two common types of sources- discrete memory- less sources and Gaussian sources. We then characterize the error exponent for a general source with feed-forward. The results are then extended to feed-forward with an arbitrary delay larger than the block length. <s> BIB004 </s> The relation between Granger causality and directed information theory: a review <s> III. DIRECTED INFORMATION THEORY AND DIRECTIONAL DEPENDENCE <s> A coding theorem is proved for a class of stationary channels with feedback in which the output Y_n = f(X_{n-m}^n, Z_{n-m}^n) is the function of the current and past m symbols from the channel input X_n and the stationary ergodic channel noise Z_n. In particular, it is shown that the feedback capacity is equal to $$ \limp_{n\to\infty} \sup_{p(x^n||y^{n-1})} \frac{1}{n} I(X^n \to Y^n), $$ where I(X^n \to Y^n) = \sum_{i=1}^n I(X^i; Y_i|Y^{i-1}) denotes the Massey directed information from the channel input to the output, and the supremum is taken over all causally conditioned distributions p(x^n||y^{n-1}) = \prod_{i=1}^n p(x_i|x^{i-1},y^{i-1}). The main ideas of the proof are the Shannon strategy for coding with side information and a new elementary coding technique for the given channel model without feedback, which is in a sense dual to Gallager's lossy coding of stationary ergodic sources. A similar approach gives a simple alternative proof of coding theorems for finite state channels by Yang-Kavcic-Tatikonda, Chen-Berger, and Permuter-Weissman-Goldsmith. <s> BIB005 </s> The relation between Granger causality and directed information theory: a review <s> III. DIRECTED INFORMATION THEORY AND DIRECTIONAL DEPENDENCE <s> In this paper, we introduce a general framework for treating channels with memory and feedback. First, we prove a general feedback channel coding theorem based on Massey's concept of directed information. Second, we present coding results for Markov channels. This requires determining appropriate sufficient statistics at the encoder and decoder. We give a recursive characterization of these sufficient statistics. Third, a dynamic programming framework for computing the capacity of Markov channels is presented. Fourth, it is shown that the average cost optimality equation (ACOE) can be viewed as an implicit single-letter characterization of the capacity. Fifth, scenarios with simple sufficient statistics are described. Sixth, error exponents for channels with feedback are presented. <s> BIB006 </s> The relation between Granger causality and directed information theory: a review <s> III. DIRECTED INFORMATION THEORY AND DIRECTIONAL DEPENDENCE <s> We investigate the role of directed information in portfolio theory, data compression, and statistics with causality constraints. In particular, we show that directed information is an upper bound on the increment in growth rates of optimal portfolios in a stock market due to causal side information. This upper bound is tight for gambling in a horse race, which is an extreme case of stock markets. Directed information also characterizes the value of causal side information in instantaneous compression and quantifies the benefit of causal inference in joint compression of two stochastic processes. In hypothesis testing, directed information evaluates the best error exponent for testing whether a random process Y causally influences another process X or not. These results lead to a natural interpretation of directed information I(Yn → Xn) as the amount of information that a random sequence Yn = (Y1,Y2,..., Yn) causally provides about another random sequence Xn = (X1,X2,...,Xn). A new measure, directed lautum information, is also introduced and interpreted in portfolio theory, data compression, and hypothesis testing. <s> BIB007
Directed information theory is a recent extension of information theory, even if its roots go back to the 1960's and 1970's and the studies of Marko BIB001 . The developments began in the late 1990's, after the impetus given by James Massey in 1990 BIB002 . The basic theory was then extended by Gerhard Kramer  , and then further developed by many authors BIB003 , BIB006 , BIB004 , BIB005 , BIB007 to cite a few. We provide here a short review of the essentials of directed information theory. We will, moreover, adopt a presentation close to the spirit of Granger causality to highlight the links between Granger causality and information theory. We begin by recalling some basics from information theory. Then, we describe the information-theoretic approach to study directional dependence between stochastic processes, first in the bivariate case, and then, from section III-E, for networks, i.e., the multivariate case.