Mutual Information (MI) measures the degree of statistical dependence between random variables. Given random variables \(A\)  and \(B\) , the MI between them, \(I(A, B)\) , can be understood as how much knowing \(A\)  will reduce the uncertainty of \(B\)  or vice versa. For distributions that do not have analytical forms, maximizing MI directly is often intractable.
To overcome this difficulty, recent work resorts to variational bounds [1]}, [2]}, [3]} and deep learning [4]} to estimate MI. These works utilize flexible parametric distributions or critics that are parameterized neural networks (NNs) to approximate unknown densities that appear in MI calcuations. [5]} provides a review of existing MI estimators and proposes novel bounds that trade-off bias and variance. [6]} unified language representation learning objective functions from the MI maximization perspective.
