Traditional approaches for the mutual information estimation rely on binning, density and kernel estimation [1]} and k-nearest neighbours [2]}. Nevertheless, they do not scale to problems where high-dimensional data is present as it is the case in modern machine learning applications. Hence, deep neural networks have been recently leveraged to maximize variational lower bounds on the mutual information [3]}, [4]}, [5]}. The expressive power of neural networks has shown promising results in this direction although less is known about the effectiveness of such estimators [6]}, especially since they suffer from either high bias or high variance.
