Several particularly promising approaches for filter methods rely on information theory. Mutual information maximization (a notion explained in the paper) has been extensively used to eliminate redundant inputs among large sets of inputs. For instance, information gain is a standard tool for text classification [1]}, where mutual information between a single feature and a class
variable estimates the relevance of each feature one by one. Along the same line, [2]} developed the mutual information feature selection (MIFS) algorithm for classification problems that
search greedily a set of features with high-mutual information with class labels but low mutual information among chosen features. These approaches allow for the elimination of redundant information within features. Among other works, one may cite [3]}, which used mutual information between class variables to form optimal linear combinations of inputs. However, as mentioned by [4]}, these methods, which deal with continuous variables in classification tasks, demand large amounts of data and high-computational complexity.
