In this paper we examine the application of the random forest classifier for the all relevant feature selection problem. To this end we first examine two recently proposed all relevant feature selection algorithms, both being a random forest wrappers, on a series of synthetic data sets with varying size. We show that reasonable accuracy of predictions can be achieved and that heuristic algorithms that were designed to handle the all relevant problem, have performance that is close to that of the reference ideal algorithm. Then, we apply one of the algorithms to four families of semi-synthetic data sets to assess how the properties of particular data set influence results of feature selection. Finally we test the procedure using a well-known gene expression data set. The relevance of nearly all previously established important genes was confirmed, moreover the relevance of several new ones is discovered.
The algorithm of Rogers & Gunn @cite_26 uses internals of construction for feature selection. It relies on a theoretical model giving an estimate of the information gain of a split done on a non-informative attribute, which, averaged over the forest, is used to test the relevance of original attributes. This method, while elegant, is not particularly good at discerning between relevant and irrelevant features. Authors present the results for the Madalon problem, where 130 features where deemed relevant by this algorithm at confidence level @math , whereas there are only 20 relevant features in this set. For a comparison, this problem is solved nearly perfectly by Boruta algorithm @cite_14 .
@cite_14 @cite_26
2156665896 1536061269
This article describes a R package Boruta, implementing a novel feature selection algorithm for finding emph all relevant variables . The algorithm is designed as a wrapper around a Random Forest classification algorithm. It iteratively removes the features which are proved by a statistical test to be less relevant than random probes. The Boruta package provides a convenient interface to the algorithm. The short description of the algorithm and examples of its application are presented. It is known that feature selection and feature relevance can benefit the performance and interpretation of machine learning algorithms. Here we consider feature selection within a Random Forest framework. A feature selection technique is introduced that combines hypothesis testing with an approximation to the expected performance of an irrelevant feature during Random Forest construction. It is demonstrated that the lack of implicit feature selection within Random Forest has an adverse effect on the accuracy and efficiency of the algorithm. It is also shown that irrelevant features can slow the rate of error convergence and a theoretical justification of this effect is given.