The process of feature selection aims to select a small subset of relevant features while still constructing models with sufficient or even optimal predictive performance.
There are two distinct model-agnostic approaches to feature selection [1]}: Filters and wrappers.
Filters use proxy measures to rank features by their estimated explanatory power, independently of the learning algorithm being employed.
These include information theoretic measures, correlation measures, distance measures or consistency measures [2]}.
In contrast, wrappers [3]} optimize model test-set performance directly over the space of feature subsets. Because every feature subset evaluation requires either one or multiple model fits, exhaustive search is usually infeasible, and a black-box discrete optimization search strategy is necessary. Commonly used are simple greedy methods like forward or backward search. More advanced methods like evolutionary algorithms can improve upon this [4]}.
Because they directly optimize learner performance, wrappers often yield better results [4]}.
