<jupyter_start><jupyter_text>Feature SelectionThe classes in the sklearn.feature_selection module can be used for feature selection/dimensionality reduction on sample sets, either to improve estimators’ accuracy scores or to boost their performance on very high-dimensional datasets.  Remove Low Var FeaturesVarianceThreshold is a simple baseline approach to feature selection. It removes all features whose variance doesn’t meet some threshold. By default, it removes all zero-variance features, i.e. features that have the same value in all samples.Again we are starting to see fit and fit_transform pop up again. Sklearn provides a ton of functionality that's not just prediction. Some of the functionality is preprocessing the data. Again these are like models (they can only rely on the training data) but don't really predict anything. Thus they do have a fit method, but don't have a predict method. We will see two examples of this type of paradigm below.<jupyter_code>from sklearn.feature_selection import VarianceThreshold

X = [[0, 0, 1], [0, 1, 0], [1, 0, 0], [0, 1, 1], [0, 1, 0], [0, 1, 1]]

sel = VarianceThreshold(threshold=(.8 * (1 - .8)))

sel.fit(X)

sel.transform(X)<jupyter_output><empty_output><jupyter_text>Univariate Feature SelectionUnivariate feature selection works by selecting the best features based on univariate statistical tests. It can be seen as a preprocessing step to an estimator. Scikit-learn exposes feature selection routines as objects that implement the transform method:* SelectKBest removes all but the k highest scoring features* SelectPercentile removes all but a user-specified highest scoring percentage of features* using common univariate statistical tests for each feature: false positive rate SelectFpr, false discovery rate SelectFdr, or family wise error SelectFwe.* GenericUnivariateSelect allows to perform univariate feature selection with a configurable strategy. This allows to select the best univariate selection strategy with hyper-parameter search estimator.These objects take as input a scoring function that returns univariate scores and p-values (or only scores for SelectKBest and SelectPercentile):* For regression: f_regression, mutual_info_regression* For classification: chi2, f_classif, mutual_info_classifThe methods based on F-test estimate the degree of linear dependency between two random variables. On the other hand, mutual information methods can capture any kind of statistical dependency, but being nonparametric, they require more samples for accurate estimation.<jupyter_code>from sklearn.datasets import load_iris
from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import chi2

SelectKBest?
X, y = load_iris(return_X_y=True)

sel = SelectKBest(chi2, k=2)

sel.fit(X, y)
sel.transform(X).shape
sel.scores_<jupyter_output><empty_output><jupyter_text>Recursive feature eliminationGiven an external estimator that assigns weights to features (e.g., the coefficients of a linear model), recursive feature elimination (RFE) is to select features by recursively considering smaller and smaller sets of features. First, the estimator is trained on the initial set of features and weights are assigned to each one of them. Then, features whose absolute weights are the smallest are pruned from the current set features. That procedure is recursively repeated on the pruned set until the desired number of features to select is eventually reached.So it is very important to normalize these features in linear models!<jupyter_code>from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_selection import RFECV

RFECV?
m = RFECV(RandomForestClassifier(), scoring='accuracy')
m.fit(X, y)<jupyter_output><empty_output><jupyter_text>Feature selection using SelectFromModelSelectFromModel is a meta-transformer that can be used along with any estimator that has a coef_ or feature_importances_ attribute after fitting. The features are considered unimportant and removed, if the corresponding coef_ or feature_importances_ values are below the provided threshold parameter. Apart from specifying the threshold numerically, there are built-in heuristics for finding a threshold using a string argument. Available heuristics are “mean”, “median” and float multiples of these like “0.1*mean”.For examples on how it is to be used refer to the sections below.<jupyter_code>from sklearn.svm import LinearSVC
from sklearn.feature_selection import SelectFromModel

SelectFromModel?
m = SelectFromModel(LinearSVC(C=0.01, penalty='l1', dual=False))

m.fit(X, y)
m.transform(X).shape<jupyter_output><empty_output><jupyter_text>A little bit more complex!<jupyter_code>from sklearn.linear_model import LassoCV
from sklearn.datasets import load_boston

X, y = load_boston(return_X_y=True)

print X.shape

m = SelectFromModel(LassoCV())

m.fit(X, y)

m.transform(X).shape<jupyter_output>(506, 13)
{'fraction_non_alphanumeric': 0.05553287055941201, 'fraction_numerical': 0.007349938750510413, 'mean_word_length': 6.0590778097982705, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '594546', 'license': 'MIT', 'n_tokens_mistral': 1312, 'n_tokens_neox': 1237, 'n_words': 598, 'path': 'notebooks/.ipynb_checkpoints/FeatureSelection-checkpoint.ipynb', 'repo_name': 'knathanieltucker/bit-of-data-science-and-scikit-learn', 'size': '4898'}