<jupyter_start><jupyter_text>Feature Selection* Selecting features from the dataset* Improve estimator's accuracy* Boost preformance for high dimensional datsets<jupyter_code>from sklearn import feature_selection
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>VarianceThreshold* Drop the columns whose variance is below configured level* This method is unsupervised .i.e target not taken into action* Intution : Columns whose values arn't petty much the same won't have much impact on target<jupyter_code>df = pd.DataFrame({'A':['m','f','m','m','m','m','m','m'], 
              'B':[1,2,3,1,2,1,1,1], 
              'C':[1,2,3,1,2,1,1,1]})
df
from sklearn.preprocessing import LabelEncoder
le = LabelEncoder()
df['A'] = le.fit_transform(df.A)
df
vt = feature_selection.VarianceThreshold(threshold=.2)
vt.fit_transform(df)
vt.variances_<jupyter_output><empty_output><jupyter_text>Chi-Square for Non-negative feature & class* Feature data should be booleans or count* Supervised technique for feature selection* Target should be discrete* Higher chi value means more important feature for target<jupyter_code>df = pd.read_csv('datasets/tennis.csv')
df.head()
for col in df.columns:
    le = LabelEncoder()
    df[col] = le.fit_transform(df[col])
df
df.drop('play',axis=1)
chi2, pval = feature_selection.chi2(df.drop('play',axis=1),df.play)
chi2
pval<jupyter_output><empty_output><jupyter_text>4. ANOVA using f_classif* For feature variables continues in nature* And, target variable discrete in nature* Internally, this method uses ratio of variation within a columns & variation across columns<jupyter_code>from sklearn.datasets import load_breast_cancer
cancer_data = load_breast_cancer()
X = cancer_data.data
Y = cancer_data.target
print(X.shape)
F, pval = feature_selection.f_classif(X,Y)
print(pval)
print(F)<jupyter_output>[6.46981021e+02 1.18096059e+02 6.97235272e+02 5.73060747e+02
 8.36511234e+01 3.13233079e+02 5.33793126e+02 8.61676020e+02
 6.95274435e+01 9.34592949e-02 2.68840327e+02 3.90947023e-02
 2.53897392e+02 2.43651586e+02 2.55796780e+00 5.32473391e+01
 3.90144816e+01 1.13262760e+02 2.41174067e-02 3.46827476e+00
 8.60781707e+02 1.49596905e+02 8.97944219e+02 6.61600206e+02
 1.22472880e+02 3.04341063e+02 4.36691939e+02 9.64385393e+02
 1.18860232e+02 6.64439606e+01]<jupyter_text>* Each value represents importance of a feature  Univariate Regression Test using f_regression* Linear model for testing the individual effect of each of many regressors.* Correlation between each value & target is calculated* F-test captures linear dependency<jupyter_code>from sklearn.datasets import california_housing
house_data = california_housing.fetch_california_housing()
X,Y = house_data.data, house_data.target
print(X.shape,Y.shape)
F, pval = feature_selection.f_regression(X,Y)
F<jupyter_output><empty_output><jupyter_text>* Columns with top F values are the selected features   F score verses Mutual Information<jupyter_code>np.random.seed(0)
X = np.random.rand(1000, 3)
y = X[:, 0] + np.sin(6 * np.pi * X[:, 1]) + 0.1 * np.random.randn(1000)
plt.scatter(X[:,0],y,s=10)
plt.scatter(X[:,1],y,s=10)
F, pval = feature_selection.f_regression(X,y)
print(F)<jupyter_output>[187.42118421  52.52357392   0.47268298]<jupyter_text>Mutual Information for regression using mutual_info_regression* Returns dependency in the scale of 0 & 1 among feature & target* Captures any kind of dependency even if non-linear* Target is continues in nature<jupyter_code>feature_selection.mutual_info_regression(X,y)<jupyter_output><empty_output><jupyter_text>Mutual Information for classification using mutual_info_classification* Returns dependency in the scale of 0 & 1 among feature & target* Captures any kind of dependency even if non-linear* Target is discrete in nature<jupyter_code>cols = ['age','workclass','fnlwgt','education','education-num','marital-status','occupation','relationship'
        ,'race','sex','capital-gain','capital-loss','hours-per-week','native-country','Salary']
adult_data = pd.read_csv('https://raw.githubusercontent.com/zekelabs/data-science-complete-tutorial/master/Data/adult.data.txt', names=cols)
adult_data.head()
cat_cols = list(adult_data.select_dtypes('object').columns)
cat_cols.remove('Salary')
len(cat_cols)
from sklearn.preprocessing import LabelEncoder
for col in cat_cols:
    le = LabelEncoder()
    adult_data[col]  = le.fit_transform(adult_data[col])
X = adult_data.drop(columns=['Salary'])
y = le.fit_transform(adult_data.Salary)
firep = feature_selection.mutual_info_classif(X, y)
X.columns
X.columns[np.argsort(firep)[::-1]]<jupyter_output><empty_output><jupyter_text>SelectKBest* SelectKBest returns K important features based on above techniques* Based on configuration, it can use mutual_information or ANOVA or regression based techniques<jupyter_code>adult_data.head
adult_data.shape
selector = feature_selection.SelectKBest(k=7, score_func=feature_selection.f_classif)
data = selector.fit_transform(adult_data.drop('Salary',axis=1),adult_data.Salary)
data.shape
selector.scores_
selector = feature_selection.SelectKBest(k=7, score_func=feature_selection.mutual_info_classif)
data = selector.fit_transform(adult_data.drop('Salary',axis=1),adult_data.Salary)
data.shape
selector.scores_<jupyter_output><empty_output><jupyter_text>SelectPercentile* Selecting top features whose importances are in configured parameter* Default is top 10 percentile<jupyter_code>selector = feature_selection.SelectPercentile(percentile=20, score_func=feature_selection.mutual_info_classif)
data = selector.fit_transform(adult_data.drop('Salary',axis=1),adult_data.Salary)
data.shape<jupyter_output><empty_output><jupyter_text>SelectFromModel* Selecting important features from model weights* The estimator should support 'feature_importances'<jupyter_code>from sklearn.datasets import load_boston
boston = load_boston()
boston.data.shape
from sklearn.linear_model import LinearRegression
clf = LinearRegression()
sfm = feature_selection.SelectFromModel(clf, threshold=0.25)
clf = LinearRegression()
sfm.fit_transform(boston.data, boston.target).shape<jupyter_output><empty_output><jupyter_text>Recursive Feature Elimination* Uses an external estimator to calculate weights of features* First, the estimator is trained on the initial set of features and the importance of each feature is obtained either through a coef_ attribute or through a feature_importances_ attribute. * Then, the least important features are pruned from current set of features. * That procedure is recursively repeated on the pruned set until the desired number of features to select is eventually reached.<jupyter_code>from sklearn.datasets import make_regression
from sklearn.feature_selection import RFE
from sklearn.svm import SVR
X, y = make_regression(n_samples=50, n_features=10, random_state=0)
estimator = SVR(kernel="linear")
selector = RFE(estimator, 5, step=1)
data = selector.fit_transform(X, y)
X.shape
data.shape
selector.ranking_<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10351175304446332, 'fraction_numerical': 0.06060606060606061, 'mean_word_length': 7.319199057714958, 'pattern_counts': {'":': 0, '<': 47, '<?xml version=': 0, '>': 47, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '396105', 'license': 'MIT', 'n_tokens_mistral': 2635, 'n_tokens_neox': 2331, 'n_words': 559, 'path': 'SII/ML/2-Feature_Selection.ipynb', 'repo_name': 'vafaei-ar/alzahra-workshop-2019', 'size': '7062'}