<jupyter_start><jupyter_text>Feature Selection: Select a subset of input features from the dataset.    Unsupervised: Do not use the target variable (e.g. remove redundant variables).        Correlation    Supervised: Use the target variable (e.g. remove irrelevant variables).        Wrapper: Search for well-performing subsets of features.            RFE        Filter: Select subsets of features based on their relationship with the target.            Statistical Methods            Feature Importance Methods                    Intrinsic: Algorithms that perform automatic feature selection during training.                Decision Trees                Dimensionality Reduction: Project input data into a lower-dimensional feature space.<jupyter_code>Image(url= "https://machinelearningmastery.com/wp-content/uploads/2019/11/How-to-Choose-Feature-Selection-Methods-For-Machine-Learning.png")<jupyter_output><empty_output><jupyter_text>Load Input Data<jupyter_code># Imports
import numpy as np
import pandas as pd

# Visualisation 
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.datasets import make_classification, load_breast_cancer

X, y = load_breast_cancer(return_X_y = True, as_frame=True)
X.head()<jupyter_output><empty_output><jupyter_text>Statistical Tests for Feature Information Student's t-test: 1 independent categorical feature with 2 levels, how does it impact a continuous targetfor continuous variable in relation to 2 groups in categorical variable<jupyter_code># One sample t-test
from scipy import stats

Sample_A = df[df.Sample = "A"]

t_result = stats.ttest_1sample(Sample_A, 65)

alpha = 0.05

if (t_result[1] < alpha)
    print("mean(A) != 65")
# Two sample t-test
Sample_A = df[df.Sample = "A"]
Sample_B = df[df.Sample = "B"]

t_result = stats.ttest_ind(Sample_A, Sample_B)

alpha = 0.05

if (t_result[1] < alpha)
    print("A and B are different!")<jupyter_output><empty_output><jupyter_text>Chi-square: examine proportions of 1 discrete category<jupyter_code>coins = df['Flip'].values_counts()

chi = stats.chisquare(coins)
print(chi)<jupyter_output><empty_output><jupyter_text>Fischer exact test: examine proportions of 2 discrete categories<jupyter_code># Also known as two-sample of chi-square test
# Relate one discrete variable to another

import pandas as pd

table = pd.crosstab(df.Coin, df.Flip)
print(table)

chi = stats.fischer_exact(table, alternative='two-sided')
print(chi[1])
#(correlation coefficient, p-value)<jupyter_output><empty_output><jupyter_text>Pearson test: examine if continous variables are linearly correlated. Based on raw values for linear relationships<jupyter_code>import plotnine as p9
(p9.ggplot(olyAmericans)+ p9.aes(x='Weight', y='Height')+ p9.geom_point())

pearson = stats.pearsonr(df.Weight, df.Height)
print(pearson)<jupyter_output><empty_output><jupyter_text>Confounding variables  Blocking & Randomisation. Paired t-test<jupyter_code>ttest = stats.ttest_rel(yields2018, yields2019)
print(ttest[1])<jupyter_output><empty_output><jupyter_text>One-way ANOVA: 1 independent categorical feature with 3+ levels, how does it impact a continuous target<jupyter_code>anova = stats.f_oneway(array_fertA, array_fertB, array_fearC)
print(anova[1])<jupyter_output><empty_output><jupyter_text>Two-way ANOVA: 2 independent categorical features with 2+ levels, how does it impact a continuous target<jupyter_code>import statsmodels as sm

formula = 'Production ~ Fertilizer + Variety'
model = sm.api.formula.ols(formula, data=df).fit()

aov_table = sm.api.stats.anova_lm(model, typ=2)
print(aov_table)<jupyter_output><empty_output><jupyter_text>ANOVA: Interactive effects<jupyter_code>formula = 'Production ~ Fertilizer + Variety + Fertilizer:Variety'
model = sm.api.formula.ols(formula, data=df).fit()

aov_table = sm.api.stats.anova_lm(model, typ=2)
print(aov_table)<jupyter_output><empty_output><jupyter_text>Sample size, power analysis, effect size<jupyter_code>from statsmodels.stats import. power as pwr

effect_size = 0.8 # large
power = 0.8 # 80% chance of detection
alpha = 0.05 # standard
ratio = (group 2 samples / group 1 samples)
nobs1=nobs1

analysis = pwr.TTestIndPower()

ssresult = analysis.solve_power(
effect_size=effect_size
power=power
alpha=alpha
ratio=ratio
nobs1=None)

print(ssresult)<jupyter_output><empty_output><jupyter_text>Testing for normality: Same mean, median, mode & symmetrical Parametric Quantile-Quantile plot: normality test plot<jupyter_code>from scipy import stats
import plotnine as p9

tq = stats.probplot(countrydata.Life_exp, dist ="norm")

df = pd.DataFrame(data = {'Theorectical Quantiles': tq[0][0],
                          'Ordered Values:' countrydata.Life_exp.sort_values()
                         })

print(p9,ggplot(df) + p9.aes('Theoretical quantiles', 'Ordered Values') + p9.geom_point())<jupyter_output><empty_output><jupyter_text>Parametric Shapiro-Wilk: normality test for small sample sizes<jupyter_code>shapiro = stats.shapiro(countrydata.Life_exp)
print(shapiro)

(test_statistic = 1 for perfect normality, p-value)<jupyter_output><empty_output><jupyter_text>Non-parametric Wilcoxon rank-sum test (version of student's t-test):  Non-parametric Spearman correlation (version of Pearson test): Based on ranks for non-linear relationship<jupyter_code>pearcorr = stats.pearsonr(oly.Height, oly.Weight)
print(pearcorr)

spearcorr = stats.spearsonr(oly.Height, oly.Weight)
print(spearcorr)<jupyter_output><empty_output><jupyter_text>Selecting for Feature Information Remove Low Variance Features<jupyter_code>plt.figure(figsize=(20, 5))
X.boxplot(rot=90);
X_normal = X / X.mean()

plt.figure(figsize=(20, 5))
X_normal.boxplot(rot=90);
plt.figure(figsize=(20, 5))
X_normal.var().plot.bar();
print('Before removing low variance:', X.shape)

sel = VarianceThreshold(threshold=0.005)
sel.fit(X_normal)

mask = sel.get_support()
reduced_X_var = X.loc[:, mask]

print('After removing low variance:', reduced_X_var.shape)<jupyter_output>Before removing low variance: (569, 30)
After removing low variance: (569, 30)<jupyter_text>Remove High Missingness Features<jupyter_code>print('Before removing high missingness:', reduced_X_var.shape, '\n')

mask = reduced_X_var.isna().sum() / len(X) < 0.3
print(mask, '\n')

reduced_X_missing = reduced_X_var.loc[:, mask]

print('After removing high missingness:', reduced_X_missing.shape, '\n')
# reduced_X_missing.head()<jupyter_output>Before removing high missingness: (569, 30) 

mean radius                True
mean texture               True
mean perimeter             True
mean area                  True
mean smoothness            True
mean compactness           True
mean concavity             True
mean concave points        True
mean symmetry              True
mean fractal dimension     True
radius error               True
texture error              True
perimeter error            True
area error                 True
smoothness error           True
compactness error          True
concavity error            True
concave points error       True
symmetry error             True
fractal dimension error    True
worst radius               True
worst texture              True
worst perimeter            True
worst area                 True
worst smoothness           True
worst compactness          True
worst concavity            True
worst concave points       True
worst symmetry             True
worst fractal dimension   [...]<jupyter_text>Remove Highly Correlated Features<jupyter_code># sns.pairplot(reduced_X_missing, hue="target")
corr = reduced_X_missing.corr()
mask = np.triu(np.ones_like(corr, dtype=bool))
cmap = sns.diverging_palette(h_neg=10, h_pos=240, as_cmap=True)

plt.figure(figsize=(20, 20))
sns.heatmap(corr, mask=mask, 
            center=0, cmap=cmap, linewidths=1,
            annot=True, fmt=".2f");
print('Before removing highly correlated features:', reduced_X_missing.shape, '\n')

# Create positive correlation matrix
corr_df = reduced_X_missing.corr().abs()
corr_df.head()
#Create and apply mask
mask = np.triu(np.ones_like(corr_df, dtype=bool))

tri_df = corr_df.mask(mask)
tri_df

to_drop = [c for c in tri_df.columns if any(tri_df[c] > 1.00)]
print('Features to be dropped:', to_drop, '\n')

reduced_X_corr = reduced_X_missing.drop(to_drop, axis = 1)
print('After removing highly correlated features:', reduced_X_corr.shape, '\n')<jupyter_output>Before removing highly correlated features: (569, 30) 

Features to be dropped: [] 

After removing highly correlated features: (569, 30)<jupyter_text>Dependencies & Preprocessing Pipeline before Model Based Feature Selection<jupyter_code># Imports
from numpy import mean
from numpy import std

# Validation
from sklearn.model_selection import train_test_split, StratifiedKFold, RepeatedStratifiedKFold, cross_val_score

# Preprocessing
from sklearn.preprocessing import StandardScaler, MinMaxScaler
from imblearn.over_sampling import SMOTE
from sklearn.decomposition import PCA

# Pipelines
from sklearn.pipeline import Pipeline
from imblearn.pipeline import Pipeline as imbpipeline

# Feature selection
from sklearn.feature_selection import RFE, RFECV
from sklearn.manifold import TSNE

# Classification models
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
# from sklearn.neighbors import KNeighborsClassifier
# from sklearn import svm
from sklearn.svm import SVC
# from xgboost import XGBClassifier

# Regression models
from sklearn.linear_model import LinearRegression
from sklearn.linear_model import Lasso, LassoCV
from sklearn.ensemble import RandomForestRegressor
from sklearn.ensemble import GradientBoostingRegressor

# Evaluation / Scoring metrics
from sklearn.metrics import accuracy_score
# Entire input dataset
X_fs = reduced_X_corr.copy()
X_fs.shape
# Split into train & test
X_train, X_test, y_train, y_test = train_test_split(X_fs,
                                                    y,
                                                    test_size=0.2,
                                                    stratify=y,
                                                    random_state=11)
# Instantiate preprocessing steps
s_scaler = StandardScaler()
mm_scaler = MinMaxScaler()

smote = SMOTE(random_state=11)
pca = PCA(n_components=None)<jupyter_output><empty_output><jupyter_text>Pipeline with Scalerpipeline_scaler = Pipeline([    ('scaler', s_scaler),    ('model', model)]) Pipeline with Scaler & PCApipeline_pca = Pipeline([    ('scaler', s_scaler),    ('pca', pca),    ('model', model)]) Pipeline with Custom Preprocessing for all stepspipeline_full_preprocess = Pipeline([    ('preprocessor', preprocessor)    ('pca', pca),    ('model', model)]) Imbalanced Class Pipeline for SMOTE imbpipeline_smote = imbpipeline([    ('scaler', s_scaler),    ('smote', smote),    ('pca', pca),    ('model', model)]) Imbalance Pipeline for SMOTE with Custom Preprocessing for all stepsimbpipeline_full_preprocess = imbpipeline([    ('preprocessor', preprocessor),    ('smote', smote),    ('pca', pca),    ('model', model)])<jupyter_code># Selecting for Model Accuracy
## Low Model Coefficient Features<jupyter_output><empty_output><jupyter_text>print('Before removing low model coef features:', X_train.shape, '\n')  Model for coefficients of featuresmodel = LogisticRegression(random_state=11) Pipeline with Scalerpipeline_scaler = Pipeline([    ('scaler', s_scaler),    ('model', model)])pipeline_scaler.fit(X_train, y_train)y_pred = pipeline_scaler.predict(X_test)print('Accuracy on Test set:', accuracy_score(y_test, y_pred), '\n')dict_coef = dict(zip(X.columns, abs(model.coef_[0]).round(decimals=3))){k: v for k, v in sorted(dict_coef.items(), key=lambda item: item[1])}  Model for coefficients of featuresmodel = LogisticRegression(random_state=11)print('Before removing low model coef features:', X_train.shape, '\n')drop_low_coef_features = []X_train_reduced_on_coef = X_train.drop(drop_low_coef_features, axis=1)print('After removing low model coef features:', X_train_reduced_on_coef.shape, '\n')      Pipeline with Scalerpipeline_scaler = Pipeline([    ('scaler', s_scaler),    ('model', model)])pipeline_scaler.fit(X_train_reduced_on_coef, y_train)y_pred = pipeline_scaler.predict(X_test)print("Accuracy on Test set:", accuracy_score(y_test, y_pred), "\n")<jupyter_code>## Recursive Feature Elimination
### Visually decide on n_features<jupyter_output><empty_output><jupyter_text>from sklearn.svm import SVCfrom sklearn.datasets import make_classificationfrom yellowbrick.model_selection import RFECV Create a dataset with only 3 informative featuresX, y = make_classification(    n_samples=1000, n_features=25, n_informative=3, n_redundant=2,    n_repeated=0, n_classes=8, n_clusters_per_class=1, random_state=0) Instantiate RFECV visualizer with a linear SVM classifiervisualizer = RFECV(SVC(kernel='linear', C=1))visualizer.fit(X, y)         Fit the data to the visualizervisualizer.show()            Finalize and render the figure from sklearn.ensemble import RandomForestClassifierfrom sklearn.model_selection import StratifiedKFoldfrom yellowbrick.model_selection import RFECVfrom yellowbrick.datasets import load_credit Load classification datasetX, y = load_credit()cv = StratifiedKFold(5)visualizer = RFECV(RandomForestClassifier(), cv=cv, scoring='f1_weighted')visualizer.fit(X, y)         Fit the data to the visualizervisualizer.show()            Finalize and render the figure from sklearn.ensemble import RandomForestClassifierfrom sklearn.model_selection import StratifiedKFoldfrom yellowbrick.model_selection import rfecvfrom yellowbrick.datasets import load_credit Load classification datasetX, y = load_credit()cv = StratifiedKFold(5)visualizer = rfecv(RandomForestClassifier(), X=X, y=y, cv=cv, scoring='f1_weighted') print('Before applying RFE:', X_train.shape, '\n')  Models to wrap in RFElr = LogisticRegression(random_state=11)dt = DecisionTreeClassifier(random_state=11)svc = SVC(kernel="linear", random_state=11) rfecv_estimator = lrstep = 1cv = StratifiedKFold(5)scoring = Nonemin_features_to_select = 1   Minimum number of features to consider Create the RFE object and compute a cross-validated score.rfecv = RFECV(estimator=rfecv_estimator,              step=step,              cv=cv,              scoring=scoring,              min_features_to_select=min_features_to_select) Pipeline with Scalerpipeline_scaler = Pipeline([    ('scaler', s_scaler),    ('fs', rfecv)])pipeline_scaler.fit(X_train, y_train)print("Number of features for largest CV score : %d" % rfecv.n_features_, '\n') Plot number of features VS. cross-validation scoresplt.figure()plt.xlabel("Number of features selected")plt.ylabel("Cross validation score (accuracy)")plt.plot(range(min_features_to_select,               len(rfecv.grid_scores_) + min_features_to_select),         rfecv.grid_scores_)plt.show()y_pred = pipeline_scaler.predict(X_test)print("Accuracy on Test set:", accuracy_score(y_test, y_pred), "\n")rfecv_features_kept = list(X_train.columns[rfecv.support_])print(rfecv_features_kept, '\n') dict_rfecv_rank = dict(zip(X_train.columns, rfecv.ranking_)) {k: v for k, v in sorted(dict_rfecv_rank.items(), key=lambda item: item[1])}<jupyter_code>### Manually set n_features<jupyter_output><empty_output><jupyter_text>rfe_estimator = lrn_features_to_select = 7rfe = RFE(estimator=rfe_estimator, n_features_to_select=n_features_to_select, verbose=0) Pipeline with Scalerpipeline_scaler = Pipeline([    ('scaler', s_scaler),    ('fs', rfe)])pipeline_scaler.fit(X_train, y_train)y_pred = pipeline_scaler.predict(X_test)print('Accuracy on Test set:', accuracy_score(y_test, y_pred), '\n')rfe_features_kept = list(X_train.columns[rfe.support_])print(rfe_features_kept, '\n')dict_rfe_rank = dict(zip(X_train.columns, rfe.ranking_)){k: v for k, v in sorted(dict_rfe_rank.items(), key=lambda item: item[1])}<jupyter_code>### Automatically set n_features<jupyter_output><empty_output><jupyter_text>rfe_estimator = lrmodel = lr create pipelinerfecv = RFECV(estimator=rfe_estimator)pipeline_scaler = Pipeline(steps=[    ('scaler', s_scaler),    ('rfecv', rfecv),    ('m', model)]) evaluate modelcv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)n_scores = cross_val_score(pipeline, X_train, y_train, scoring='accuracy', cv=cv, n_jobs=-1, error_score='raise') report performanceprint('Mean Accuracy (std Accuracy) on Training CV folds : %.3f (%.3f)' % (mean(n_scores), std(n_scores)), '\n') fit RFEpipeline_scaler.fit(X_train, y_train)y_pred = pipeline_scaler.predict(X_test)print('Accuracy on Test set:', accuracy_score(y_test, y_pred), '\n')rfecv_features_kept = list(X_train.columns[rfecv.support_])print('Number of features kept:', len(rfecv_features_kept), '\n')print(rfecv_features_kept) summarize all features for i in range(X_train.shape[1]):     print('Column: %d, Selected %s, Rank: %.3f' % (i, rfecv.support_[i], rfecv.ranking_[i]))<jupyter_code>## Feature Importance with Tree-based feature selection
### Visualise Random Forest feature importance<jupyter_output><empty_output><jupyter_text>print('Before applying Random Forest feature selection:', X_train.shape, '\n') random_forest = RandomForestClassifier(random_state=11)pipeline = Pipeline([    ('model', random_forest)])pipeline.fit(X_train, y_train)y_pred = pipeline.predict(X_test)print("Random Forest Test Accuracy Score:", accuracy_score(y_test, y_pred), '\n')  Plot Random Forest feature importancenames = X_train.columnsprint("Sum of Random Forest Feature Importance:", sum(random_forest.feature_importances_), '\n') print(random_forest.feature_importances_, '\n')random_forest_coef = random_forest.feature_importances_plt.figure(figsize=(20, 5))_ = plt.bar(range(len(names)), random_forest_coef)_ = plt.xticks(range(len(names)), names, rotation=90)_ = plt.ylabel('Coefficients'); mask = random_forest.feature_importances_ > 0.1 print(mask, '\n')X_train_reduced_by_random_forest = X_train.loc[:, mask]list(X_train_reduced_by_random_forest.columns)<jupyter_code>### RFE Random Forest<jupyter_output><empty_output><jupyter_text>print('Before applying RFE with Random Forest feature selection:', X_train.shape, '\n') rfe_estimator = random_forestn_features_to_select = 5step=2rfe_random_forest = RFE(estimator=rfe_estimator, n_features_to_select=n_features_to_select, step=step, verbose=0)pipeline = Pipeline([    ('fs', rfe_random_forest)])pipeline.fit(X_train, y_train)y_pred = rfe_random_forest.predict(X_test)print("Random Forest Test Accuracy Score:", accuracy_score(y_test, y_pred), '\n')rfe_random_forest_features_kept = list(X_train.columns[rfe_random_forest.support_])print(rfe_random_forest_features_kept, '\n')dict_rfe_random_forest_rank = dict(zip(X_train.columns, rfe_random_forest.ranking_)){k: v for k, v in sorted(dict_rfe_random_forest_rank.items(), key=lambda item: item[1])}<jupyter_code>## Feature Selection with Lasso Regularization (for Linear Regression)
### Linear Regression<jupyter_output><empty_output><jupyter_text>print('Before applying Regularised Lasso Regression feature selection:', X_train.shape, '\n') lr = LinearRegression()pipeline_scaler = Pipeline(steps=[    ('scaler', s_scaler),    ('fs', lr)])pipeline_scaler.fit(X_train, y_train)print(lr.coef_, '\n') print(lr.intercept_) Calculate R squared on Linear Regressiony_pred = pipeline_scaler.predict(X_test)print("Linear Regression Test Accuracy Score:", pipeline_scaler.score(X_test, y_test))<jupyter_code>### Regularised Lasso Regression: Automated Alpha<jupyter_output><empty_output><jupyter_text>print('Before applying Regularised Lasso Regression:', X_train.shape, '\n')  Automate selection of alpha for regularisationlasso_cv = LassoCV()pipeline_scaler = Pipeline(steps=[    ('scaler', s_scaler),    ('fs', lasso_cv)])pipeline_scaler.fit(X_train, y_train)print(lasso_cv.alpha_) Calculate R squared on Regularised Lasso Regressiony_pred = pipeline_scaler.predict(X_test)print("Lasso CV Regression Test Accuracy Score:", pipeline_scaler.score(X_test, y_test), '\n') mask = lasso_cv.coef_ != 0 print(mask)X_train_reduced_lasso_cv = X_train.loc[:, mask]print('After applying Regularised Lasso CV Regression feature selection:', X_train_reduced_lasso_cv.shape, '\n')X_train_reduced_lasso_cv.head()<jupyter_code>### Regularised Lasso Regression: Manual Alpha<jupyter_output><empty_output><jupyter_text>alpha = 0.01lasso = Lasso(alpha=alpha)pipeline_scaler = Pipeline(steps=[    ('scaler', s_scaler),    ('fs', lasso)])pipeline_scaler.fit(X_train, y_train) print(lasso.coef_) print(lasso.intercept_) Calculate R squared on Regularised Lasso Regressiony_pred = pipeline_scaler.predict(X_test)print("Lasso Regression Test Accuracy Score:", pipeline_scaler.score(X_test, y_test), '\n')names = X_train.columnslasso_coef = lasso.coef_plt.figure(figsize=(20, 5))_ = plt.bar(range(len(names)), lasso_coef)_ = plt.xticks(range(len(names)), names, rotation=90)_ = plt.ylabel('Coefficients'); mask = lasso.coef_ != 0 print(mask)X_train_reduced_lasso = X_train.loc[:, mask]print('After applying Regularised Lasso Regression feature selection:', X_train_reduced_lasso.shape, '\n')X_train_reduced_lasso.head()list(X_train_reduced_lasso.columns)<jupyter_code>## Feature Selection Voting (by Combining Model Feature Selectors)<jupyter_output><empty_output><jupyter_text>print('Before applying feature selection combining & voting:', X_train.shape, '\n')  Create mask for Lasso CV feature selectionlasso_cv = LassoCV()pipeline_scaler = Pipeline(steps=[    ('scaler', s_scaler),    ('fs', lasso_cv)])pipeline_scaler.fit(X_train, y_train)print(pipeline_scaler.score(X_test, y_test))lasso_cv_mask = lasso_cv.coef_ != 0sum(lasso_cv_mask)  Create mask for RFE with Random Forest Regressor feature selectionrfe_estimator = RandomForestRegressor()n_features_to_select=5step=2rfe_random_forest = RFE(estimator=rfe_estimator, n_features_to_select=n_features_to_select, step=step, verbose=0)pipeline = Pipeline(steps=[    ('fs', rfe_random_forest)])pipeline.fit(X_train, y_train)print(pipeline.score(X_test, y_test))rfe_random_forest_mask = rfe_random_forest.support_  Create mask for RFE with Gradient Boosting Regressor feature selectionrfe_estimator = GradientBoostingRegressor()rfe_gb = RFE(estimator=rfe_estimator, n_features_to_select=n_features_to_select, step=step, verbose=0)pipeline = Pipeline(steps=[    ('fs', rfe_gb)])pipeline.fit(X_train, y_train)print(pipeline.score(X_test, y_test))rf_gb_mask = rfe_gb.support_  Combining the feature selectors & vote for features to keepimport numpy as npvotes = np.sum([lasso_cv_mask, rf_mask, gb_mask], axis=0)print(votes)mask = votes >= 2X_train_combining_fs = X_train.loc[:, mask]<jupyter_code># Principle Components Analysis (PCA)
## PCA exploration for 'n_components'<jupyter_output><empty_output><jupyter_text>X_train_pca_exploration = X_train.copy()print('Before PCA exploration:', X_train_pca_exploration.shape, '\n')  Fit Scaler & PCA in Pipeline in one gopca = PCA(n_components=None)pipeline_pca = Pipeline([    ('scaler', StandardScaler()),    ('reducer', pca)])pipeline_pca.fit_transform(X_train)print(pca.explained_variance_ratio_.round(decimals=3)) print(pca.explained_variance_ratio_.round(decimals=2).cumsum()) print(pca.components_) principle_components = pipeline_pca.fit_transform(X_train) print(principle_components[:2,])X_train_pca_exploration['PC_1'] = principle_components[:,0]X_train_pca_exploration['PC_2'] = principle_components[:,1]X_train_pca_exploration['PC_3'] = principle_components[:,2]X_train_pca_exploration['PC_4'] = principle_components[:,3]X_train_pca_exploration['PC_5'] = principle_components[:,4]X_train_pca_exploration.head()<jupyter_code># Visualise categories with respect to Principle Components
sns.scatterplot(data=X_train_pca_exploration, 
                x='PC 1', y='PC 2', 
                hue='CATEGORY', alpha=0.4)<jupyter_output><empty_output><jupyter_text>Manually set 'n_components' for model<jupyter_code># Decide on number of principle components to use and pipeline with model
print('Before applying PCA for dimensionality reduction:', X_train.shape, '\n')
### Decide on number of principle components to use and pipeline with model
pca = PCA(n_components=5)
model = RandomForestClassifier(random_state=11)
###

pipeline_pca = Pipeline([
    ('scaler', StandardScaler()),
    ('reducer', pca),
    ('model', model)
])

pipeline_pca.fit(X_train, y_train)
print(pipeline_pca.steps[1])
print(pipeline_pca.steps[1][1].explained_variance_ratio_.cumsum(), '\n')
print(pipeline_pca.score(X_test, y_test))<jupyter_output>[0.44807772 0.63737149 0.73138349 0.79725457 0.85115553] 

0.9385964912280702<jupyter_text>Visualise optimal 'n_components'<jupyter_code>print('Before PCA exploration:', X_train.shape, '\n')
### An optimal number of components
pca = PCA(n_components=0.9)
###

pipeline_pca = Pipeline([
    ('scaler', StandardScaler()),
    ('reducer', pca)
])

pipeline_pca.fit(X_train)
print("Number of components for 90% explained variance:", len(pipeline_pca.steps[1][1].components_), '\n')

# Pick the elbow for approx optimal number of components
var = pipeline_pca.steps[1][1].explained_variance_ratio_

plt.plot(var)
plt.xlabel('Principle component index')
plt.ylabel('Explained variance ratio');<jupyter_output>Number of components for 90% explained variance: 7<jupyter_text>t-SNE check<jupyter_code>X_tsne = X_train.copy()
print('Before X_tnse:', X_tsne.shape, '\n')
non_numeric = []
X_tsne_numeric = X_tsne.drop(non_numeric, axis=1)
print('After removing non numeric for TSNE:', X_tsne_numeric.shape, '\n')

m = TSNE(learning_rate=50)

tsne_features = m.fit_transform(X_tsne_numeric)
tsne_features[1:4,:]

X_tsne['x'] = tsne_features[:,0]
X_tsne['y'] = tsne_features[:,1]

X_tsne.head()
# Plot 2d tsne against any category (if available)
sns.scatterplot(x='x', y='x', data=X_tsne, 
#                 hue='CATEGORY'
               );<jupyter_output><empty_output><jupyter_text>Feature Engineering<jupyter_code>http://contrib.scikit-learn.org/category_encoders/index.html
https://feature-engine.readthedocs.io/en/latest/quickstart.html
https://towardsdatascience.com/the-art-of-finding-the-best-features-for-machine-learning-a9074e2ca60d
https://www.scikit-yb.org/en/latest/<jupyter_output><empty_output><jupyter_text>Feature Engineering Checklist<jupyter_code>https://elitedatascience.com/feature-engineering-best-practices
    
What is Not Feature Engineering?
That means there are certain steps we do not consider to be feature engineering:

We do not consider initial data collection to be feature engineering.
Similarly, we do not consider creating the target variable to be feature engineering.
We do not consider removing duplicates, handling missing values, or fixing mislabeled classes to be feature engineering. We put these under data cleaning.
We do not consider scaling or normalization to be feature engineering because these steps belong inside the cross-validation loop (i.e. after you’ve already built your analytical base table).
Finally, we do not consider feature selection or PCA to be feature engineering. These steps also belong inside your cross-validation loop.
Again, this is simply our categorization. Reasonable data scientists may disagree, and that’s perfectly fine.

With those disclaimers out of the way, let’s dive into the best practices and heuristics!

Indicator Variables
    Indicator variable from thresholds
    Indicator variable from multiple features
    Indicator variable for special events
    Indicator variable for groups of classes
    
Interaction Features
    Sum of two features
    Difference between two features
    Product of two features
    Quotient of two features
    
Feature Representation
    Date and time features
    Numeric to categorical mappings
    Grouping sparse classes
    Creating dummy variables

External Data
    Time series data
    External API’s
    Geocoding
    Other sources of the same data
    
Error Analysis (Post-Modeling)
    Start with larger errors
    Segment by classes
    Unsupervised clustering
    Ask colleagues or domain experts
https://towardsdatascience.com/feature-engineering-in-python-part-i-the-most-powerful-way-of-dealing-with-data-8e2447e7c69e
    
Feature Selection

Feature Transformation

Feature Extraction
Creating New Features
This is the hardest section and requires the most critical thinking, in my opinion. 
There isn’t any code I can give that will apply to lots of projects — it really depends on the dataset.
Here are a couple of cases where you may want to try creating a new feature:
You suspect that the relationship of an outcome and a feature depends on a second feature → Create an interaction variable
You want to create linear relationships → Create quadratic or higher level functions
You can think of variables/information that is missing from your dataset → Create this variable using a function of variables you do have
Iterative Process of Feature Engineering
The process might look as follows:

Brainstorm features: Really get into the problem, look at a lot of data, study feature engineering on other problems and see what you can steal.
Devise features: Depends on your problem, but you may use automatic feature extraction, manual feature construction and mixtures of the two.
Select features: Use different feature importance scorings and feature selection methods to prepare one or more “views” for your models to operate upon.
Evaluate models: Estimate model accuracy on unseen data using the chosen features.
https://adataanalyst.com/machine-learning/comprehensive-guide-feature-engineering/

Representing timestamps
Missing Value Treatment
Create new ratios and proportions
Binning/Bucketing
Reframe Numerical Quantities
Variable Transformation
Outlier Detection and Treatment
Decomposing Categorical Attributes
Encode all the categorical features in Python
One Hot Encoding in Python
Feature Crosses
Feature Importance:<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1011690046760187, 'fraction_numerical': 0.010988643954575819, 'mean_word_length': 5.848353156450138, 'pattern_counts': {'":': 0, '<': 139, '<?xml version=': 0, '>': 139, 'https://': 7, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '35', 'dir': 'jupyter-structured-clean-dedup', 'id': '425700', 'license': 'MIT', 'n_tokens_mistral': 9935, 'n_tokens_neox': 9450, 'n_words': 2367, 'path': 'notebooks/machine_learning_algorithms/6b-Feature-Selection-or-Extraction-Model-Methods.ipynb', 'repo_name': 'dilip-albert-datascientist/traditional-machine-learning', 'size': '29940'}