<jupyter_start><jupyter_text>**Feature selection by a cross-validation using random forest classifier**<jupyter_code>from util import *
import pickle
import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import RandomizedSearchCV
from sklearn.model_selection import train_test_split
from sklearn.model_selection import StratifiedKFold
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.compose import ColumnTransformer
import seaborn as sns
import matplotlib.pyplot as plt
import warnings
import sklearn.naive_bayes as nb
from sklearn.metrics import confusion_matrix
from sklearn.metrics import precision_score, recall_score, precision_recall_curve,f1_score, fbeta_score
from sklearn.metrics import roc_auc_score, roc_curve
from sklearn.inspection import permutation_importance
warnings.filterwarnings('ignore')
pd.options.display.float_format = '{:,.2f}'.format
pd.set_option("display.max_rows", 999)
pd.set_option("display.max_columns", None)
pd.set_option("precision", 3)
sns.set_context("talk") # talk context is good for RISE presentations
%pylab inline
%config InlineBackend.figure_formats = ['retina']<jupyter_output>Populating the interactive namespace from numpy and matplotlib<jupyter_text>**1. Load clean data set**<jupyter_code>q = """
SELECT * from ma_refinance_new
"""
df = do(q)
df.head(3)<jupyter_output><empty_output><jupyter_text>**1.1 Separate categorical and continuous variables to facilitate OneHotEncoding incross validation**<jupyter_code>x_cat = gen_cat(df)
x_cont = gen_cont(df)
y = df[['action_taken']]<jupyter_output><empty_output><jupyter_text>**2. Run randomized search cross-validation** Randomized search cv gives similar results to that from grid search cv, but it leads to drastically low computation cost.   Refs: [sklearn doc](https://scikit-learn.org/stable/auto_examples/model_selection/plot_randomized_search.html) [Hyperparameter Tuning the Random Forest in Python](https://towardsdatascience.com/hyperparameter-tuning-the-random-forest-in-python-using-scikit-learn-28d2aa77dd74)  **2.1. Split the data** Split the data into training and testing datasets (70% vs. 30%). Also, split testing dataset into validation and testing datasets (30% vs 70%). The validation dataset is used to generate permutation feature importances<jupyter_code>categorical_columns = list(x_cat.columns)
numerical_columns = list(x_cont.columns)
x = df[categorical_columns + numerical_columns]
#-------------------------------------------------------------------------
testing_portion = 0.3
x_train, x_test, y_train, y_test = train_test_split(
    x, y, stratify=y, random_state=15, test_size=testing_portion)
#-------------------------------------------------------------------------
validation_portion = 0.3
x_test, x_val, y_test, y_val = train_test_split(
    x_test, y_test, stratify=y_test, random_state=15, test_size=validation_portion)

data_dict = make_data_dict(x_train, y_train,
                           x_test, y_test,
                           x_val, y_val,
                           )
fn = '../data/data_vf.pickle'
save_as_pickle(fn, data_dict)<jupyter_output>Saved data to "../data/data_vf.pickle"<jupyter_text>**2.2 Run randomized search cross-validation using random forest classifier**<jupyter_code>random_grid = gen_hypergrid_for_rf_cv()
base_mdl = RandomForestClassifier()
rf_random = make_cv_pipelinie(base_mdl,
                              categorical_columns,
                              numerical_columns,
                              random_grid,
                              scoring='roc_auc'
                             )
rf_random.fit(x_train, y_train)

# save the pipeline model
fn = '../data/rf_pipeline.pickle'
save_as_pickle(fn, rf_random)<jupyter_output>{'classifier__n_estimators': [100, 200, 300, 400, 500], 'classifier__max_features': ['auto', 'sqrt', 'log2'], 'classifier__max_depth': [10, 20, 30, 40, 50, None], 'classifier__min_samples_split': [2, 5, 10, 15, 20], 'classifier__min_samples_leaf': [1, 2, 4], 'classifier__bootstrap': [True, False]}
Fitting 5 folds for each of 100 candidates, totalling 500 fits<jupyter_text>**3. Get permutation importance using the validation dataset generated in section 2.1**- Impurity-based feature importance can inflate the importance of numerical features.- The impurity-based feature importance of random forests suffers from being computed on statistics derived from the training dataset: the importances can be high even for features that are not predictive of the target variable, as long as the model has the capacity to use them to overfit.Ref: [Permutation Importance vs Random Forest Feature Importance (MDI)](https://scikit-learn.org/stable/auto_examples/inspection/plot_permutation_importance.htmlsphx-glr-auto-examples-inspection-plot-permutation-importance-py)<jupyter_code>fn = '../data/rf_pipeline.pickle'
rf_random = read_from_pickle(fn)
result = permutation_importance(rf_random.best_estimator_, x_val, y_val, n_repeats=10,
                                random_state=15, n_jobs=-1)
sorted_idx = result.importances_mean.argsort()[::-1]
mean_importance = result.importances_mean[sorted_idx]
fig, ax = plt.subplots(figsize=(10, 5))

ax = sns.barplot(x=mean_importance,
                 y=x_test.columns[sorted_idx],
                color=[0.2, 0.4, 0.6])
ax.set_ylim([15, -0.5])
ax.set_title("Permutation Importance (Validation set)")
fig.tight_layout()

plt.savefig("../figures/feat_sel_permut_importance.svg",
            dpi=300, bbox_inches='tight', facecolor="white")
thresh = np.percentile(mean_importance, 75)
idx_sel_feat_sorted = mean_importance > thresh
idx_sel_feat = sorted_idx[idx_sel_feat_sorted]
fn = '../data/idx_sel_feat_permu_importance.pickle'
save_as_pickle(fn, idx_sel_feat)<jupyter_output>Saved data to "../data/idx_sel_feat_permu_importance.pickle"
{'fraction_non_alphanumeric': 0.1195097037793667, 'fraction_numerical': 0.018556350017024173, 'mean_word_length': 4.345768880800728, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 22, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '338327', 'license': 'MIT', 'n_tokens_mistral': 1933, 'n_tokens_neox': 1816, 'n_words': 441, 'path': 'src/feature_selection.ipynb', 'repo_name': 'weizhao-BME/metis-project3', 'size': '5874'}