<jupyter_start><jupyter_text>"My Boss Wants to Know How Important My Features Are"> "A tutorial on feature importances included in your models."- toc: true- branch: master- badges: true- comments: true- categories: [feature importance, classification, F1 score]- hide: false- search_exclude: true  Theory What is feature importance?Feature importance is the relative weights of your features in `X` in predicting your target variable `y`.For this post, I will focus on the **feature importances that are included in your models in Scikit-learn**, a machine learning library in Python.You can also derive feature importance by (1) shuffling your data for a feature and seeing how that affects your predictions, (2) removing a feature, retraining your model, and seeing how that affects your predictions, and (3) fit local linear models around small changes in the features to explain specific target variable results. You can read more about those techniques [at this post on *Towards Data Science*](https://towardsdatascience.com/explaining-feature-importance-by-example-of-a-random-forest-d9166011959e). These methods are more accurate but more computationally intensive. Examples Why do we care about feature importance?- A client wants you to make inferences about which feature affects home prices most.- You want to know whether your model implicitly says that people of a certain race should not see advertisements for your company's retirement plan.- Your supervisor wants you to figure out the groups of words most indicative of a spam email. Code How to extract and graph the feature importances included in Scikit-learn.For this example, we will be using a dataset about retirement plans. Our target variable will be eligibility for the 401(k) plan.- e401k = 1 if eligible for 401(k)- inc = annual income (\$)- marr = 1 if married- male = 1 if male- age = age (years)- fsize = family size- nettfa = net total financial assets (\$1000s)- p401k = 1 if participates in 401(k)- pira = 1 if has an IRA- incsq = inc2- agesq = age2We will drop the feature that indicates whether or not the potential customer is already participating in a 401(k) plan. That wouldn't be very interesting to model since that offers no insight on features affecting eligibility.Grab the [dataset from Google Drive](https://drive.google.com/file/d/1yJqI34SsQj5klPPa8YWRxZHJ9jucojr2/view) if you want to follow along. *Imports, Define Variables, and Train, Test, Split*- Start by importing the libraries and loading the data we will need:<jupyter_code>import numpy as np
import pandas as pd

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler, MinMaxScaler

from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import MultinomialNB
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import BaggingClassifier, RandomForestClassifier, ExtraTreesClassifier, AdaBoostClassifier

from sklearn.metrics import f1_score

import matplotlib.pyplot as plt
import matplotlib.style as style
import seaborn as sns

style.use('fivethirtyeight')
df = pd.read_csv("401ksubs.csv", sep = ",")
df.head()<jupyter_output><empty_output><jupyter_text>- Define the features and the target variable (remember, we're not using p401k):<jupyter_code>X = df[["inc", "marr", "male", "age", "fsize", "nettfa", "pira", "incsq", "agesq"]]
y = df["e401k"]

X_train, X_test, y_train, y_test = train_test_split(X,
                                                    y,
                                                    test_size = 0.33,
                                                    stratify = y,
                                                    random_state = 42)<jupyter_output><empty_output><jupyter_text>- I've set a `random_state` of `42` so that you can follow along and get the same results. If you're not trying to get the same results, you can leave the `random_state` blank.- I am only including only one value of each hyperparameter here, since hyperparameter tuning is not the focus of this blog post. Feel free to play around with the hyperparameters yourself! *Logistic Regression Most Important Features*<jupyter_code># Calculate Logistic Regression Feature Importances
sc = StandardScaler()
X_train_sc = sc.fit_transform(X_train)
X_test_sc = sc.transform(X_test)
lr = LogisticRegression(C = 2.32995, penalty = "l2", solver = "liblinear", random_state = 42)
lr.fit(X_train_sc, y_train)
print("Accuracy for training data: ", lr.score(X_train_sc, y_train))
print("Accuracy for test data: ", lr.score(X_test_sc, y_test))
print("F1 for training data: ", f1_score(y_train, lr.predict(X_train_sc)))
print("F1 for test data: ", f1_score(y_test, lr.predict(X_test_sc)))         

feature_importances = lr.coef_.reshape(len(X_train.columns),1)
feature_importances_df = pd.DataFrame(data = feature_importances,
                                      index = X_train.columns,
                                      columns = ["feature_importances"])

# Plot Feature Importances
plt.bar(x = feature_importances_df.index,
        height = feature_importances_df["feature_importances"])
ax = plt.gca()
fig = plt.gcf()
fig.set_size_inches(11,8)
ax.set_ylabel("Beta Coefficient for Feature", size = 18)
ax.set_xlabel("Feature", size = 18)
ax.set_title("Effect of Each Feature on 401(k) Eligibility", size = 25)
plt.show()<jupyter_output>Accuracy for training data:  0.6557772771161893
Accuracy for test data:  0.6546880104541
F1 for training data:  0.4717214126944925
F1 for test data:  0.47017543859649125<jupyter_text>- The feature importances are the beta coefficients of the logistic regression. For more information about beta coefficients, see this post on the [Display R Blog](https://www.displayr.com/how-to-interpret-logistic-regression-coefficients/).- The most significant features are `inc`, `agesq`, and `age`, in that order. *Multinomial Naive Bayes Classifier Most Important Features*<jupyter_code># Calculate Multinomial Naive Bayes Feature Importances
mms = MinMaxScaler()
X_train_sc = mms.fit_transform(X_train)
X_test_sc = mms.transform(X_test)
mnb = MultinomialNB(alpha = 0.55908)
mnb.fit(X_train_sc, y_train)
print("Accuracy for training data: ", mnb.score(X_train_sc, y_train))
print("Accuracy for test data: ", mnb.score(X_test_sc, y_test))
print("F1 for training data: ", f1_score(y_train, mnb.predict(X_train_sc)))
print("F1 for test data: ", f1_score(y_test, mnb.predict(X_test_sc)))         

feature_importances = mnb.coef_.reshape(len(X_train.columns),1)
feature_importances_df = pd.DataFrame(data = feature_importances,
                                      index = X_train.columns,
                                      columns = ["feature_importances"])

# Plot Feature Importances
plt.bar(x = feature_importances_df.index,
        height = feature_importances_df["feature_importances"])
ax = plt.gca()
fig = plt.gcf()
fig.set_size_inches(11,8)
ax.set_ylabel("Log of Feature Probability Given 401(k) Eligibility", size = 18)
ax.set_xlabel("Feature", size = 18)
ax.set_title("Effect of Each Feature on 401(k) Eligibility", size = 25)
plt.show()<jupyter_output>Accuracy for training data:  0.615545542323785
Accuracy for test data:  0.6128716105847762
F1 for training data:  0.06643220007815552
F1 for test data:  0.06324110671936758<jupyter_text>- The feature importances are the logarithms of the probabilities of each feature given 401(k) eligibility.- The most significant features are `marr`, `age`, and `agesq`, in that order. *Decision Tree Most Important Features*<jupyter_code># Calculate Decision Tree Feature Importances
sc = StandardScaler()
X_train_sc = sc.fit_transform(X_train)
X_test_sc = sc.transform(X_test)
dt = DecisionTreeClassifier(max_depth = 7, min_samples_leaf = 2, min_samples_split = 18, random_state = 42)
dt.fit(X_train_sc, y_train)
print("Accuracy for training data: ", dt.score(X_train_sc, y_train))
print("Accuracy for test data: ", dt.score(X_test_sc, y_test))
print("F1 for training data: ", f1_score(y_train, dt.predict(X_train_sc)))
print("F1 for test data: ", f1_score(y_test, dt.predict(X_test_sc)))         

feature_importances = dt.feature_importances_
feature_importances_df = pd.DataFrame(data = feature_importances,
                                      index = X_train.columns,
                                      columns = ["feature_importances"])

# Plot Feature Importances
plt.bar(x = feature_importances_df.index,
        height = feature_importances_df["feature_importances"])
ax = plt.gca()
fig = plt.gcf()
fig.set_size_inches(11,8)
ax.set_ylabel("Gini Feature Importance", size = 18)
ax.set_xlabel("Feature", size = 18)
ax.set_title("Effect of Each Feature on 401(k) Eligibility", size = 25)
plt.show()<jupyter_output>Accuracy for training data:  0.7190215642098488
Accuracy for test data:  0.6680823260372427
F1 for training data:  0.5962072155411655
F1 for test data:  0.5234521575984991<jupyter_text>- The importance of a feature is computed as (normalized) total reduction of the criterion brought by that feature, also known as the Gini importance.- The most significant features are the `nettfa`, `inc`, and `incsq`, in that order. *Bootstrap Aggregated (Bagged) Decision Tree Most Important Features*<jupyter_code># Calculate Bootstrap Aggregated (Bagged) Decision Tree Feature Importances
sc = StandardScaler()
X_train_sc = sc.fit_transform(X_train)
X_test_sc = sc.transform(X_test)
bag = BaggingClassifier(max_samples = 0.3,
                        n_estimators = 15,
                        random_state = 42, 
                        base_estimator = DecisionTreeClassifier(max_depth = 4, 
                                                                min_samples_leaf = 3,
                                                                min_samples_split = 11,
                                                                random_state = 42))
bag.fit(X_train_sc, y_train)
print("Accuracy for training data: ", bag.score(X_train_sc, y_train))
print("Accuracy for test data: ", bag.score(X_test_sc, y_test))
print("F1 for training data: ", f1_score(y_train, bag.predict(X_train_sc)))
print("F1 for test data: ", f1_score(y_test, bag.predict(X_test_sc)))         

feature_importances = np.mean([tree.feature_importances_ for tree in bag.estimators_], axis = 0)
feature_importances_df = pd.DataFrame(data = feature_importances,
                                      index = X_train.columns,
                                      columns = ["feature_importances"])

# Plot Feature Importances
plt.bar(x = feature_importances_df.index,
        height = feature_importances_df["feature_importances"])
ax = plt.gca()
fig = plt.gcf()
fig.set_size_inches(11,8)
ax.set_ylabel("Gini Feature Importance", size = 18)
ax.set_xlabel("Feature", size = 18)
ax.set_title("Effect of Each Feature on 401(k) Eligibility", size = 25)
plt.show()<jupyter_output>Accuracy for training data:  0.6971355004827808
Accuracy for test data:  0.6863770009800719
F1 for training data:  0.5669581224114129
F1 for test data:  0.5522388059701493<jupyter_text>- The importance of a feature is computed as (normalized) total reduction of the criterion brought by that feature, also known as the Gini importance, for the average decision tree.- If you've also set the `max_features` parameter of your `BaggingClassifier` to anything less than `1`, you won't be able to use this code to average the features on the decision trees and compare their importances.- The most significant features are the `nettfa`, `incsq`, and `inc`, in that order. *Random Forest Most Important Features*<jupyter_code># Calculate Random Forest Feature Importances
sc = StandardScaler()
X_train_sc = sc.fit_transform(X_train)
X_test_sc = sc.transform(X_test)
rf = RandomForestClassifier(max_depth = 7, n_estimators = 20, random_state = 42)
rf.fit(X_train_sc, y_train)
print("Accuracy for training data: ", rf.score(X_train_sc, y_train))
print("Accuracy for test data: ", rf.score(X_test_sc, y_test))
print("F1 for training data: ", f1_score(y_train, rf.predict(X_train_sc)))
print("F1 for test data: ", f1_score(y_test, rf.predict(X_test_sc)))         

feature_importances = rf.feature_importances_
feature_importances_df = pd.DataFrame(data = feature_importances,
                                      index = X_train.columns,
                                      columns = ["feature_importances"])

# Plot Feature Importances
plt.bar(x = feature_importances_df.index,
        height = feature_importances_df["feature_importances"])
ax = plt.gca()
fig = plt.gcf()
fig.set_size_inches(11,8)
ax.set_ylabel("Gini Feature Importance", size = 18)
ax.set_xlabel("Feature", size = 18)
ax.set_title("Effect of Each Feature on 401(k) Eligibility", size = 25)
plt.show()<jupyter_output>Accuracy for training data:  0.7318957193434181
Accuracy for test data:  0.6853969291081345
F1 for training data:  0.6220508166969148
F1 for test data:  0.5497896213183732<jupyter_text>- The importance of a feature is computed as (normalized) total reduction of the criterion brought by that feature, it is also known as the Gini importance.- The most significant features are the `nettfa`, `inc`, and `incsq`, in that order. *Extra Trees Most Important Features*<jupyter_code># Calculate Extra Trees Feature Importances
sc = StandardScaler()
X_train_sc = sc.fit_transform(X_train)
X_test_sc = sc.transform(X_test)
et = RandomForestClassifier(max_depth = 10, n_estimators = 45, random_state = 42)
et.fit(X_train_sc, y_train)
print("Accuracy for training data: ", et.score(X_train_sc, y_train))
print("Accuracy for test data: ", et.score(X_test_sc, y_test))
print("F1 for training data: ", f1_score(y_train, et.predict(X_train_sc)))
print("F1 for test data: ", f1_score(y_test, et.predict(X_test_sc)))         

feature_importances = et.feature_importances_
feature_importances_df = pd.DataFrame(data = feature_importances,
                                      index = X_train.columns,
                                      columns = ["feature_importances"])

# Plot Feature Importances
plt.bar(x = feature_importances_df.index,
        height = feature_importances_df["feature_importances"])
ax = plt.gca()
fig = plt.gcf()
fig.set_size_inches(11,8)
ax.set_ylabel("Gini Feature Importance", size = 18)
ax.set_xlabel("Feature", size = 18)
ax.set_title("Effect of Each Feature on 401(k) Eligibility", size = 25)
plt.show()<jupyter_output>Accuracy for training data:  0.8202446089475378
Accuracy for test data:  0.6798431885004901
F1 for training data:  0.7532582284073337
F1 for test data:  0.5446096654275093<jupyter_text>- The importance of a feature is computed as (normalized) total reduction of the criterion brought by that feature, it is also known as the Gini importance.- The most significant features are the `nettfa`, `incsq`, and `inc`, in that order. *Adaptively Boosted (AdaBoosted) Decision Tree Most Important Features*<jupyter_code># Calculate Adaptively Boosted (AdaBoosted) Decision Tree Feature Importances
sc = StandardScaler()
X_train_sc = sc.fit_transform(X_train)
X_test_sc = sc.transform(X_test)
ada = AdaBoostClassifier(learning_rate = 0.3,
                         n_estimators = 2,
                         random_state = 42, 
                         base_estimator = DecisionTreeClassifier(max_depth = 4, 
                                                                 min_samples_leaf = 2,
                                                                 min_samples_split = 2,
                                                                 random_state = 42))
ada.fit(X_train_sc, y_train)
print("Accuracy for training data: ", ada.score(X_train_sc, y_train))
print("Accuracy for test data: ", ada.score(X_test_sc, y_test))
print("F1 for training data: ", f1_score(y_train, ada.predict(X_train_sc)))
print("F1 for test data: ", f1_score(y_test, ada.predict(X_test_sc)))         

feature_importances = np.mean([tree.feature_importances_ for tree in ada.estimators_], axis = 0)
feature_importances_df = pd.DataFrame(data = feature_importances,
                                      index = X_train.columns,
                                      columns = ["feature_importances"])

# Plot Feature Importances
plt.bar(x = feature_importances_df.index,
        height = feature_importances_df["feature_importances"])
ax = plt.gca()
fig = plt.gcf()
fig.set_size_inches(11,8)
ax.set_ylabel("Gini Feature Importance", size = 18)
ax.set_xlabel("Feature", size = 18)
ax.set_title("Effect of Each Feature on 401(k) Eligibility", size = 25)
plt.show()<jupyter_output>Accuracy for training data:  0.6956871580302543
Accuracy for test data:  0.6860503103560928
F1 for training data:  0.5882865229697366
F1 for test data:  0.5772107347118345
{'fraction_non_alphanumeric': 0.09052316890881913, 'fraction_numerical': 0.04382660687593423, 'mean_word_length': 3.763884933067502, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 31, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 34, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '453761', 'license': 'Apache-2.0', 'n_tokens_mistral': 5622, 'n_tokens_neox': 5045, 'n_words': 1571, 'path': '_notebooks/2020-04-17-My-Boss-Wants-to-Know-How-Important-My-Features-Are.ipynb', 'repo_name': 'ynusinovich/Data-Science-Journeys', 'size': '16725'}