<jupyter_start><jupyter_text>---title: Predicting IBM Churn with LDA and XGBoostauthor: R package builddate: '2021-01-05'slug: hi-hugocategories: []tags: []subtitle: ''summary: 'Uncover the factors that lead to employee attrition and explore important questions such as ‘show me a breakdown of distance from home by job role and attrition’ or ‘compare average monthly income by education and attrition’. This is a fictional data set created by IBM data scientists.'authors: []lastmod: '2021-09-18T13:33:09-04:00'featured: noimage:  caption: ''  focal_point: ''  preview_only: noprojects: []---  HR Churn Analysis This is me fiddling around with an employee attrition dataset on my last few days at PNC. DescriptionUncover the factors that lead to employee attrition and explore important questions such as ‘show me a breakdown of distance from home by job role and attrition’ or ‘compare average monthly income by education and attrition’. This is a fictional data set created by IBM data scientists.Education1 'Below College'2 'College'3 'Bachelor'4 'Master'5 'Doctor'EnvironmentSatisfaction1 'Low'2 'Medium'3 'High'4 'Very High'JobInvolvement1 'Low'2 'Medium'3 'High'4 'Very High'JobSatisfaction1 'Low'2 'Medium'3 'High'4 'Very High'PerformanceRating1 'Low'2 'Good'3 'Excellent'4 'Outstanding'RelationshipSatisfaction1 'Low'2 'Medium'3 'High'4 'Very High'WorkLifeBalance1 'Bad'2 'Good'3 'Better'4 'Best'  Libraries<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import plotly.express as px
import seaborn as sns
import sklearn

from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = 'all'
%load_ext autoreload<jupyter_output><empty_output><jupyter_text>Read in Data<jupyter_code>df = pd.read_csv('ibm_attrition_file.csv').sort_values('Attrition')

df.head()
df.shape
df.dtypes

sns.displot(df, x='Attrition', shrink = .8)<jupyter_output><empty_output><jupyter_text>EDAI first look at some features that might be important in the dataset  Attrition by AgeAttrition rates are higher among younger employees.<jupyter_code>sns.displot(
    df.Age,
    kde = True
)
sns.displot(data=df, x="Age", hue="Attrition", multiple="stack")<jupyter_output><empty_output><jupyter_text>Job SatisfactionLower job satisfaction unsurprisingly yields a higher attrition rate although perhaps not as high as one might expect.<jupyter_code>g = sns.displot(
    data = df,
    x = 'JobSatisfaction',
    hue = 'Attrition',
    multiple = 'stack',
    discrete=True,
    shrink = .8
    )

g.set_axis_labels("Job Satisfaction", "Count", labelpad=10)
g.set(title="Lower Job Satisfaction leads to Higher Attrition")
g.fig.set_size_inches(6.5, 4.5)
g.despine(trim=True)<jupyter_output><empty_output><jupyter_text>Gender and Marital StatusSingle men and women are more likely to leave the company but age may be a confounder. Men are slightly more likely to leave than women.<jupyter_code>df_new = df.copy()
df_new['Attr'] = df_new['Attrition'].apply(lambda x: 1 if x == 'Yes' else 0)
df_new['Count'] = 1
df_new_group = df_new.groupby(['Gender', 'MaritalStatus'])[['Attr', 'Count']].sum()
df_new_group['pct'] = df_new_group['Attr'] / df_new_group['Count']

df_heat = df_new_group.reset_index()[['Gender','MaritalStatus','pct']]\
    .pivot('MaritalStatus', 'Gender')

sns.heatmap(
    data=df_heat
)

# sns.heatmap(
#     data=df[['Gender', 'MaritalStatus', ]]
#     )<jupyter_output><empty_output><jupyter_text>Correlation Plots<jupyter_code>df_num = df[df.columns[df.dtypes == 'int64']][['Age', 'DailyRate', 'DistanceFromHome',\
                                              'MonthlyIncome', 'PerformanceRating', 'JobSatisfaction', \
                                             'WorkLifeBalance']]


g = sns.PairGrid(df_num)
g.map_upper(sns.histplot)
g.map_lower(sns.kdeplot, fill=True)
g.map_diag(sns.histplot, kde=True)<jupyter_output><empty_output><jupyter_text>Many features are correlated although no features have a higher correlation than 0.3 or less than -0.1.<jupyter_code>df_heat = df[df.columns[df.dtypes == 'int64']]\
    .drop(['EmployeeCount', 'EmployeeNumber', 'StandardHours'], axis=1)

corr = df_heat.corr()

# Set up the matplotlib figure
f, ax = plt.subplots(figsize=(11, 9))

# Generate a custom diverging colormap
cmap = sns.diverging_palette(230, 20, as_cmap=True)

# Draw the heatmap with the mask and correct aspect ratio
sns.heatmap(corr,  cmap=cmap, vmax=.3, center=0,
            square=True, linewidths=.5, cbar_kws={"shrink": .5})<jupyter_output><empty_output><jupyter_text>Predictive Analytics - LDALinear Discriminant Analysis (LDA) is a feature reduction method for data with discrete classes. It is like PCA except that it takes advantage of information about the classification in the training data. It projects data into fewer dimensions by maximizing both the mean distance between the median data point of each class and minimizing the "spread" within each class.<jupyter_code># Drop columns with no information
df.columns
df_clean = df.drop(['EmployeeCount', 'EmployeeNumber', 'StandardHours', 'Over18'], axis=1)
df_clean['Attrition'] = df_clean['Attrition'].apply(lambda x: 1 if x == 'Yes' else 0)<jupyter_output><empty_output><jupyter_text>Encode Features<jupyter_code>df_objs = df_clean.columns[df_clean.dtypes == 'object']

df_clean[df_objs]
df_enc = pd.get_dummies(df_clean, prefix = df_objs)
df_enc['Attrition'] = df_clean['Attrition'].apply(lambda x: 'Yes' if x == 1 else 'No')
df_enc.head()<jupyter_output><empty_output><jupyter_text>Split and Normalize Data<jupyter_code>import sklearn.preprocessing as sp
import sklearn.model_selection as skms
train, test = skms.train_test_split(df_enc, random_state=42)
trainY, testY = train.Attrition, test.Attrition
trainX, testX = train.drop(['Attrition'], axis=1), test.drop(['Attrition'], axis=1)

df_objs = df_clean.columns[df_clean.dtypes == 'object']

df_clean[df_objs]
df_enc = pd.get_dummies(df_clean, prefix = df_objs)
df_enc.head()

scaler = sp.StandardScaler().fit(trainX)
scale_train = scaler.transform(trainX)
scale_test = scaler.transform(testX)<jupyter_output><empty_output><jupyter_text>Cross Validation Training<jupyter_code>import sklearn.discriminant_analysis as sda
import sklearn.metrics as sm

from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedStratifiedKFold

model = sda.LinearDiscriminantAnalysis()

# model.fit(scale_train, trainY)
# define model evaluation method
cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=42)
# evaluate model
scores = cross_val_score(model, scale_train, trainY, scoring='accuracy', cv=cv, n_jobs=2)
# summarize result
scores

print('Mean Accuracy: %.3f (%.3f)' % (np.mean(scores), np.std(scores)))

model.fit(scale_train, trainY)

test_preds = model.predict(scale_test)<jupyter_output><empty_output><jupyter_text>Quick evaluation on test set<jupyter_code>model = sda.LinearDiscriminantAnalysis()
model.fit(scale_train, trainY)

test_preds = model.predict(scale_test)
sm.confusion_matrix(test_preds, testY)
print(f'Test Accuracy: {round(sm.accuracy_score(test_preds, testY),3)*100}%')<jupyter_output><empty_output><jupyter_text>Extract Feature Importances<jupyter_code>lda_features = pd.DataFrame(data={'Feature': trainX.columns, 'Contribution':model.coef_[0], 'abs_cont': np.abs(model.coef_[0])})\
    .sort_values('abs_cont', ascending = False)[['Feature', 'Contribution']]<jupyter_output><empty_output><jupyter_text>The test accuracy is actually even higher than the cross validated training accuracy  Visualize in one dimensionThe accuracy is pretty high even reducing everything down to just one dimension which is pretty impressive. I show with some jitter below<jupyter_code>fitted_model = model.fit(scale_train, trainY)

one_d_train = fitted_model.transform(scale_train)
one_d_train = pd.DataFrame(one_d_train, columns = ['X'])
one_d_train['Label'] = trainY.values

jitter = np.random.random(one_d_train.shape[0])

fig, axs = plt.subplots(ncols=2)

fig.set_size_inches(15, 6)
# df['korisnika'].plot(ax=axs[0])
# df['osiguranika'].plot(ax=axs[1])

sns.scatterplot(data=one_d_train, x = 'X', y=jitter,  hue="Label", hue_order = ['No', 'Yes'], ax = axs[0])

one_d_test = fitted_model.transform(scale_test)
one_d_test = pd.DataFrame(one_d_test, columns = ['X'])
one_d_test['Label'] = testY.values

jitter = np.random.random(one_d_test.shape[0])

sns.scatterplot(data=one_d_test, x = 'X', y=jitter,  hue="Label", hue_order = ['No', 'Yes'], ax = axs[1])<jupyter_output><empty_output><jupyter_text>But how does this compare to XGBoost?<jupyter_code>from xgboost import XGBClassifier
from xgboost import plot_importance

model = XGBClassifier()
model.fit(trainX, trainY)

train_xgb_preds = model.predict(trainX)
test_xgb_preds = model.predict(testX)

sm.confusion_matrix(train_xgb_preds, trainY)
print(f'Test Accuracy: {round(sm.accuracy_score(train_xgb_preds, trainY),3)*100}%')

sm.confusion_matrix(test_xgb_preds, testY)
print(f'Test Accuracy: {round(sm.accuracy_score(test_xgb_preds, testY),3)*100}%')<jupyter_output>/Users/marshallkrassenstein/anaconda3/lib/python3.7/site-packages/xgboost/sklearn.py:892: UserWarning:

The use of label encoder in XGBClassifier is deprecated and will be removed in a future release. To remove this warning, do the following: 1) Pass option use_label_encoder=False when constructing XGBClassifier object; and 2) Encode your labels (y) as integers starting with 0, i.e. 0, 1, 2, ..., [num_class - 1].<jupyter_text>Untrained, XGBoost classifies the training data perfectly but is actually outperformed by LDA on the test data. I suspect a simple grid search would amerliorate this. First, however I want to compare the feature imporance in the two jobs.<jupyter_code>ax = plot_importance(model)
fig = ax.figure
fig.set_size_inches(10, 10)

lda_features<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11712897347641223, 'fraction_numerical': 0.014780319902814336, 'mean_word_length': 6.372388059701493, 'pattern_counts': {'":': 1, '<': 64, '<?xml version=': 0, '>': 64, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '236630', 'license': 'MIT', 'n_tokens_mistral': 3515, 'n_tokens_neox': 3357, 'n_words': 845, 'path': 'content/post/2021-01-05-ibm-churn/index.ipynb', 'repo_name': 'mpkrass7/m_pkfire', 'size': '9878'}