<jupyter_start><jupyter_text>Importing what I may use and reading the data<jupyter_code>import pandas as pd
import numpy as np
import sklearn as skl
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn import svm
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegressionCV
from sklearn import tree
from math import log
import datetime
df = pd.read_csv('employee_retention_data.csv')<jupyter_output><empty_output><jupyter_text>Let's take a look at the features<jupyter_code>list(df)<jupyter_output><empty_output><jupyter_text>And now let's do some EDA<jupyter_code>df.shape
plt.hist(df.seniority)
plt.hist(df.salary)
df.groupby('dept')['employee_id'].count()
df.groupby('company_id')['employee_id'].count()<jupyter_output><empty_output><jupyter_text>Want to know how many people quit<jupyter_code>df.quit_date.isna().sum()<jupyter_output><empty_output><jupyter_text>So the percent of people who were hired and quit is<jupyter_code>1 - df.quit_date.isna().sum()/df.employee_id.count()<jupyter_output><empty_output><jupyter_text>Since every row has an employee id (see below)<jupyter_code>df.employee_id.isna().sum()<jupyter_output><empty_output><jupyter_text>The classes of quit vs. not quit are pretty balanced if 45% have quit and 55% have not Converting the dates to datetime format<jupyter_code>df.join_date = pd.to_datetime(df.join_date)
df.quit_date = pd.to_datetime(df.quit_date)
plt.hist(df.join_date)
plt.hist(df.quit_date[~df.quit_date.isna()])<jupyter_output><empty_output><jupyter_text>Want to know how long they had the job before they quit<jupyter_code>df['job_length'] = df.quit_date - df.join_date<jupyter_output><empty_output><jupyter_text>Now converting this to just how many days it is.<jupyter_code>sec = lambda t: t.total_seconds() 
df.job_length = df.job_length.map(sec)/(60*60*24)
df.job_length.max()<jupyter_output><empty_output><jupyter_text>I decided that, in order to understand employee churn, I should try to predict IF someone will quit and what features are the most important for that classification. Ultimately, I would like to predict WHEN that would be. Perhaps after deciding someone would quit using a classification algorithm I could try to predict when it would be using a regressor? This may need to wait. Dealing with the data a little more: I noticed the seniority was very skewed, so let's look at the max values<jupyter_code>sorted(df.seniority)[-10:]<jupyter_output><empty_output><jupyter_text>Values of 98 or 99 don't make sense, so let's remove them<jupyter_code>df = df[df.seniority < 50]
df.shape
plt.hist(df.seniority)<jupyter_output><empty_output><jupyter_text>That looks a lot better! I would also like it if the salary were more normal<jupyter_code>plt.hist(df.salary)
plt.hist(df.salary**(1/3))<jupyter_output><empty_output><jupyter_text>That's a little better. Let's make it official:<jupyter_code>df.salary = df.salary**(1/3)<jupyter_output><empty_output><jupyter_text>Now let's scale salary and seniority<jupyter_code>sal_mean = df.salary.mean()
sal_std = df.salary.std()
sen_mean = df.seniority.mean()
sen_std = df.seniority.std()
print('sal_mean: {}, sal_std: {}, sen_mean: {}, sen_std: {}'.format(sal_mean, sal_std, sen_mean, sen_std))<jupyter_output>sal_mean: 49.851674699458314, sal_std: 9.801621996433013, sen_mean: 14.120971659919029, sen_std: 8.0541371816683<jupyter_text>Let's scale these values and then plot/look at<jupyter_code>df.salary = (df.salary - sal_mean)/sal_std
df.seniority = (df.seniority - sen_mean)/sen_std
plt.hist(df.salary)
plt.hist(df.seniority)<jupyter_output><empty_output><jupyter_text>Alright, those are in a comparable range now! Let's start applying some machine learning algorithms. I will look at the F1 score. I am interested in the recall, because I want to make sure the people who quit are identified. However, I do not want to predict that everyone will quit, so I will use F1. Adding the T/F quit feature<jupyter_code>df['quit_job'] = ~df.quit_date.isna()
bigdf = df #in case I want this later
list(df)
df = df.drop(columns = ['employee_id', 'join_date',
 'quit_date',
 'job_length'])
list(df)<jupyter_output><empty_output><jupyter_text>The first time I did this, I did not get dummies for the company id. I am doing that now!<jupyter_code>type(df.company_id.iloc[12])
df.company_id = df.company_id.map(str)
type(df.company_id.iloc[12])
df = pd.get_dummies(df, drop_first = True)
list(df)
X_train, X_test, y_train, y_test = train_test_split(df.drop(columns = ['quit_job']), df.quit_job, test_size=0.2, random_state = 0)<jupyter_output><empty_output><jupyter_text>First, **Logistic Regression**<jupyter_code>best_F1 = 0
best_solver = ''
best_LogReg = ''
best_pen = ''
for s in ['newton-cg', 'lbfgs', 'sag']:
    clf = skl.linear_model.LogisticRegression(penalty = 'l2', solver = s).fit(X_train, y_train)
    F1 = skl.metrics.f1_score(y_test, clf.predict(X_test))
    print('F1 score of {} for solver {} with penalty {}'.format(F1, s, 'l2'))
    if F1 > best_F1:
        best_F1 = F1
        best_solver = s
        best_pen = 'l2'
        best_LogReg = clf
for s in ['liblinear', 'saga']:
    clf = skl.linear_model.LogisticRegression(penalty = 'l1', solver = s).fit(X_train, y_train)
    F1 = skl.metrics.f1_score(y_test, clf.predict(X_test))
    print('F1 score of {} for solver {} with penalty {}'.format(F1, s, 'l1'))
    if F1 > best_F1:
        best_F1 = F1
        best_solver = s
        best_pen = 'l1'
        best_LogReg = clf
print('The best logistic regression F1 score I found is {} for solver {} with penalty {}'.format(best_F1, best_solver, best_pen))<jupyter_output>F1 score of 0.7002164502164503 for solver newton-cg with penalty l2
F1 score of 0.7002164502164503 for solver lbfgs with penalty l2
F1 score of 0.7002164502164503 for solver sag with penalty l2
F1 score of 0.7010671349452925 for solver liblinear with penalty l1
F1 score of 0.7008916509051607 for solver saga with penalty l1
The best logistic regression F1 score I found is 0.7010671349452925 for solver liblinear with penalty l1<jupyter_text>The F1 scores are all pretty comparable, but I will go with the largest (by a little) Need a way to consider the feature importance Let's consider another model. This time, random forest!<jupyter_code>RFC_10 = RandomForestClassifier(n_estimators = 10).fit(X_train, y_train)
F1 = skl.metrics.f1_score(y_test, RFC_10.predict(X_test))
print('For random forest with 10 trees I have an F1 score of {}'.format(F1))
RFC_20 = RandomForestClassifier(n_estimators = 20).fit(X_train, y_train)
F1 = skl.metrics.f1_score(y_test, RFC_20.predict(X_test))
print('For random forest with 20 trees I have an F1 score of {}'.format(F1))
RFC_50 = RandomForestClassifier(n_estimators = 50).fit(X_train, y_train)
F1 = skl.metrics.f1_score(y_test, RFC_50.predict(X_test))
print('For random forest with 50 trees I have an F1 score of {}'.format(F1))
RFC_100 = RandomForestClassifier(n_estimators = 100).fit(X_train, y_train)
F1 = skl.metrics.f1_score(y_test, RFC_100.predict(X_test))
print('For random forest with 100 trees I have an F1 score of {}'.format(F1))<jupyter_output>For random forest with 100 trees I have an F1 score of 0.5555959302325582<jupyter_text>So, RFC_100 is my best random forest, but it is not as good as my logistic regression. I could also have done more hyperparameter tuning. Let's look at decision trees now. Should I have done trees before forests? Probably!<jupyter_code>T = tree.DecisionTreeClassifier().fit(X_train, y_train)
F1 = skl.metrics.f1_score(y_test, T.predict(X_test))
print('For the default decision tree, I have an F1 score of {}'.format(F1))<jupyter_output>For the default decision tree, I have an F1 score of 0.5282071097372488<jupyter_text>I guess it makes sense that the decision tree wouldn't be as good as a random forest. Let's look at one more classifier: SVM. I'll play with a few parameters, but I'm not doing cross-validation (I know I should)<jupyter_code>S = skl.svm.SVC().fit(X_train, y_train)
F1 = skl.metrics.f1_score(y_test, S.predict(X_test))
print('For the default SVM, I have an F1 score of {}'.format(F1))
best_F1 = 0
best_C = -1
best_deg = -1
for my_C in range(1,5):
    for deg in range(1,5):
        S = skl.svm.SVC(C = my_C, degree = deg, gamma = 'auto').fit(X_train, y_train) 
        # gamma = 'auto' to prevent warning from printing
        F1 = skl.metrics.f1_score(y_test, S.predict(X_test))
        print('For SVC with C = {} and degree {}, I have an F1 score of {}'.format(my_C, deg, F1))
        if F1 < best_F1:
            best_F1 = F1
            best_C = my_C
            best_deg = deg
            best_SVC = S<jupyter_output>For SVC with C = 1 and degree 1, I have an F1 score of 0.7027102154273802
For SVC with C = 1 and degree 2, I have an F1 score of 0.7027102154273802
For SVC with C = 1 and degree 3, I have an F1 score of 0.7027102154273802
For SVC with C = 1 and degree 4, I have an F1 score of 0.7027102154273802
For SVC with C = 2 and degree 1, I have an F1 score of 0.7027627377481603
For SVC with C = 2 and degree 2, I have an F1 score of 0.7027627377481603
For SVC with C = 2 and degree 3, I have an F1 score of 0.7027627377481603
For SVC with C = 2 and degree 4, I have an F1 score of 0.7027627377481603
For SVC with C = 3 and degree 1, I have an F1 score of 0.699665365309537
For SVC with C = 3 and degree 2, I have an F1 score of 0.699665365309537
For SVC with C = 3 and degree 3, I have an F1 score of 0.699665365309537
For SVC with C = 3 and degree 4, I have an F1 score of 0.699665365309537
For SVC with C = 4 and degree 1, I have an F1 score of 0.697943082558467
For SVC with C = 4 and degree 2, I have an [...]<jupyter_text>I guess changing the degree didn't change anything. Well I'm keeping it here to show I tried adjusting parameters. So, I'm done playing with models. The best F1 score I got BEFORE making the company_id categorical was  0.7074 for logistic regression with solver saga and penalty l1. The best score I got AFTER doing that was the default SVC. Now I want to figure out what is contributing most to this model. First, for the logistic regression...<jupyter_code>best_LogReg.coef_
list(df)
df.std()
best_LogReg.coef_[0]*df.drop(columns = ['quit_job']).std()
sum([0.006337,
0.000872,
0.033981,
0.024311,
0.026914,
0.027337,
0.009854,
0.024824,
0.013631,
0.006393])<jupyter_output><empty_output><jupyter_text>The internet makes it sound like a coeffiecient times the std dev of the feature is related to feature importance. If so, it seems like the company has a big influcence on quitting (if you sum the absolute values, as above), and working in DS or DE influences you not to quit. Salary also has an impact. **Questions/Answers BASED ON WORK SO FAR** * _What are the main factors that drive employee churn? Do they make sense? Explain your findings._It appears the main factors that drive employee churn are where they work (company), their salary, and if they work in DS or DE (this seems to mean they don't quit). Since salary has a positive coefficient, does that mean an employee with a higher salary is more likely to quit?* _What might you be able to do for the company to address employee Churn, what would be follow-up actions?_There are different companies, aren't there? Are they sub-companies?* _If you could add to this data set just one variable that could help explain employee churn, what would that be?_I would like a variable of the reason an employee gives for leaving. This could be in the form of a survey, or we could use NLP. There may be something we are not considering **Other notes**I would have liked to:* Combine features* Hyperparameter tuning with cross validation* Regression. Is that actualy what this problem is about? I don't think this model is actually good, by the way. Look at what it predicts for the test set:<jupyter_code>best_LogReg.predict(X_test).sum()
X_test.shape<jupyter_output><empty_output><jupyter_text>It predicts almost everyone as quitting! NOTE TO FUTURE ME: I will need to read through everything and make sure what I wrote makes sense! **BUT WAIT, there's more!**So, I heard someone say they didn't even use a model, so now I'm going to just do some more EDA and see how the quit/didn't quit populations differ Going back to the original df before I scaled some features<jupyter_code>origdf = pd.read_csv('employee_retention_data.csv')
origdf['quit_job'] = ~origdf.quit_date.isna()
origdf = origdf[df.seniority < 50]<jupyter_output><empty_output><jupyter_text>More EDA time!<jupyter_code>origdf[origdf.quit_job == 1].salary.mean()
origdf[origdf.quit_job == 0].salary.mean()
origdf[origdf.quit_job == 1].salary.median()
origdf[origdf.quit_job == 0].salary.median()
origdf[origdf.quit_job == 1].seniority.mean()
origdf[origdf.quit_job == 0].seniority.mean()
origdf[origdf.quit_job == 1].seniority.median()
origdf[origdf.quit_job == 0].seniority.median()<jupyter_output><empty_output><jupyter_text>Now, I would like to do hypothesis testing, but my first instinct is that these aren't very different (definitely for seniority) Okay, let's try a hypothesis test in Python for comparing the salaries...<jupyter_code>np.std(origdf[origdf.quit_job == 0].salary)
np.std(origdf[origdf.quit_job == 1].salary)
from scipy import stats
t, p = stats.ttest_ind(origdf[origdf.quit_job == 1].salary, origdf[origdf.quit_job == 0].salary, equal_var=False)
print(t)
print(p)<jupyter_output>1.2474275862013815e-08<jupyter_text>Okay, that's a small p-value! So it seems we can reject the null hypothesis that the two populations (quit and not quit) have identical mean salaries. So, it looks like the people who quit had lower salaries, so I'm thinking that's a contributor to why they quit. Using statistics but not machine learning. Now let's look at the salary divided by seniority (dollars per year of experience). Order: mean (quit, not quit), median (quit, not quit), std dev (quit, not quit)<jupyter_code>(origdf[origdf.quit_job == 1].salary/origdf[origdf.quit_job == 1].seniority).mean()
(origdf[origdf.quit_job == 0].salary/origdf[origdf.quit_job == 0].seniority).mean()
(origdf[origdf.quit_job == 1].salary/origdf[origdf.quit_job == 1].seniority).median()
(origdf[origdf.quit_job == 0].salary/origdf[origdf.quit_job == 0].seniority).median()
(origdf[origdf.quit_job == 1].salary/origdf[origdf.quit_job == 1].seniority).std()
(origdf[origdf.quit_job == 0].salary/origdf[origdf.quit_job == 0].seniority).std()
t, p = stats.ttest_ind(origdf[origdf.quit_job == 1].salary/origdf[origdf.quit_job == 1].seniority, origdf[origdf.quit_job == 0].seniority, equal_var=False)
print(p)
t, p = stats.ttest_ind(origdf[origdf.quit_job == 1].salary/origdf[origdf.quit_job == 1].seniority, origdf[origdf.quit_job == 0].seniority, equal_var=True)
print(p)<jupyter_output>0.0<jupyter_text>Okay, so whether or not the standard deviations are the same, we get a small p-value, so the population means are different. That is, on average, the people who quit had a lower salary per year of experience than the people who didn't quit. Below is what percent of employees from each company quit<jupyter_code>origdf[origdf.quit_job == 1].groupby('company_id').employee_id.count()/origdf.groupby('company_id').employee_id.count()<jupyter_output><empty_output><jupyter_text>It looks like a lot of employees in company 11 are quitting, but it's not a large company (it only has 16 employees):<jupyter_code>origdf[origdf.company_id == 11].employee_id.count()<jupyter_output><empty_output><jupyter_text>Below is the percent that quit by department<jupyter_code>origdf[origdf.quit_job == 1].groupby('dept').employee_id.count()/origdf.groupby('dept').employee_id.count()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10162292379865602, 'fraction_numerical': 0.04760999112463547, 'mean_word_length': 5.32011217948718, 'pattern_counts': {'":': 0, '<': 117, '<?xml version=': 0, '>': 116, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '30', 'dir': 'jupyter-structured-clean-dedup', 'id': '539575', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 6012, 'n_tokens_neox': 5398, 'n_words': 1777, 'path': 'others/Data_Challenge1/Rebecca_Moran_data_challenge_1.ipynb', 'repo_name': 'davestanley-cogrhythms/insight-datachallenge1', 'size': '15774'}