<jupyter_start><jupyter_text>Need of the day: People Analytics   1. IntroductionThe goal of this project is to find a systematic pattern or trend (using employee data) and provide actionable insights towards enhancing employee retention. Per latest statistics from Bureau of labor [2], voluntary quits rose in professional and business services by 82,000 as compared to last year and is projected to increase further in future. The cost of turnover is huge and it impacts both employee and employers. For an employee, the investment employee puts in the company is wasted because the employee has to start from scratch at new workplace. Employee may not be able to realize their full potential due to frequent job shifts. Moreover, in a team with high attrition rate, morale of remaining employees is hit and they may feel disengaged. On the other hand, for an employer new employees need to be hired and trained. Losing a long-term employee leads to ‘wisdom withdrawal’ as the historical knowledge an employee gathers over the years is lost. Ongoing projects take a hit and commitments cannot be delivered in time. If we can turn to data and predict which employee will leave and why, then the employer can enrich employee at risk of leaving and make them stay. Its a win win situation for both the employer and employee.      In an effort to do a pilot study, I have created a quick prototype of a statistical model to predict attrition. Since employee data can be huge for big organizations and the number of features required to predict attrition may vary from domain to domain, I started with a simpler and smaller dataset. Kaggle competition [1] with simulated dataset [9] fits my requirements well. Analysis on this dataset provides intriguing insights on what people value the most in their job. Using these insights, employers can intervene to make employees stay. Looking at a larger picture, this research is not only applicable to employee retention but also a wide variety of similar problems across society. For example, it can also can be applied to academia where students inclined to drop out can be identified and proactive action can be taken to encourage them to continue their studies.  2. Related Work Even though industry has perennially faced the problem of attrition, using data to analyze employees and predict attrition is fairly new. Even tech companies with deep data skills never thought of applying data analytics on their own employees until recently. I got interested in this problem while doing literature survey for my capstone project. There are countless articles on linked-in [6, 7] and data science blogs [4, 8] on analyzing and predicting employee turnover. Most of the literature written on People analytics is in the last 5 years. This is very surprising as business intelligence has been used to solve various business problems for decades [3]. Despite being a relatively new analytics fields, interest of organizations in People analytics is growing. According to Harvard Business Review analytics study [14], 15% executives (out of 230) said they use “predictive analytics based on HR data and data from other sources within or outside the organization,” while 48% executives predicted they would be doing so in two years. There are several commercial offerings in the market for people analytics. Visier [12] and Workday [13] are two popular tools used by large companies. These tools pull basic employee information from company database and analyze the costs and frequency of turnover. Though these analytic tools use data to create insights and predictions of employee behaviors, all these tools are geared towards benefitting employers. Most of the tools are employer oriented and produce insights based on profitability and head count which benefits employers. Whereas my research focuses on employee centric aspect of the same problem.**Research Questions**: Goal of this research is to apply data science techniques on employee data and answer the following questions: * Question 1: Which employees are most likely to quit? * Question 2: What are the reasons for the employees to quit? * Question 3: Can we identify major signals, events and\or themes which can be used by employers to enrich and retain employees?  3. MethodologyThere are four major modules in this project. The module 'Data' describes the data source information and the data munging. The module 'Statistical Analysis' describes the analytical methods used to explore data. The module 'Fitting the Logistic Regression Model' describes how employees are classified using Logistic Regression. The module 'Applying Logistic Regression results' describes how probabilites from logistic regression are used to classify employees on the basis of their risk of quitting.Following image depicts how these four modules interact with each other. Each module is described in detail in following subsections. 3.1 Data:**Data Source and License information:** The data is borrowed from Kaggle competition [Human Resources Analytics]('https://www.kaggle.com/ludobenistant/hr-analytics/data'). This dataset is released under CC BY-SA 4.0, the details of this license can be found [here]('https://creativecommons.org/licenses/by-sa/4.0/'). There are 15000 observations in the dataset with 10 features.**Data preprocessing:** The data is fairly neat. Although there is no data processing needed for the dataset, a few column names are misleading in context of this research. For example, 'sales' column is actually the department name for the employee. This column is renamed to 'department'. Similarly, 'left' column indicates if the employee has left the job. Since 'left' is also a target variable for building the attrition model, this column name is also not suitable. 'Left' column is renamed to 'attrition'. Additional data exploration revealed that there is not missing data. There are two categorical variables in this dataset: 'department' and 'salary'. Labels are changed to numeric so that these variables can be used while fitting the model. 3.2 Statistical Analysis: In this module, analytical methods are applied to get a sense of data. Once the data is summarized, correlation analysis is done to identify features for the classification model.**Descriptive Statistics:** First step in statistical analysis is to summarize data and find distribution of features in the dataset. Descriptive statistics is performed on the data to summarize it. As a part of descriptive statistics; mean, median, standard deviation, min, max, 25th and 75th percentile of columns with quantitative values are found.**Correlation Analysis:** Second step in statistical analysis is to find the features which are most correlated to attrition. This is done by performing correlation analysis of all features. A correlation matrix and correlation heatmap are created to find the features affecting target variable (attrition rate). Correlation analysis helped in identifying both positively and negatively correlated features.  3.3 Fitting the Logistic Regression model:Logistic Regression commonly evaluates the likelihood of an observation belonging to each group. This model is used to predict the likelihood of an event occurring. In contrast to linear regression, the output of logistic regression is transformed with a logit function. This makes the output either 0 or 1. This is a useful model to take advantage of for this problem because we are interested in predicting whether an employee will leave (0) or stay (1). Logistic regression is used to classify employees on the basis of their probability to leave. Given our research questions, this is an ideal model to use for couple of reasons. First, its a very straightforward model and python provides ready to use package to apply logistic regression [10]. Second, sklearn.linear_model.LogisticRegression method 'predict_proba(X)' is used to do the classification and find the probabilities to group the employees in different bands based on their risk of leaving. To assess the quality of the model, available data is divided into appropriately sized training and test sets, and these sets are varied randomly to ensure that the models thus constructed have optimum trade off of low variance versus low bias.  3.4 Applying Logistic Regression results:Probabilities calculated by Logistic regression are used to classify employees depending on their probability of quitting. Employees are classified in different bands as given below[16]:1.	**Extremely likely** – Employees with highest probability to quit (>90%). 2.	**Moderately likely** – Employees with medium probability to quit (50%< p< 90%).3.	**Least likely** – Employees with less than 50% probability to quit (<50%).Having identified employees with high risk of leaving, employer can take necessary actions to retain the employee.  4. Human centered research considerationsMost of the full time employees spend at least one third of their time in job. This research helps in exploring ways on how this time can be made more productive and fruitful. Most of the research done in people analytics focuses on employer's perspective and how the results can benefit large organizations in retaining employees. This project focuses on employees and aims at finding insights on what employees value the most in their jobs. Is it the money? or the number of hours worked or time spent in the company? or the number of years since they are promoted?In order to follow all the human centered values discussed in class, three major design considerations are observed in this research; Firstly, data is ethically sourced. The source data is free to be shared and adapted as long as appropriate credit is given and changes made are indicated. The source data has not been modified for the purpose of this project. Secondly, special attention is given to make sure that data is anonymized so as to address any privacy related concerns. Last but not the least, a conscious effort is made to thoroughly document the code and data.  5. FindingsThere are some pretty interesting findings from both the statistical analysis and regression model. Findings from each module are summerized below.  5.1 Findings from descriptive statistics* Mean satisfaction level of employees who stayed is 66% and employees who left is 44%. This is not so surprising finding as we expect more satisfied employees to stay with the company and less satisfied employees to quit. * The turnover rate for this company is 31%. This is a surprisingly high number. According to this survey[17] the maximum voluntary turnover rate within any industry is ~20%.* Mean last evaluation of employees who stayed and left is almost the same. This is again a very surprising finding as one would expect the performance rating to be correlated to attrition. The reason here may be that employees usually make up their mind way before evaluation and hence the rating doesnt affect the decision to quit.   5.2 Findings from correlation analysis* Most positively correlated feature with employee attrition is time spent at the company(correlation coefficient: 0.144). This is somewhat expected in professional sector. After a couple of years in a company, employees want to explore new oppurtunities and challenges which may lead them to quit.* Most negatively correlated feature with employee attrition is satisfaction level(correlation coefficient: -0.38) and work accidents(correlation coefficient: -0.15). We have already discussed that satisfaction level is inversely correlated to attrition. The inverse correlation of work accidents with attrition is very unexpected and may need deeper analysis. One possible reason can be that work accident may lead to long paid leaves and employee may want to stay with the same company for related compensation and benefits.   5.3 Findings from logistics regression model* Satisfaction level is the most important factor which makes employees stay (regression coefficient is highest: -4.06). Satisfaction level has come up in all our findings above and this is definitely the most important factor to determine attrition.* Employees promoted in last 5 years are more likely to stay (regression coefficient is high: -1.46). This definitely makes sense that if an employee has not been promoted for 5 years, the employees career has plateaued out for some reason. Employer should make efforts to find the root cause of this stagnation.* Salary is not a factor which makes employees stay (regression coefficient is quite low: 0.003). This implies that money is not the only thing employees value. Satisfaction level and career growth are more important considerations for an employee.   6. Discussion and ImplicationsGiven the findings from statistical analysis and regression model, we can draw the four major implications. Firstly, if at any point the employees satisfaction takes a hit of >30% the employee is at a risk of leaving. This is a crucial datapoint for an employer to intervene. Secondly, if an employee has not been promoted in last 4 years, employer should dig deeper to find the root cause. There can be various reasons for an employee to be not promoted such as employee is in the wrong role, wrong team or does not have conducive team environment. Thirdly, money is not the only thing which employees value. There are other factors  of value such as dignity, happiness, respect and recognition. Lastly, evaluation rating does not affect employees decision to stay. This implies that employers shouldn't bank on performance ratings as a datapoint to evaluate whether the employee is at risk of leaving or not. Employees usually make up their mind way before evaluation and hence the rating doesnt affect the decision to quit. 6.1 Limitations* One of the biggest limitation of this dataset is that it does not contain time series data. If we had a time element in data, we could have estimated the probability of employee attrition over time. Survival analysis [8] would have been a great statistical measure for this purpose.* This model is a prototype for building more complex models on real life data. This model uses simulated dataset with very limited number of features.  7. ConclusionThis project focuses on employees and aims at finding insights on what employees value the most in their jobs. The research successfully helped in exploring reasons on voluntary quits by employees. Using logistic regression on this dataset, we could predict employee attrition with an accuracy of ~76%. The probabilities calculated for each employee can be further used to enrich employees by providing personalized support and benefits. This is a win-win situation for both employer and employee. Significant findings from the research are summarized below:* Satisfaction level of employees is the most important factor which determines employee turnover* Employees promoted in last 5 years are more likely to stay* Salary is not strongly correlated to attrition * Last evaluation for employees makes no difference in their decision to stay   8. References* [1] Kaggle project https://www.kaggle.com/ludobenistant/hr-analytics* [2] Latest report from Bureau of Labour https://www.bls.gov/news.release/jolts.nr0.htm * [3] History of business intelligence https://www.betterbuys.com/bi/history-of-business-intelligence/* [4] Article on solving attrition with data https://towardsdatascience.com/solving-staff-attrition-with-data-3f09af2694cd * [5] Article on descriptive statistics https://www.marsja.se/pandas-python-descriptive-statistics/ * [6] Helpful linked in artcile on predicting employee attrition https://www.linkedin.com/pulse/predicting-employee-attrition-who-quit-when-praful-tickoo/* [7] Helpful linked in artciles on predicting employee attrition  https://www.linkedin.com/pulse/analyzing-employee-turnover-predictive-methods-richard-rosenow-pmp/* [8] Use of Survival analysis for predicting attrition https://www.slideshare.net/twbriggs/survival-analysis-for-predicting-employee-turnover/24* [9] Kaggle dataset https://www.kaggle.com/ludobenistant/hr-analytics/data* [10]sklearn logistic regression http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html* [11]Logistic Regression example https://towardsdatascience.com/building-a-logistic-regression-in-python-step-by-step-becd4d56c9c8* [12] Workforce analytics tool Visier https://www.visier.com/solutions/predict/* [13] People analytics by Workday https://www.workday.com/en-us/applications/human-capital-management/people-analytics.html* [14] Harvard Business Review article https://hbr.org/2017/06/hr-must-make-people-analytics-more-user-friendly* [15] Categorical Encoding http://pbpython.com/categorical-encoding.html* [16] Employee Attrition Risk Assessment paper http://rupeshkhare.com/wp-content/uploads/2013/12/Employee-Attrition-Risk-Assessment-using-Logistic-Regression-Analysis.pdfhttp://www.compensationforce.com/2017/04/2016-turnover-rates-by-industry.html* [17] Turnover rates by industry http://www.compensationforce.com/2017/04/2016-turnover-rates-by-industry.html  9. Attributions* Link to the license for data https://creativecommons.org/licenses/by-sa/4.0/* Changes to data: The values have not been changed in data. However there are few cosmetic changes made in data as follows:    - 'sales' column name changed to 'department'    - 'left' column name changed to 'attrition'    - 'salary' column encoded to quantitative values [15]    - 'department' column encoded to quantitative values [15]  10. Code  Import required libraries<jupyter_code># importing modules
import pandas as pd
import seaborn as sbn
import sklearn
from sklearn.linear_model import LogisticRegression
from sklearn import preprocessing
import numpy as np
import matplotlib.pyplot as plt
import matplotlib as matplot
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Get a glimpse of data<jupyter_code># importing the csv into dataframe
data = pd.DataFrame.from_csv('HR_comma_sep.csv', index_col=None)
data.head()
# Changing column names to make them more understandable
data['attrition'] = data['left']
data['department'] = data['sales']

data.drop('left', axis=1, inplace='true')
data.drop('sales', axis=1, inplace='true')

data = data[['attrition', 'department', 'satisfaction_level', 'last_evaluation', 'number_project',
       'average_montly_hours', 'time_spend_company', 'Work_accident',
       'promotion_last_5years', 'salary']]
data.head()<jupyter_output><empty_output><jupyter_text>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;There are 2 categorical variables in this dataset: 'department' and 'salary'. We need to change the labels to numeric so that we can use these variables while fitting the model.<jupyter_code># Get unique departments
data['department'].unique()
# Get unique salaries
data['salary'].unique()
# Transform the salary categorical to numeric so that it can be fit into model
le_salary = preprocessing.LabelEncoder()
data['salary'] = le_salary.fit_transform(data['salary'])
# Get unique salary
data['salary'].unique()
# Transform the department categorical to numeric so that it can be fit into model
le_dept = preprocessing.LabelEncoder()
data['department'] = le_dept.fit_transform(data['department'])
# Get unique department
data['department'].unique()
# describe data
print('rows, column:',data.shape)
data.describe()
# Group by attrition and find the mean
data.groupby('attrition').mean()
# Group by attrition and get the count
data.groupby('attrition').count()<jupyter_output><empty_output><jupyter_text>Summary of data based on above table:* About 15,000 employee observations and 10 features* Mean satisfaction of employees who stayed is 0.66%* The turnover rate is 31% which is very high<jupyter_code>#Correlation Matrix
data.corr()
# Create heatmap from correlation matrix
corr = data.corr()
corr = (corr)
sbn.heatmap(corr, 
            xticklabels=corr.columns.values, 
            yticklabels=corr.columns.values)<jupyter_output><empty_output><jupyter_text>** Positively Correlated Features:** - number of projects vs average monthly hours:  0.417211- average monthly hours vs : 0.339742- last evaluation vs number of projects: 0.349333**Negatively Correlated Feature:** - satisfaction vs attrition:  -0.388375<jupyter_code># Plot satisfation level of employees
ax=sbn.kdeplot(data.loc[(data.attrition==0),'satisfaction_level'],label="Stayed")
ax=sbn.kdeplot(data.loc[(data.attrition==1),'satisfaction_level'],label="Left")
ax.set(xlabel='Satisfaction Level',ylabel="Frequency")
plt.title("Satisfaction Level of Employees",fontsize=18)
# Plot attrition vs promotion
sbn.barplot(data.attrition, data.promotion_last_5years)
plt.title("Promotion in last 5 years vs Attrition",fontsize=18)
# plot attrition vs time spent in the company
sbn.barplot(data.attrition, data.time_spend_company)
plt.title("Time spent in company vs Attrition",fontsize=18)<jupyter_output><empty_output><jupyter_text>Training the model:<jupyter_code># Assign vlues
X = data.drop('attrition', axis=1) 
Y = data.attrition
# Train the model
X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X, Y, random_state=0)
# Apply logistic regression
lr_model = LogisticRegression()
fit = lr_model.fit(X_train, y_train)
# get features and weights
coeff = []
coeff = np.array(fit.coef_)
features = np.array(X.columns.values)
weights = np.vstack((features, coeff))
# print weights of each feature
weights
# 76.4% accuracy achieved
lr_model.score(X_test,y_test)
# Find predicted probability from the fitted model
predict_proba = lr_model.predict_proba(X_test)
predict_proba<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.056107593474141515, 'fraction_numerical': 0.012432407450201044, 'mean_word_length': 5.230348401957961, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 27, 'https://': 16, 'lorem ipsum': 0, 'www.': 13, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '86079', 'license': 'MIT', 'n_tokens_mistral': 5381, 'n_tokens_neox': 4972, 'n_words': 2867, 'path': '.ipynb_checkpoints/d512_final_project_report-checkpoint.ipynb', 'repo_name': 'dipsuw/data-512-final-project', 'size': '21637'}