<jupyter_start><jupyter_text>HR Anaytics -  Employee Attrition & PeformanceI took the IBM HR Analytics dataset from Kaggle. This is a fictional dataset created by IBM Scientists. In this project, I will be using python to do analysis around reasons for employee attrition. I will be applying all my learnings from the course: [Data Analysis with Python: Zero to Pandas](zerotopandas.com). This course is an excellent pathway for all the data science aspirants. What distingushes this course from any other data analytics course is the course structure. Here, you dont just listen but you get hands-on practice as well. Thankful to <NAME> for launching this course. Looking forward to learn more from this team.<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Downloading the Dataset> - Got the IBM HR Analytics dataset in csv format from : https://www.kaggle.com/datasets?fileType=csv> - Downloaded the dataset using the [`opendatasets` Python library](https://github.com/JovianML/opendatasetsopendatasets)<jupyter_code>emp_df= pd.read_csv('WA_Fn-UseC_-HR-Employee-Attrition.csv')<jupyter_output><empty_output><jupyter_text>Data Preparation and Cleaning> - Reading the csv file and doing initial statistical analysis (shape,values etc)> - Data Preprocessing: Reading the uniques values for each column and removing those which wont be significant in the analysis further.> - Create new dataframe to proceed with the analysis further.<jupyter_code>emp_df.head(50)
emp_df.Attrition.value_counts()
emp_df.info()
emp_df.describe()<jupyter_output><empty_output><jupyter_text>Here, we are counting the number of unique values of all the columns. This helped us decide that the following columns: EmployeeCount','StandardHours','EmployeeNumber','Over18 can be removed from the final data frame.Employee Number doesnt have any impact in anaysis as its a unique key for each record. Employee Count, Standard Hours and Over18 doesn't have enough data to do the analysis.<jupyter_code>for i in emp_df:
    print("No of Values for {} is {}".format(i,str(emp_df[i].nunique())))
emp_new_df=emp_df.copy()
emp_new_df.drop(['EmployeeCount','StandardHours','EmployeeNumber','Over18'],axis="columns",inplace=True)
emp_new_df.shape<jupyter_output><empty_output><jupyter_text>Exploratory Analysis and Visualization> - Find patterns of data through visualization and reveal the hidden trends from data.> - Using both matplotlib abd seaborn library to visualize the data> - Finding relationships between features using bar graphs, histograms,boxplots, heatmap etc> - Analysing both the numerical and the categorial columns separately<jupyter_code>sns.set_style('darkgrid')
matplotlib.rcParams['font.size'] = 14
matplotlib.rcParams['figure.figsize'] = (9, 5)
matplotlib.rcParams['figure.facecolor'] = '#00000000'<jupyter_output><empty_output><jupyter_text>Visualizing all the columns which are integer datatype and has unique values less than 30. We will be creating a new array with only these features and then visualize it all together. This step has helped us optimize our process and limit manual effort, as we are not looking at each of the columns manually. PS: I have taken 30 , just as an estimated number.<jupyter_code>num_col=[]
for i in emp_new_df.columns:
    if(emp_new_df[i].dtypes!=object and emp_new_df[i].nunique()<30):
        print(i, emp_new_df[i].unique())
        num_col.append(i)
#print(num_col)
plt.figure(figsize=(20,20))
for i, column in enumerate(num_col,1):
    plt.subplot(4,4,i)
    emp_new_df[emp_new_df['Attrition']=='Yes'][column].hist(bins=36,color='green',label="Attrition=Yes",alpha=0.6)
    emp_new_df[emp_new_df['Attrition']=='No'][column].hist(bins=36,color='gold',label="Attrition=No",alpha=0.6)
    plt.xlabel(column)
    plt.legend()<jupyter_output><empty_output><jupyter_text>As we can see, environment satisfaction, job satisfaction, performance rating and relationship satisfactions don't have much impact on the attrition of employees. While percentage salary hike, years since last promotion, Job Level etc have an impact on attrition. We will study it in detail later in this notebook. Same as done above, now we are visualizing all the columns which are of object datatype and has unique values less than 30. We will be creating a new array with only these features and then visualize it all together.<jupyter_code>cat_col=[]
for i in emp_new_df.columns:
    if(emp_new_df[i].dtypes==object and emp_new_df[i].nunique()<30):
        print(i, emp_new_df[i].unique())
        cat_col.append(i)
plt.figure(figsize=(20,20))
cat_col_new= ['BusinessTravel','Gender','Department','MaritalStatus','OverTime']
for i, column in enumerate(cat_col_new,1):
    plt.subplot(3,3,i)
    emp_new_df[emp_new_df['Attrition']=='Yes'][column].hist(bins=36,color='green',label="Attrition=Yes",alpha=0.6)
    emp_new_df[emp_new_df['Attrition']=='No'][column].hist(bins=36,color='gold',label="Attrition=No",alpha=0.6)
    plt.xlabel(column)
    plt.legend()<jupyter_output><empty_output><jupyter_text>All the dimensions shown above seems to have an impact on attrition. OverTime graph clearly shows that associates who do over time have higher attrition rate. Similar observation can be made from Marital Status, it shows single employees have higher attrition rate. We will study the other dimensions in this notebook. Let's go ahead.<jupyter_code>sns.set_style("darkgrid")
plt.figure(figsize=(12, 6))
plt.title('Age')
plt.xlabel('Age')
plt.ylabel('Number of person')

plt.hist([Attr_yes_df.Age,Attr_no_df.Age], bins=np.arange(10,80,5), color=['orange','green'],stacked=True);
plt.legend(['Attrited', 'Not Attrited']);<jupyter_output><empty_output><jupyter_text>Created a heatmap to understand the correlation between all the numerical features. It shows:> - Job Level and Monthly Income has strong correlation> - Higher the performance rating, higher the salary hike.> - Total Working years and YearsAtCompany are highly correlated.> - Yearsatcompany and yearswithcurrent manager is highly correlated.> - There is very low correlation between number of companies worked and total working years.<jupyter_code>f,ax=plt.subplots(figsize=(20,20))
sns.heatmap(emp_new_df.corr(),annot=True,linewidth=.5,fmt='.1f')<jupyter_output><empty_output><jupyter_text>Exploring the employees distribution based on marital status us<jupyter_code>mar_stat=emp_new_df['MaritalStatus'].value_counts()
plt.bar(mar_stat.index,mar_stat)
plt.title('Marital Status')<jupyter_output><empty_output><jupyter_text>Asking and Answering QuestionsHave come up with few questions to do my analysis around attrition reasons. I have also tried using all the learned concepts to acieve the same.  Q1: Which department and job role has seen most attrition?<jupyter_code>Attr_yes_df=emp_df[emp_df['Attrition']=='Yes']
Attr_no_df=emp_df[emp_df['Attrition']=='No']

dept_count_yes=Attr_yes_df['EducationField'].value_counts()
dept_count_no=Attr_no_df['EducationField'].value_counts()
job_role_yes=Attr_yes_df['JobRole'].value_counts()
job_role_no=Attr_no_df['JobRole'].value_counts()
fig, axes = plt.subplots(1,2, figsize=(20, 6))

# Pie wedgeprops with width being the donut thickness
axes[0].pie((dept_count_yes*100/dept_count_no), wedgeprops=dict(width=0.6), labels=(dept_count_yes/dept_count_no).index,autopct='%1.1f%%', startangle=180);
axes[0].text(0., 0., '', horizontalalignment='center', verticalalignment='center')
axes[0].axis('equal')

#plt.show()

axes[1].pie((job_role_yes*100/job_role_no), wedgeprops=dict(width=0.6), labels=(job_role_yes/job_role_no).index,autopct='%1.1f%%', startangle=180);
axes[1].text(0., 0., '', horizontalalignment='center', verticalalignment='center')
axes[1].axis('equal')
plt.show()<jupyter_output><empty_output><jupyter_text>I have displayed the rate of attrition across department and job role. As we can see,  Human Resources and Technical Degree had most attrition rate. While Sales representative job role was most impacted by attrition. Manager, Research Director and Manufacturing Director had comparatively lesser attrition rate. I have calculated attrition rate by dividing data from attrited employees upon non attrited employees.  Q2: What is the relation between gender wise job satisfaction for all the employees?.<jupyter_code>ax=sns.boxplot(x=emp_new_df["Gender"],y=emp_new_df['JobSatisfaction'],hue=emp_new_df['Attrition'],palette="Set1",linewidth=2.5)<jupyter_output><empty_output><jupyter_text>For individuals who stayed in the organization,distribution of job satisfaction is same irrespective of gender. While amongst those who left, females had lower job satisfaction.  Q3: What is the average no of companies worked previously?<jupyter_code>def genfunc(age2):
    gen=[]
    if (age2)<37:
        gen='Millenials'
    elif age2>=37 and age2<54:
        gen='Generation X'
    elif age2>=54 and age2<73:
        gen='Boomers'
    else:
        gen='Silent'
    return(gen)
emp_new_df['Generation']=emp_new_df['Age'].apply(genfunc)
gen_df=emp_new_df.groupby('Generation')['NumCompaniesWorked'].mean()
gen_df
ax=sns.boxplot(x=emp_new_df["Generation"],y=emp_new_df['NumCompaniesWorked'],hue=emp_new_df['Attrition'],palette="Set2",linewidth=2.5,order=['Millenials','Generation X','Boomers'])<jupyter_output><empty_output><jupyter_text>Created a column called generation based on age of employees. This was done to understand if there is a difference in behaviour of young versus old generation employees. It currently shows lesser number for millenials compared to the boomers as they obviously are younger. In future, I would be studying the attrition rate of the generation category to get a better picture.  Q4: Does income parity across department have an impact on attrition?<jupyter_code>inc=Attr_yes_df.groupby(['Department'])['MonthlyIncome'].mean()

inc_att=Attr_no_df.groupby(['Department'])['MonthlyIncome'].mean()

fig, axes = plt.subplots(1,2, figsize=(20, 6))
axes[0].bar(inc.index, inc, color='brown')
axes[0].set_title('Attrited Employees')
axes[1].bar(inc_att.index, inc_att, color='gold')
axes[1].set_title('Not Attrited Employees')

As we can see, attrited employess of the same departments have lesser monthly income than employees who haven't attrited. Human Resources department has a difference of almost double monthly pay.While, Sales department doesnt have much significant pay parity.<jupyter_output><empty_output><jupyter_text>Q5: Does promotion affect attrition?<jupyter_code>yearpromo_count_df=emp_df['YearsSinceLastPromotion'].value_counts()
sns.scatterplot(emp_new_df.YearsSinceLastPromotion, yearpromo_count_df, hue=emp_new_df.Attrition, s=100)
plt.xlabel("YearsSinceLastPromotion")
plt.ylabel("No of Associates");<jupyter_output><empty_output><jupyter_text>As we can see in the above scatter plot, all the employees who attrited have got promotion in less than 1 year duration. While employees who haven't got promotion are still with the company. It shows the pattern that employees maybe wait in a company till they get promotion and start looking for opportunities ouside as soon as they get promoted.   Inferences and Conclusion> - Correlation graph showed correlation between different features. It showed how performance rating & hike is highly correlated.> - Sales Representative Job Role has the highest attrition rate.> - Most employees leave job after getting promotion.> - Employees  of same department who get attrited have lesser monthly income compared to those who did not get attrited> -  Promotion impacts attrition rate: Employees who get promoted leaves sooner that those who didnt get promotion.<jupyter_code>import json
f=open('Kaggle.json',"r")
data = json.loads(f.read())
data<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10208914827032056, 'fraction_numerical': 0.0126025543432293, 'mean_word_length': 6.6879063719115734, 'pattern_counts': {'":': 0, '<': 71, '<?xml version=': 0, '>': 87, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '399215', 'license': 'MIT', 'n_tokens_mistral': 3721, 'n_tokens_neox': 3520, 'n_words': 1131, 'path': 'Code/Day 17 HR Analytics.ipynb', 'repo_name': 'chetnashahi/100daysofML', 'size': '11827'}