<jupyter_start><jupyter_text>**Assignment 1** (Intro to Machine Learning)  Note:* The dataset to be used is `attrition.csv`. This dataset reveals whether a particular employee left the company or not.  * In this dataset, the column named `Attrition` is the target variable and the remaining columns are features. Please note that `Attrition = 1` means the employee left the company, whereas `Attrition = 0` means the opposite of this statement. *  This dataset has been loaded in this assignment as shown below. In case, you are not able to load this data, please note that this data is stored separately as well in the assignment folder.   Instructions:* You are required to use a classifier, which can predict the `Attrition` for the employees. * Before implementing any model, you will have to apply suitable encoding to the features and implement exploratory data analysis to know your data better. * You can either define your own custom-made classifer or select classifier(s) available in the `scikit-learn`.* You are supposed to implement a minimum of three classifiers (e.g.  `RandomForestClassifier`, `XGBClassifier`, `DecisionTreeClassifier`, `KNeighborsClassifier`, etc.) and evaluate which one is giving the best peformance. * For each of the classifier,  report the accuracy, precision, recall, roc curve, etc.<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
data_url = "https://raw.githubusercontent.com/shala2020/shala2020.github.io/master/Lecture_Materials/Assignments/MachineLearning/L1/attrition.csv"
attrition_data = pd.read_csv(data_url)
df = attrition_data
df = df.drop('ID',axis=1)
df.head(5)
# Let's do some preprocessing in data
print("The description of data is :")
print(df.isna().sum())
print(df.describe())
# Employee count is a rdundant column let's drop it as it just has a value 1
df.EmployeeCount.unique()
df = df.drop('EmployeeCount',axis=1)
# Find out the correlation matrix
corr_df = df.corr()
plt.figure(figsize=(15,15))
# Creating mask
mask = np.zeros_like(corr_df, dtype=np.bool)
mask[np.triu_indices_from(mask)] = True
# Draw the heatmap 
sns.heatmap(corr_df,vmin = -1, annot=False)<jupyter_output><empty_output><jupyter_text>There exist a strong and very strong linear relationship between monthly income and job level which could lead to multicollinearity<jupyter_code># Let's drop job level for that purpose
df = df.drop('JobLevel',axis=1)
# For one hot encoding the categorical variables
def create_one_hot(df, column_name):
    """One-hot encode column values. Takes the df and column name as
    input and return the df with one-hot encoded columns as output.
    """
    df[column_name] = pd.Categorical(df[column_name])
    one_hot = pd.get_dummies(df[column_name], prefix = column_name)
    # add dummies to original df:
    df = pd.concat([one_hot, df], axis = 1)
    return df
# Getting the dummy vars to categorical data
col = df.select_dtypes(['object']).columns
for c in col:
  df = create_one_hot(df,c)
  df = df.drop(c,axis=1)<jupyter_output><empty_output><jupyter_text>Since the dataset is highly imbalanced we need to balance the dataset with estimator variables<jupyter_code># Separate majority and minority classes
data_ori_majority = df[df.Attrition==0]
data_ori_minority = df[df.Attrition==1]
 
# Downsample majority class
from sklearn.utils import resample
data_ori_majority_downsampled = resample(data_ori_majority, 
                                         replace=False,    # sample without replacement
                                         n_samples=len(data_ori_minority),     # to match minority class
                                         random_state=0) # reproducible results
 # Combine minority class with downsampled majority class
data_ori_downsampled = pd.concat([data_ori_majority_downsampled, data_ori_minority])
print(data_ori_downsampled['Attrition'].value_counts())
from sklearn.model_selection import train_test_split
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.ensemble import RandomForestClassifier
from xgboost import XGBClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import f1_score, precision_score, confusion_matrix, recall_score, accuracy_score

class classification:
  
  def __init__(self, X, y):
    
    self.X_train, self.X_test, self.y_train, self.y_test = train_test_split(X, y, test_size=0.2, random_state=42)


  def calc_metrics_class(self):

    precision = precision_score(self.pred, self.y_test)
    recall = recall_score(self.pred,self.y_test)
    f1 = f1_score(self.pred,self.y_test)
    accuracy = accuracy_score(self.pred,self.y_test)
    print("precision", precision, '\n', "recall", recall, '\n', "f1", f1, '\n', "accuracy", accuracy)

  def gradient_boost(self):
      print("Performing modelling for Gradient boost")
      GradBoostClasCV = GradientBoostingClassifier(random_state=42)
      model_params = {
                        "max_depth": [30],
                        "subsample": [ 0.9],
                        "n_estimators":[300],
                        "learning_rate": [0.03],
                        "criterion": ['mae']
                      }
      grid_model = GridSearchCV(estimator=GradBoostClasCV, param_grid=model_params, cv=5, n_jobs=-1)
      grid_model.fit(self.X_train,self.y_train)
      print("Best parameters =", grid_model.best_params_)
      model_clf = GradBoostClasCV.set_params(**grid_model.best_params_)
      model_clf.fit(self.X_train, self.y_train)
      self.pred = model_clf.predict(self.X_test)
      self.calc_metrics_class()

  def random_forest(self):
      print("Performing modelling for Random forest")
      rf_model = RandomForestClassifier(random_state=1)
      param_grid = {
                    'n_estimators': [350],
                    'max_features': [0.9],
                    'min_samples_split': [6]
                    }
      grid_model = GridSearchCV(estimator=rf_model, param_grid=param_grid, cv=5, n_jobs=-1)
      grid_model.fit(self.X_train,self.y_train)
      print("Best parameters =", grid_model.best_params_)
      model_clf = rf_model.set_params(**grid_model.best_params_)
      model_clf.fit(self.X_train, self.y_train)
      self.pred = model_clf.predict(self.X_test)
      self.calc_metrics_class()

  def decision_tree(self):
      print("Performing modelling for decision tree")
      #create a dictionary of all values we want to test
      param_grid = { 'criterion':['gini'],'max_depth': [20]}
      # decision tree model
      dtree_model=DecisionTreeClassifier()
      #use gridsearch to test all values
      grid_model = GridSearchCV(dtree_model, param_grid, cv=5, n_jobs=-1)
      grid_model.fit(self.X_train,self.y_train)
      print("Best parameters =", grid_model.best_params_)
      model_clf = dtree_model.set_params(**grid_model.best_params_)
      model_clf.fit(self.X_train, self.y_train)
      self.pred = model_clf.predict(self.X_test)
      self.calc_metrics_class()<jupyter_output><empty_output><jupyter_text>Due to memory constraints using only one hyperparameter<jupyter_code>X = data_ori_downsampled.drop('Attrition',axis=1)
y = data_ori_downsampled['Attrition']
p1 = classification(X,y)
p1.random_forest()
p1.gradient_boost()
p1.decision_tree()<jupyter_output>Best parameters = {'max_features': 0.9, 'min_samples_split': 6, 'n_estimators': 350}
precision 0.6666666666666666 
 recall 0.6470588235294118 
 f1 0.6567164179104478 
 accuracy 0.6666666666666666
Best parameters = {'criterion': 'mae', 'learning_rate': 0.03, 'max_depth': 30, 'n_estimators': 300, 'subsample': 0.9}
precision 0.5757575757575758 
 recall 0.5757575757575758 
 f1 0.5757575757575758 
 accuracy 0.5942028985507246
Best parameters = {'criterion': 'gini', 'max_depth': 20}
precision 0.6060606060606061 
 recall 0.6451612903225806 
 f1 0.625 
 accuracy 0.6521739130434783<jupyter_text>Resources:* You might want to refer to the [lecture notebook](https://github.com/shala2020/shala2020.github.io/tree/master/Lecture_Materials/Google_Colab_Notebooks/MachineLearning/L1) for revising the concepts.* You can also refer to the pre-work material for the first lecture on ML.<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0950363196125908, 'fraction_numerical': 0.034987893462469735, 'mean_word_length': 4.159900062460962, 'pattern_counts': {'":': 5, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '107500', 'license': 'MIT', 'n_tokens_mistral': 2802, 'n_tokens_neox': 2547, 'n_words': 690, 'path': 'Saurabh_ML_L1_Assignment.ipynb', 'repo_name': 'SaurabhChakravorty/SHALA2020', 'size': '8260'}