<jupyter_start><jupyter_text>Classification Tutorial - Employee Attrition  A notebook about binary classification using a modified dataset from Kaggle.https://www.kaggle.com/c/sm/data  Import data and the necessary libraries<jupyter_code>!pip install xgboost
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.metrics import confusion_matrix, classification_report, accuracy_score
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier, AdaBoostClassifier, ExtraTreesClassifier
from xgboost import XGBClassifier
from sklearn import model_selection
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import Normalizer,StandardScaler
pd.set_option('display.max_rows', 500)
pd.set_option('display.max_columns', 500)
pd.set_option('display.width', 1000)
pd.set_option("display.precision", 3)
df = pd.read_csv('Employee_Attrition.csv')
df.shape
df.head(10)
df.info()
df.describe()
df.corr()
df.isna().sum()
df.left.value_counts()<jupyter_output><empty_output><jupyter_text>We choose the dependent and the independent variables and define a repeated Stratified K-Fold cross validator.<jupyter_code>X = df.iloc[:,:-1]
y = df.left
cv = model_selection.RepeatedStratifiedKFold(n_splits=6, n_repeats=3, random_state=1)<jupyter_output><empty_output><jupyter_text>First, we try some basic alogrithms:<jupyter_code>models = [
    ('CART', DecisionTreeClassifier(max_depth=2)),
    ('LR', LogisticRegression()),
    ('SVM', SVC()),
    ('kNN', KNeighborsClassifier(n_neighbors = 7))
]

for name, clf in models:
    n_scores = model_selection.cross_val_score(clf, X, y, scoring='accuracy', cv=cv, n_jobs=-1)
    print('{}: Mean Accuracy: {:.3f} ({:.3f})'.format(name,np.mean(n_scores), np.std(n_scores)))<jupyter_output>CART: Mean Accuracy: 0.850 (0.004)
LR: Mean Accuracy: 0.766 (0.006)
SVM: Mean Accuracy: 0.784 (0.002)
kNN: Mean Accuracy: 0.951 (0.004)<jupyter_text>Then, we move on to ensemble methods:<jupyter_code>models = [
    ('RT', RandomForestClassifier()),
    ('GB', GradientBoostingClassifier()),
    ('XGB', XGBClassifier()),
    ('Ada', AdaBoostClassifier(n_estimators=100)),
    ('Extra', ExtraTreesClassifier(n_estimators=100))
]

for name, clf in models:
    n_scores = model_selection.cross_val_score(clf, X, y, scoring='accuracy', cv=cv, n_jobs=-1)
    print('{}: Mean Accuracy: {:.3f} ({:.3f})'.format(name,np.mean(n_scores), np.std(n_scores)))<jupyter_output>RT: Mean Accuracy: 0.992 (0.002)
GB: Mean Accuracy: 0.975 (0.003)
XGB: Mean Accuracy: 0.987 (0.002)
Ada: Mean Accuracy: 0.961 (0.004)
Extra: Mean Accuracy: 0.990 (0.002)<jupyter_text>We try some data preprocessing to check if we have any improvement in the results.  At first, we use data normalization:<jupyter_code>X_norm = Normalizer().fit_transform(X)
models = [
    ('RT', RandomForestClassifier()),
    ('GB', GradientBoostingClassifier()),
    ('XGB', XGBClassifier()),
    ('Ada', AdaBoostClassifier(n_estimators=100)),
    ('Extra', ExtraTreesClassifier(n_estimators=100))
]

for name, clf in models:
    n_scores = model_selection.cross_val_score(clf, X_norm, y, scoring='accuracy', cv=cv, n_jobs=-1)
    print('{}: Mean Accuracy: {:.3f} ({:.3f})'.format(name,np.mean(n_scores), np.std(n_scores)))<jupyter_output>RT: Mean Accuracy: 0.984 (0.002)
GB: Mean Accuracy: 0.956 (0.005)
XGB: Mean Accuracy: 0.980 (0.003)
Ada: Mean Accuracy: 0.937 (0.004)
Extra: Mean Accuracy: 0.984 (0.003)<jupyter_text>And then data standardization:<jupyter_code>X_scaled = StandardScaler().fit_transform(X)
models = [
    ('RT', RandomForestClassifier()),
    ('GB', GradientBoostingClassifier()),
    ('XGB', XGBClassifier()),
    ('Ada', AdaBoostClassifier(n_estimators=100)),
    ('Extra', ExtraTreesClassifier(n_estimators=100))
]

for name, clf in models:
    n_scores = model_selection.cross_val_score(clf, X_scaled, y, scoring='accuracy', cv=cv, n_jobs=-1)
    print('{}: Mean Accuracy: {:.3f} ({:.3f})'.format(name,np.mean(n_scores), np.std(n_scores)))<jupyter_output>RT: Mean Accuracy: 0.992 (0.002)
GB: Mean Accuracy: 0.975 (0.003)
XGB: Mean Accuracy: 0.987 (0.002)
Ada: Mean Accuracy: 0.961 (0.004)
Extra: Mean Accuracy: 0.990 (0.002)<jupyter_text>Afterwards, we try feature extraction using PCA method. For more information:https://en.wikipedia.org/wiki/Principal_component_analysis<jupyter_code>from sklearn.decomposition import PCA
pca = PCA(n_components=5)
X_pca = pd.DataFrame(pca.fit_transform(X))
X_pca
print(pca.explained_variance_ratio_)
models = [
    ('RT', RandomForestClassifier(n_estimators = 100)),
    ('GB', GradientBoostingClassifier()),
    ('XGB', XGBClassifier()),
    ('Ada', AdaBoostClassifier(n_estimators=100)),
    ('Extra', ExtraTreesClassifier(n_estimators=100))
]

for name, clf in models:
    n_scores = model_selection.cross_val_score(clf, X_pca, y, scoring='accuracy', cv=cv, n_jobs=-1)
    print('{}: Mean Accuracy: {:.3f} ({:.3f})'.format(name,np.mean(n_scores), np.std(n_scores)))<jupyter_output>RT: Mean Accuracy: 0.988 (0.002)
GB: Mean Accuracy: 0.969 (0.004)
XGB: Mean Accuracy: 0.985 (0.003)
Ada: Mean Accuracy: 0.940 (0.003)
Extra: Mean Accuracy: 0.987 (0.003)<jupyter_text>As we conclude from the above results, none of the aforementioned methods improves the results, therefore we choose to proceed with the initial dataset, and the algorithm that performed better (in our case Random Forest). We could also employ Grid/Random search for fine-tuning.<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(
         X, y, test_size = 0.3, random_state=7)
model = RandomForestClassifier(random_state=7,max_features='auto', n_estimators= 200, 
                            max_depth=6, criterion='entropy') 
model.fit(X_train, y_train) 
y_pred=model.predict(X_test) 
print("Accuracy: ",round(accuracy_score(y_test,y_pred),3))
print(confusion_matrix(y_test,y_pred))
print(classification_report(y_test,y_pred))<jupyter_output>precision    recall  f1-score   support

           0       0.97      1.00      0.98      3449
           1       0.99      0.90      0.94      1051

    accuracy                           0.97      4500
   macro avg       0.98      0.95      0.96      4500
weighted avg       0.97      0.97      0.97      4500
{'fraction_non_alphanumeric': 0.13495130622971094, 'fraction_numerical': 0.05023960426650178, 'mean_word_length': 5.098020735155513, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 27, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '129944', 'license': 'MIT', 'n_tokens_mistral': 2604, 'n_tokens_neox': 2322, 'n_words': 528, 'path': 'Projects/Supervised_Learning/Binary_Classification.ipynb', 'repo_name': 'Tasosk92/Machine-Learning', 'size': '6469'}