<jupyter_start><jupyter_text>Handling imbalanced data in customer churn prediction ***Customer churn prediction*** is to measure why customers are leaving a business. In this tutorial we will be looking at customer churn in telecom business. We will build a deep learning model to predict the churn and use precision,recall, f1-score to measure performance of our model. We will then handle imbalance in data using various techniques and improve f1-score<jupyter_code>import pandas as pd
from matplotlib import pyplot as plt
import numpy as np
%matplotlib inline
import warnings
warnings.filterwarnings('ignore')<jupyter_output><empty_output><jupyter_text>Load Data<jupyter_code>df = pd.read_csv("/Users/Asus/Documents/Atmel Studio/Customer_Churn.csv")
df.sample(5)

df.Churn.value_counts()
517400/df.shape[0]  #To get the Precentage<jupyter_output><empty_output><jupyter_text>First of all, drop customerID column as it is of no use<jupyter_code>df.drop('customerID',axis='columns',inplace=True)
df.dtypes<jupyter_output><empty_output><jupyter_text>***Quick glance*** at above makes me realize that TotalCharges should be float but it is an object. Let's check what's going on with this column<jupyter_code>df.TotalCharges.values<jupyter_output><empty_output><jupyter_text>Convert String to Number<jupyter_code>pd.to_numeric(df.TotalCharges,errors='coerce').isnull()
df[pd.to_numeric(df.TotalCharges,errors='coerce').isnull()]
df.shape
df.iloc[488].TotalCharges
df[df.TotalCharges!=' '].shape
df1 = df[df.TotalCharges!=' ']
df1.shape
df1.dtypes
df1.TotalCharges = pd.to_numeric(df1.TotalCharges)
df1.TotalCharges.values
df1[df1.Churn=='No']<jupyter_output><empty_output><jupyter_text>Data Vizualization<jupyter_code>tenure_churn_no = df1[df1.Churn=='No'].tenure
tenure_churn_yes = df1[df1.Churn=='Yes'].tenure

plt.xlabel("tenure")
plt.ylabel("Number Of Customers")
plt.title("Customer Churn Prediction Visualiztion")

blood_sugar_men = [113, 85, 90, 150, 149, 88, 93, 115, 135, 80, 77, 82, 129]
blood_sugar_women = [67, 98, 89, 120, 133, 150, 84, 69, 89, 79, 120, 112, 100]

plt.hist([tenure_churn_yes, tenure_churn_no], rwidth=0.95, color=['green','red'],label=['Churn=Yes','Churn=No'])
plt.legend()
mc_churn_no = df1[df1.Churn=='No'].MonthlyCharges      
mc_churn_yes = df1[df1.Churn=='Yes'].MonthlyCharges      

plt.xlabel("Monthly Charges")
plt.ylabel("Number Of Customers")
plt.title("Customer Churn Prediction Visualiztion")

blood_sugar_men = [113, 85, 90, 150, 149, 88, 93, 115, 135, 80, 77, 82, 129]
blood_sugar_women = [67, 98, 89, 120, 133, 150, 84, 69, 89, 79, 120, 112, 100]

plt.hist([mc_churn_yes, mc_churn_no], rwidth=0.95, color=['green','red'],label=['Churn=Yes','Churn=No'])
plt.legend()<jupyter_output><empty_output><jupyter_text>Many of the columns are yes, no etc. Let's print unique values in object columns to see data values<jupyter_code>def print_unique_col_values(df):
       for column in df:
            if df[column].dtypes=='object':
                print(f'{column}: {df[column].unique()}')
print_unique_col_values(df1)<jupyter_output>gender: ['Female' 'Male']
Partner: ['Yes' 'No']
Dependents: ['No' 'Yes']
PhoneService: ['No' 'Yes']
MultipleLines: ['No phone service' 'No' 'Yes']
InternetService: ['DSL' 'Fiber optic' 'No']
OnlineSecurity: ['No' 'Yes' 'No internet service']
OnlineBackup: ['Yes' 'No' 'No internet service']
DeviceProtection: ['No' 'Yes' 'No internet service']
TechSupport: ['No' 'Yes' 'No internet service']
StreamingTV: ['No' 'Yes' 'No internet service']
StreamingMovies: ['No' 'Yes' 'No internet service']
Contract: ['Month-to-month' 'One year' 'Two year']
PaperlessBilling: ['Yes' 'No']
PaymentMethod: ['Electronic check' 'Mailed check' 'Bank transfer (automatic)'
 'Credit card (automatic)']
Churn: ['No' 'Yes']<jupyter_text>Some of the columns have no internet service or no phone service, that can be replaced with a simple No<jupyter_code>df1.replace('No internet service','No',inplace=True)
df1.replace('No phone service','No',inplace=True)

print_unique_col_values(df1)<jupyter_output>gender: ['Female' 'Male']
Partner: ['Yes' 'No']
Dependents: ['No' 'Yes']
PhoneService: ['No' 'Yes']
MultipleLines: ['No' 'Yes']
InternetService: ['DSL' 'Fiber optic' 'No']
OnlineSecurity: ['No' 'Yes']
OnlineBackup: ['Yes' 'No']
DeviceProtection: ['No' 'Yes']
TechSupport: ['No' 'Yes']
StreamingTV: ['No' 'Yes']
StreamingMovies: ['No' 'Yes']
Contract: ['Month-to-month' 'One year' 'Two year']
PaperlessBilling: ['Yes' 'No']
PaymentMethod: ['Electronic check' 'Mailed check' 'Bank transfer (automatic)'
 'Credit card (automatic)']
Churn: ['No' 'Yes']<jupyter_text>Convert Yes/No Columns into binary values<jupyter_code>yes_no_columns = ['Partner','Dependents','PhoneService','MultipleLines','OnlineSecurity','OnlineBackup',
                  'DeviceProtection','TechSupport','StreamingTV','StreamingMovies','PaperlessBilling','Churn']
for col in yes_no_columns:
    df1[col].replace({'Yes': 1,'No': 0},inplace=True)

for col in df1:
    print(f'{col}: {df1[col].unique()}')
df1['gender'].replace({'Female':1,'Male':0},inplace=True)
df1.gender.unique()<jupyter_output><empty_output><jupyter_text>One hot encoding for categorical columns<jupyter_code>df2 = pd.get_dummies(data=df1, columns=['InternetService','Contract','PaymentMethod'])
df2.columns
df2.sample(5)
df2.dtypes
cols_to_scale = ['tenure','MonthlyCharges','TotalCharges']

from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()
df2[cols_to_scale] = scaler.fit_transform(df2[cols_to_scale])
for col in df2:
    print(f'{col}: {df2[col].unique()}')<jupyter_output>gender: [1 0]
SeniorCitizen: [0 1]
Partner: [1 0]
Dependents: [0 1]
tenure: [0.         0.46478873 0.01408451 0.61971831 0.09859155 0.29577465
 0.12676056 0.38028169 0.85915493 0.16901408 0.21126761 0.8028169
 0.67605634 0.33802817 0.95774648 0.71830986 0.98591549 0.28169014
 0.15492958 0.4084507  0.64788732 1.         0.22535211 0.36619718
 0.05633803 0.63380282 0.14084507 0.97183099 0.87323944 0.5915493
 0.1971831  0.83098592 0.23943662 0.91549296 0.11267606 0.02816901
 0.42253521 0.69014085 0.88732394 0.77464789 0.08450704 0.57746479
 0.47887324 0.66197183 0.3943662  0.90140845 0.52112676 0.94366197
 0.43661972 0.76056338 0.50704225 0.49295775 0.56338028 0.07042254
 0.04225352 0.45070423 0.92957746 0.30985915 0.78873239 0.84507042
 0.18309859 0.26760563 0.73239437 0.54929577 0.81690141 0.32394366
 0.6056338  0.25352113 0.74647887 0.70422535 0.35211268 0.53521127]
PhoneService: [0 1]
MultipleLines: [0 1]
OnlineSecurity: [0 1]
OnlineBackup: [1 0]
DeviceProtection: [0 1]
TechSupport: [[...]<jupyter_text>Train Test Data<jupyter_code>X = df2.drop('Churn',axis='columns')
y = testLabels = df2.Churn.astype(np.float32)

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=15, stratify=y)
y_train.value_counts()
y.value_counts()
#Finfing Churn Ratio in Test Data
5163/1869  
y_test.value_counts()
X_train.shape
X_test.shape
X_train[:10]
len(X_train.columns)<jupyter_output><empty_output><jupyter_text>Build a model (ANN) in tensorflow/keras<jupyter_code>from tensorflow_addons import losses
import tensorflow as tf
from tensorflow import keras
from sklearn.metrics import confusion_matrix , classification_report
def ANN(X_train, y_train, X_test, y_test, loss, weights):
    model = keras.Sequential([
        keras.layers.Dense(26, input_dim=26, activation='relu'),
        keras.layers.Dense(15, activation='relu'),
        keras.layers.Dense(1, activation='sigmoid')
    ])

    model.compile(optimizer='adam', loss=loss, metrics=['accuracy'])
    
    if weights == -1:
        model.fit(X_train, y_train, epochs=100)
    else:
        model.fit(X_train, y_train, epochs=100, class_weight = weights)
    
    print(model.evaluate(X_test, y_test))
    
    y_preds = model.predict(X_test)
    y_preds = np.round(y_preds)
    
    print("Classification Report: \n", classification_report(y_test, y_preds))
    
    return y_preds
y_preds = ANN(X_train, y_train, X_test, y_test, 'binary_crossentropy', -1)<jupyter_output>Epoch 1/100
176/176 [==============================] - 7s 11ms/step - loss: 0.4899 - accuracy: 0.7648
Epoch 2/100
176/176 [==============================] - 2s 12ms/step - loss: 0.4269 - accuracy: 0.7941
Epoch 3/100
176/176 [==============================] - 2s 11ms/step - loss: 0.4197 - accuracy: 0.7982
Epoch 4/100
176/176 [==============================] - 2s 12ms/step - loss: 0.4148 - accuracy: 0.7986
Epoch 5/100
176/176 [==============================] - 2s 12ms/step - loss: 0.4123 - accuracy: 0.7980
Epoch 6/100
176/176 [==============================] - 2s 11ms/step - loss: 0.4097 - accuracy: 0.8021
Epoch 7/100
176/176 [==============================] - 2s 11ms/step - loss: 0.4077 - accuracy: 0.8025
Epoch 8/100
176/176 [==============================] - 2s 12ms/step - loss: 0.4066 - accuracy: 0.8032
Epoch 9/100
176/176 [==============================] - 2s 12ms/step - loss: 0.4060 - accuracy: 0.8053
Epoch 10/100
176/176 [==============================] - 2s 10ms/step - loss: 0.404[...]<jupyter_text>Mitigating Skewness  Method 1: Undersampling **Reference:*** https://www.kaggle.com/rafjaa/resampling-strategies-for-imbalanced-datasets<jupyter_code># Class count
count_class_0, count_class_1 = df1.Churn.value_counts()

# Divide by class
df_class_0 = df2[df2['Churn'] == 0]
df_class_1 = df2[df2['Churn'] == 1]
# Undersample 0-class and concat the DataFrames of both class
df_class_0_under = df_class_0.sample(count_class_1)
df_test_under = pd.concat([df_class_0_under, df_class_1], axis=0)

print('Random under-sampling:')
print(df_test_under.Churn.value_counts())
X = df_test_under.drop('Churn',axis='columns')
y = df_test_under['Churn']

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=15, stratify=y)
# Number of classes in training Data
y_train.value_counts()<jupyter_output><empty_output><jupyter_text>Printing Classification in the last, Scroll down till the last epoch to watch the classification report<jupyter_code>y_preds = ANN(X_train, y_train, X_test, y_test, 'binary_crossentropy', -1)<jupyter_output>Epoch 1/100
94/94 [==============================] - 1s 6ms/step - loss: 0.6514 - accuracy: 0.6435
Epoch 2/100
94/94 [==============================] - 1s 6ms/step - loss: 0.5298 - accuracy: 0.7548
Epoch 3/100
94/94 [==============================] - 1s 6ms/step - loss: 0.4916 - accuracy: 0.7615
Epoch 4/100
94/94 [==============================] - 1s 5ms/step - loss: 0.4859 - accuracy: 0.7599
Epoch 5/100
94/94 [==============================] - 1s 7ms/step - loss: 0.4816 - accuracy: 0.7736
Epoch 6/100
94/94 [==============================] - 1s 6ms/step - loss: 0.4800 - accuracy: 0.7659
Epoch 7/100
94/94 [==============================] - 1s 6ms/step - loss: 0.4766 - accuracy: 0.7679
Epoch 8/100
94/94 [==============================] - 1s 5ms/step - loss: 0.4735 - accuracy: 0.7666
Epoch 9/100
94/94 [==============================] - 1s 6ms/step - loss: 0.4737 - accuracy: 0.7729
Epoch 10/100
94/94 [==============================] - 1s 6ms/step - loss: 0.4695 - accuracy: 0.7719
Epoch 11/[...]<jupyter_text>Check classification report above. f1-score for minority class 1 improved from **0.57 to 0.76.** Score for class 0 reduced to 0.75 from 0.85 but that's ok. We have more generalized classifier which classifies both classes with similar prediction score  Method2: Oversampling<jupyter_code># Oversample 1-class and concat the DataFrames of both classes
df_class_1_over = df_class_1.sample(count_class_0, replace=True)
df_test_over = pd.concat([df_class_0, df_class_1_over], axis=0)

print('Random over-sampling:')
print(df_test_over.Churn.value_counts())
X = df_test_over.drop('Churn',axis='columns')
y = df_test_over['Churn']

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=15, stratify=y)
# Number of classes in training Data
y_train.value_counts()
loss = keras.losses.BinaryCrossentropy()
weights = -1
y_preds = ANN(X_train, y_train, X_test, y_test, 'binary_crossentropy', -1)<jupyter_output>Epoch 1/100
259/259 [==============================] - 2s 6ms/step - loss: 0.5630 - accuracy: 0.7081
Epoch 2/100
259/259 [==============================] - 2s 6ms/step - loss: 0.4979 - accuracy: 0.7548
Epoch 3/100
259/259 [==============================] - 2s 6ms/step - loss: 0.4895 - accuracy: 0.7607
Epoch 4/100
259/259 [==============================] - 2s 6ms/step - loss: 0.4854 - accuracy: 0.7608
Epoch 5/100
259/259 [==============================] - 2s 6ms/step - loss: 0.4793 - accuracy: 0.7650
Epoch 6/100
259/259 [==============================] - 2s 6ms/step - loss: 0.4769 - accuracy: 0.7699
Epoch 7/100
259/259 [==============================] - 2s 6ms/step - loss: 0.4733 - accuracy: 0.7714
Epoch 8/100
259/259 [==============================] - 2s 6ms/step - loss: 0.4702 - accuracy: 0.7719
Epoch 9/100
259/259 [==============================] - 2s 6ms/step - loss: 0.4680 - accuracy: 0.7735
Epoch 10/100
259/259 [==============================] - 1s 5ms/step - loss: 0.4651 - accura[...]<jupyter_text>Check classification report above. f1-score for minority class 1 improved from 0.57 to 0.76. Score for class 0 reduced to **0.75 from 0.85** but that's ok. We have more generalized classifier which classifies both classes with similar prediction score  Method3: SMOTE To install imbalanced-learn library use **pip install imbalanced-learn** command<jupyter_code>X = df2.drop('Churn',axis='columns')
y = df2['Churn']
from imblearn.over_sampling import SMOTE

smote = SMOTE(sampling_strategy='minority')
X_sm, y_sm = smote.fit_resample(X, y)

y_sm.value_counts()
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X_sm, y_sm, test_size=0.2, random_state=15, stratify=y_sm)
# Number of classes in training Data
y_train.value_counts()
y_preds = ANN(X_train, y_train, X_test, y_test, 'binary_crossentropy', -1)<jupyter_output>Epoch 1/100
259/259 [==============================] - 2s 6ms/step - loss: 0.5381 - accuracy: 0.7237
Epoch 2/100
259/259 [==============================] - 1s 6ms/step - loss: 0.4683 - accuracy: 0.7753
Epoch 3/100
259/259 [==============================] - 1s 5ms/step - loss: 0.4600 - accuracy: 0.7799
Epoch 4/100
259/259 [==============================] - 2s 6ms/step - loss: 0.4550 - accuracy: 0.7863
Epoch 5/100
259/259 [==============================] - 1s 6ms/step - loss: 0.4496 - accuracy: 0.7914
Epoch 6/100
259/259 [==============================] - 2s 6ms/step - loss: 0.4448 - accuracy: 0.7923
Epoch 7/100
259/259 [==============================] - 1s 5ms/step - loss: 0.4409 - accuracy: 0.7937
Epoch 8/100
259/259 [==============================] - 1s 5ms/step - loss: 0.4377 - accuracy: 0.7961
Epoch 9/100
259/259 [==============================] - 1s 5ms/step - loss: 0.4343 - accuracy: 0.7988
Epoch 10/100
259/259 [==============================] - 1s 5ms/step - loss: 0.4324 - accura[...]<jupyter_text>SMOT Oversampling increases f1 score of minority class 1 from **0.57 to 0.81** (huge improvement) Also over all accuracy improves from 0.78 to 0.80  Method4: Use of Ensemble with undersampling<jupyter_code>df2.Churn.value_counts()
# Regain Original features and labels
X = df2.drop('Churn',axis='columns')
y = df2['Churn']
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=15, stratify=y)
y_train.value_counts()
df3 = X_train.copy()
df3['Churn'] = y_train
df3.head()
df3_class0 = df3[df3.Churn==0]
df3_class1 = df3[df3.Churn==1]
def get_train_batch(df_majority, df_minority, start, end):
    df_train = pd.concat([df_majority[start:end], df_minority], axis=0)

    X_train = df_train.drop('Churn', axis='columns')
    y_train = df_train.Churn
    return X_train, y_train
X_train, y_train = get_train_batch(df3_class0, df3_class1, 0, 1495)

y_pred1 = ANN(X_train, y_train, X_test, y_test, 'binary_crossentropy', -1)
X_train, y_train = get_train_batch(df3_class0, df3_class1, 1495, 2990)

y_pred2 = ANN(X_train, y_train, X_test, y_test, 'binary_crossentropy', -1)

X_train, y_train = get_train_batch(df3_class0, df3_class1, 2990, 4130)

y_pred3 = ANN(X_train, y_train, X_test, y_test, 'binary_crossentropy', -1)
len(y_pred1)
y_pred_final = y_pred1.copy()
for i in range(len(y_pred1)):
    n_ones = y_pred1[i] + y_pred2[i] + y_pred3[i]
    if n_ones>1:
        y_pred_final[i] = 1
    else:
        y_pred_final[i] = 0
cl_rep = classification_report(y_test, y_pred_final)
print(cl_rep)<jupyter_output>precision    recall  f1-score   support

           0       0.90      0.70      0.79      1033
           1       0.49      0.79      0.60       374

    accuracy                           0.73      1407
   macro avg       0.70      0.75      0.70      1407
weighted avg       0.79      0.73      0.74      1407
{'fraction_non_alphanumeric': 0.19849791703338615, 'fraction_numerical': 0.11406442527723992, 'mean_word_length': 5.139769452449568, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 62, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '74793', 'license': 'Apache-2.0', 'n_tokens_mistral': 7639, 'n_tokens_neox': 6217, 'n_words': 1473, 'path': 'Supervised Learning/Imbalancing Data Model/Imbalance Data Handling in Customer Churn.ipynb', 'repo_name': 'Pandula1234/Data-Analysis-Intern', 'size': '17043'}