<jupyter_start><jupyter_text>**Machine Learning for Churn Prediction**Customer churn is the name for when a subscriber or a regular customer cancels his subscription or stops doing business with a company. Therefore, the [churn rate](https://en.wikipedia.org/wiki/Churn_rate) is the measure of how many people stopped being a client of the company ina determined time period.In business administration, churn is a very important [metric](http://thinkapps.com/blog/post-launch/customer-churn-most-important-metric/) of how well the business is doing. If the churn rate is high, then the business is losing a lot of clients and not performing well.![](https://i.imgur.com/0XFmvWc.jpg)Business card photo created by rawpixel.com - www.freepik.comWith the evolution of machine learning algorithms and data science, [churn prediction](https://baremetrics.com/academy/churn-prediction-can-improve-business) has become a very important part of every company's strategy. If a company can accurately predict that a customer is about to churn, it can then act to prevent the churn. Usually working to keep a client is cheaper than working to get a new client.In this project, we'll work with a churn prediction [dataset](https://raw.githubusercontent.com/carlosfab/dsnp2/master/datasets/WA_Fn-UseC_-Telco-Customer-Churn.csv) of a phone/internet company from the [IBM Developer platform](https://developer.ibm.com/technologies/data-science/patterns/predict-customer-churn-using-watson-studio-and-jupyter-notebooks/). The main goal is to build a machine learning model capable of accurately predict that a customer will churn based on the information available in the dataset. In order to accomplish that, we'll go through some main steps, such as:* Exploratory data analysis;* Data preparation;* Train, tune, test, and evaluate machine learning models.  **Exploratory Data Analysis**We'll first import everything we'll need and configure the notebook. Then we'll read the data into a DataFrame and display its first five rows.<jupyter_code># Importing
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import train_test_split
from imblearn.under_sampling import RandomUnderSampler
from sklearn.preprocessing import StandardScaler
from xgboost import XGBClassifier
from sklearn.model_selection import cross_validate
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.svm import SVC
from sklearn.model_selection import StratifiedKFold
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix

# Configuring the notebook
%matplotlib inline
sns.set()

# Reading the data
churn = pd.read_csv('https://raw.githubusercontent.com/carlosfab/dsnp2/master/datasets/WA_Fn-UseC_-Telco-Customer-Churn.csv')
churn.head()<jupyter_output><empty_output><jupyter_text>We can see that the first column is an individual key. Also, the majority of the columns contain categorical data. Let's use the `info` method to get more information about the data.<jupyter_code>churn.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 7043 entries, 0 to 7042
Data columns (total 21 columns):
 #   Column            Non-Null Count  Dtype  
---  ------            --------------  -----  
 0   customerID        7043 non-null   object 
 1   gender            7043 non-null   object 
 2   SeniorCitizen     7043 non-null   int64  
 3   Partner           7043 non-null   object 
 4   Dependents        7043 non-null   object 
 5   tenure            7043 non-null   int64  
 6   PhoneService      7043 non-null   object 
 7   MultipleLines     7043 non-null   object 
 8   InternetService   7043 non-null   object 
 9   OnlineSecurity    7043 non-null   object 
 10  OnlineBackup      7043 non-null   object 
 11  DeviceProtection  7043 non-null   object 
 12  TechSupport       7043 non-null   object 
 13  StreamingTV       7043 non-null   object 
 14  StreamingMovies   7043 non-null   object 
 15  Contract          7043 non-null   object 
 16  PaperlessBilling  7043 non-null   object 
[...]<jupyter_text>From this output we know:* The DataFrame contains 7,043 rows and 21 columns;* There are no missing values;* The `TotalCharges` column is listed as object but it should be listed as float.Now, let's see how many unique values each column contains. This is important so we can know more about the categorical variables.<jupyter_code>churn.nunique()<jupyter_output><empty_output><jupyter_text>There are seven columns with only two unique values, including the `Churn` column, the target variable. We'll call these columns binaries. The other categorical columns contain either three or four unique values.  **Data Cleaning**Let's now address the problem in the `TotalCharges` column. If we ran the code below, and error would be raised informing that some values in this column are filled with whitespaces. Let's then run the next cell and see how many of these values are stored in the column.<jupyter_code>#churn['TotalCharges'] = pd.to_numeric(churn['TotalCharges'])
n_missing = len(churn[churn['TotalCharges'] == ' '])
print(f'Number of missing values in the TotalCharges column: {n_missing}')<jupyter_output>Number of missing values in the TotalCharges column: 11<jupyter_text>Eleven rows present this problem. In the next cell we will:* Change the whitespaces to null values using `np.nan`;* Convert the column to float;* Use the column's median to fill the null values;* Check if the values are filled properly and if the values in the column are in the right format.* Drop the `customerID` column.<jupyter_code># Changing the whitespaces to null values using np.nan
churn.loc[churn['TotalCharges'] == ' ', 'TotalCharges'] = np.nan

# Converting the column to float
churn['TotalCharges'] = churn['TotalCharges'].astype(float)

# Using the column's median to fill the null values
churn['TotalCharges'].fillna(churn['TotalCharges'].median(), inplace=True)

# Checking the values and the format
new_missing = len(churn[churn['TotalCharges'] == ' '])
print(f'Number of missing values in the TotalCharges column: {new_missing}')
print(f'Value type: {churn.TotalCharges.dtypes}')

# Dropping the customerID column
churn.drop('customerID', axis=1, inplace=True)<jupyter_output>Number of missing values in the TotalCharges column: 0
Value type: float64<jupyter_text>The `PaymentMethod` column contains four unique values:* Electronic check    * Mailed check        * Bank transfer (automatic)      * Credit card (automatic) We'll remove the string `(automatic)` from the values only so it's easier to understand the charts we'll plot.<jupyter_code>churn['PaymentMethod'] = churn['PaymentMethod'].str.replace('(automatic)', '').str.replace('(', '').str.replace(')', '').str.strip()
churn['PaymentMethod'].value_counts()<jupyter_output><empty_output><jupyter_text>**Analysing the Data**We'll now plot a bar chart for each categorical column so we can see how each category in these columns impacts the Churn column.Before plotting, we'll transform the values in the `Churn` column from labels `Yes` and `No` to numbers 0 and 1. We'll do the opposite with the `SeniorCitizen` column so the column's chart is properly labeled.We'll then create three lists:* One list containing the binary variables, except the `Churn`;* One list containing the other categorical variables;* A third list that will be the addition of the first two.The third list will be used to plot the charts and the first two we'll be used later on this project to encode the categorical variables.<jupyter_code># Transforming the values in the Churn and SeniorCitizen columns
churn.loc[churn['Churn'] == 'No','Churn'] = 0 
churn.loc[churn['Churn'] == 'Yes','Churn'] = 1
churn['Churn'] = churn['Churn'].astype(int)
churn.loc[churn['SeniorCitizen'] == 0,'SeniorCitizen'] = 'No' 
churn.loc[churn['SeniorCitizen'] == 1,'SeniorCitizen'] = 'Yes'

# Creating the lists
binary = [col for col in churn.columns if len(churn[col].unique()) == 2 and col != 'Churn']
categorical = [col for col in churn.columns if 2 < len(churn[col].unique()) < 5]
bin_cat = binary + categorical

# Plotting
fig = plt.figure(figsize=(20,20))
    
for i in range(len(bin_cat)):
    churn_plot = churn.groupby(bin_cat[i]).Churn.mean()
    ax = fig.add_subplot(5, 4, i+1)
    rects = ax.bar(churn_plot.index, churn_plot, color=sns.color_palette('Set1'), alpha=0.6)
    ax.grid(False)
    ax.set_ylim(0,1)
    ax.tick_params(labelleft=False)
    ax.set_title(f'{bin_cat[i].title()} vs Churn', alpha=0.5)
    
    for rect in (rects):
        height = rect.get_height()
        ax.text(x=rect.get_x() + rect.get_width() / 2, y=height * 0.5, s=round(height, 2), ha='center', va='center', alpha=0.5, fontweight='bold')

plt.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>We can learn a lot from these charts. Here are some insights:* Customers without dependents are two times more likely to churn.* Customers that use paperless billing and optical fiber are more likely to churn. * Customers with no online security or backup, no device protection, and no tech support are from two to three times more likely to churn.* Customers with no internet service are unlikely to churn.* Customers with month-to-month contracts are almost four times more likely to churn than customers with yearly contracts. Two-year contractors are very unlikely to churn.* Customers that use electronic checks to pay their bills are more likely to churn.We'll now use scatter plots to perform the same analysis for the numeric columns.<jupyter_code>numeric_columns = ['tenure', 'MonthlyCharges', 'TotalCharges']

fig = plt.figure(figsize=(20, 5))

for i in range(len(numeric_columns)):
    num_plot = churn.groupby(numeric_columns[i]).Churn.mean()
    ax = fig.add_subplot(1, 3, i+1)
    ax.scatter(num_plot.index, num_plot)
    ax.set_title(f'{numeric_columns[i]} vs Churn')
    ax.set_ylim(-0.05, 1.05)

plt.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>We can see there is a significant correlation between the `tenure` and `Churn` columns. The highest the tenure, the lowest the chances that the customer will churn.Unfortunately, there's no much we can say about the `MonthlyCharges` and `TotalCharges` columns.Finally, let's see how balanced the dataset is and decide whether or not to balance it.<jupyter_code>fig, ax = plt.subplots(figsize=(10,8))
sns.countplot(churn['Churn'])
plt.show()<jupyter_output><empty_output><jupyter_text>The dataset is not properly balanced, but it is no highly unbalanced either. Therefore, we'll train machine learning models using unbalanced and balanced data and see which data presents the best results. **Preparing the Data**We'll now prepare the data to train the models. First, we need to encode the categorical variables. For the binary variables, we'll use the `LabelEncoder` class and for the other categorical variables we'll use the `pandas.get_dummies` function.<jupyter_code>for col in binary:
    le = LabelEncoder().fit(churn[col])
    churn[col] = le.transform(churn[col])

churn = pd.get_dummies(churn, columns=categorical)

churn.head()<jupyter_output><empty_output><jupyter_text>Before training the models, we need to split the data between train and test. In the next cell, we'll also use the `RandomUnderSampler` class to balance the dataset. We'll then have both balanced and unbalanced training sets.<jupyter_code>X = churn.drop('Churn', axis=1)
y = churn['Churn']

X_train, X_test, y_train, y_test = train_test_split(X, y)

rus = RandomUnderSampler()

X_rus, y_rus = rus.fit_sample(X_train, y_train)

fig, ax = plt.subplots(figsize=(8,5))
sns.countplot(y_rus)

plt.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>One last step before training the models, we'll standardize the data using the `StandardScaler` class.<jupyter_code>scaler = StandardScaler()
scaler.fit(X_train)
X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)
X_rus = scaler.transform(X_rus)<jupyter_output><empty_output><jupyter_text>**Models and Metrics**We'll use four different machines learning algorithms:* [Logistic Regression](https://en.wikipedia.org/wiki/Logistic_regression);* [Decision Trees](https://towardsdatascience.com/decision-trees-in-machine-learning-641b9c4e8052);* [Support Vector Machines - SVM](https://en.wikipedia.org/wiki/Support_vector_machine);* [XGBoost](https://pt.wikipedia.org/wiki/Xgboost).Each of the algorithms we'll be trained using both balanced and unbalanced data so we can see which algorithm-data combination yields the bests results. **Metrics**The most important metric we'll use is [Recall](https://en.wikipedia.org/wiki/Precision_and_recallRecall). This metric indicates the proportion of positive results yielded by the model by the total number of positive labels in the dataset. In this case, the Recall reveals the proportion of churns identified correctly by the total number of churns.Recall is calculated using the following equation:$$\frac{TP}{TP+FN}$$Where:* TP: True Positives;* FN: False  Negatives.We'll also keep track of [Precision](https://en.wikipedia.org/wiki/Precision_and_recallPrecision) as a secondary metric. Precision indicates the proportion of positives yielded by the models that are actually true positives.For the problem we are dealing with, Recall is more important because it's preferable to have a model that does not miss any churns but sometimes classify a non-churns as churns, than a model that does not classify non-churns as churns but misses a lot of churns. In other words, we rather be incorrect when classifying a non-churning costumer than when classifying a churning customer.We'll now train all the models using [cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html) and store all the results in a DataFrame so we can better visualize them.<jupyter_code># Training the models
logreg_balanced = cross_validate(LogisticRegression(), X_rus, y_rus, cv=5, scoring=['recall', 'precision'])
logreg_unbalanced = cross_validate(LogisticRegression(), X_train, y_train, cv=5, scoring=['recall', 'precision'])

trees_balanced = cross_validate(DecisionTreeClassifier(), X_rus, y_rus, cv=5, scoring=['recall', 'precision'])
trees_unbalanced = cross_validate(DecisionTreeClassifier(), X_train, y_train, cv=5, scoring=['recall', 'precision'])

svm_balanced = cross_validate(SVC(), X_rus, y_rus, cv=5, scoring=['recall', 'precision'])
svm_unbalanced = cross_validate(SVC(),  X_train, y_train, cv=5, scoring=['recall', 'precision'])

xgb_balanced = cross_validate(XGBClassifier(), X_rus, y_rus, cv=5, scoring=['recall', 'precision'])
xgb_unbalanced = cross_validate(XGBClassifier(), X_train, y_train, cv=5, scoring=['recall', 'precision'])
# Storing the results in a DataFrame
summary = pd.DataFrame({
            'labels': ['recall', 'precision'],
            'logreg_balanced': [logreg_balanced['test_recall'].mean(), logreg_balanced['test_precision'].mean()],
            'logreg_unbalanced': [logreg_unbalanced['test_recall'].mean(), logreg_unbalanced['test_precision'].mean()],
            'trees_balanced': [trees_balanced['test_recall'].mean(), trees_balanced['test_precision'].mean()],
            'trees_unbalanced': [trees_unbalanced['test_recall'].mean(), trees_unbalanced['test_precision'].mean()],
            'svm_balanced': [svm_balanced['test_recall'].mean(), svm_balanced['test_precision'].mean()],
            'svm_unbalanced': [svm_unbalanced['test_recall'].mean(), svm_unbalanced['test_precision'].mean()],
            'xgb_balanced': [xgb_balanced['test_recall'].mean(), xgb_balanced['test_precision'].mean()],
            'xgb_unbalanced': [xgb_unbalanced['test_recall'].mean(), xgb_unbalanced['test_precision'].mean()]           
}
).set_index('labels')
summary.index.name=None
     
summary.style.applymap(lambda x: 'background-color: lightgreen' if x >= 0.7 else '')<jupyter_output><empty_output><jupyter_text>**Hyperparameters Tunning**We can see from the DataFrame above that the models that used balanced data yielded better results. So we'll stick with that balanced data.SVM, Logistic Regression, and XGBoost provided similar results in both metrics. Therefore, we'll tune hyperparameters using [grid search](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html?highlight=grid%20searchsklearn.model_selection.GridSearchCV) in all these three models. For that, we'll only considerer Recall for evaluation.Let's start by tunning the `kernel` and `C` in the SVM model.<jupyter_code>kfold = StratifiedKFold(n_splits=5, shuffle=True)

svm = SVC()

param_grid = {'kernel': ['linear', 'poly', 'rbf', 'sigmoid'],
              'C': [0.001, 0.01, 1, 10, 100]
              }

search = GridSearchCV(svm, param_grid, scoring='recall', cv=kfold)
result = search.fit(X_rus, y_rus)

print(f'Best recall: {result.best_score_} for {result.best_params_}')<jupyter_output>Best recall: 0.9306468185349471 for {'C': 0.01, 'kernel': 'poly'}<jupyter_text>There was a great improvement in Recall after the SVM model was tuned.Now let's tune `solver` and `C` in the Logistic Regression model.<jupyter_code>log_reg = LogisticRegression()

param_grid = {'solver': ['newton-cg', 'lbfgs', 'liblinear'],
              'C': [0.001, 0.01, 1, 10, 100]}

search = GridSearchCV(log_reg, param_grid, scoring='recall', cv=kfold)
result = search.fit(X_rus, y_rus)

print(f'Best recall: {result.best_score_} for {result.best_params_}')<jupyter_output>Best recall: 0.8344084404681352 for {'C': 0.001, 'solver': 'liblinear'}<jupyter_text>The Logistic Regression model presented a slight improvement after tuned. Not as good as the SVM model, though.We'll now tune the XGBoost model. XGBoost has lots of hyperparameters. Because of this, we'll split the process into four steps.Considering the `learning_rate` equals to 0.1, we'll tune `n_estimators`, then we'll tune `max_depth` and `min_child_weight` together. After this, we'll tune `gamma` and finally tune the `learning_rate`.<jupyter_code>xgb = XGBClassifier(learning_rate=0.1)

param_grid = {'n_estimators':range(0,1000,50)}

search = GridSearchCV(xgb, param_grid, scoring='recall', cv=kfold)
result = search.fit(X_rus, y_rus)

print(f'Best recall: {result.best_score_} for {result.best_params_}')
xgb = XGBClassifier(learning_rate=0.1, n_estimators=100)

param_grid = {'max_depth':range(1,8,1),
              'min_child_weight':range(1,5,1)
}

search = GridSearchCV(xgb, param_grid, scoring='recall', cv=kfold)
result = search.fit(X_rus, y_rus)

print(f'Best recall: {result.best_score_} for {result.best_params_}')
xgb = XGBClassifier(learning_rate=0.1, n_estimators=100, max_depth=1 , min_child_weight=1)

param_grid = {'gamma':[i/10.0 for i in range(0,5)]}

search = GridSearchCV(xgb, param_grid, scoring='recall', cv=kfold)
result = search.fit(X_rus, y_rus)

print(f'Best recall: {result.best_score_} for {result.best_params_}')
xgb = XGBClassifier(n_estimators=100, max_depth=1 , min_child_weight=1, gamma=0)

param_grid = {'learning_rate':[0.001, 0.01, 0.1, 1]}

search = GridSearchCV(xgb, param_grid, scoring='recall', cv=kfold)
result = search.fit(X_rus, y_rus)

print(f'Best recall: {result.best_score_} for {result.best_params_}')<jupyter_output>Best recall: 0.8853720271658773 for {'learning_rate': 0.001}<jupyter_text>There was a considerable improvement in the XGBoost model as well, but SVM still provides the best Recall.We'll now use the test set to evaluate these two models and see if there's any significant difference from the results yielded by the models during training.<jupyter_code>model_xgb = XGBClassifier(learning_rate=0.001, n_estimators=100, max_depth=1 , min_child_weight=1, gamma=0)
model_xgb.fit(X_rus, y_rus)
y_pred_xgb = model_xgb.predict(X_test)

print(classification_report(y_test, y_pred_xgb))

fig, ax = plt.subplots()
sns.heatmap(confusion_matrix(y_test, y_pred_xgb, normalize='true'), annot=True, ax=ax)
ax.set_title('Confusion Matrix')
ax.set_ylabel('True')
ax.set_xlabel('Predicted')

plt.show()
model_svm = SVC(kernel='poly', C=0.01)
model_svm.fit(X_rus, y_rus)
y_pred_svm = model_svm.predict(X_test)

print(classification_report(y_test, y_pred_svm))

fig, ax = plt.subplots()
sns.heatmap(confusion_matrix(y_test, y_pred_svm, normalize='true'), annot=True, ax=ax)
ax.set_title('Confusion Matrix')
ax.set_ylabel('True')
ax.set_xlabel('Predicted')

plt.show()<jupyter_output>precision    recall  f1-score   support

           0       0.95      0.38      0.54      1305
           1       0.35      0.94      0.51       456

    accuracy                           0.53      1761
   macro avg       0.65      0.66      0.52      1761
weighted avg       0.79      0.53      0.53      1761
{'fraction_non_alphanumeric': 0.10617307600210718, 'fraction_numerical': 0.019299841961591877, 'mean_word_length': 5.139958835636578, 'pattern_counts': {'":': 0, '<': 65, '<?xml version=': 0, '>': 64, 'https://': 14, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '267875', 'license': 'MIT', 'n_tokens_mistral': 6896, 'n_tokens_neox': 6386, 'n_words': 2078, 'path': 'Machine Learning for Churn Prediction.ipynb', 'repo_name': 'kuleafenu/data_science', 'size': '20881'}