<jupyter_start><jupyter_text>**Predicting POS Handler Churn**In This study we aim to predict Churn from the data set of POS Clients in a  Bank, we aim to accomplish the following:* Classify if a customer is going to churn or not* Determine the business sector of customer with high churning probability* To determine if customers who maintain high balance are likely to churn* To find out whether Clients benefitting from the bank services are likely to churn **EDA (Exploratory Data Analysis)**<jupyter_code>#pip install seaborn
#pip install chardet
## REQUIRED LIBRARIES
# For data wrangling
import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)
import os
#print(os.listdir("../input"))

# For Visualization
import matplotlib.pyplot as plt
import seaborn as sns#visualization
sns.set(style="ticks", color_codes=True)
import matplotlib.ticker as mtick # For specifying the axes tick format 
import plotly.offline as py#visualization
py.init_notebook_mode(connected=True)#visualization
import plotly.graph_objs as go#visualization
import plotly.tools as tls#visualization
import plotly.figure_factory as ff#visualization
#import urllib, chardet
#u = chardet.UniversalDetector()

#Import dataset into and display head of your dataset

churny = 'churn.csv'
#churn = pd.read_csv(churny)

import chardet
with open(churny, 'rb') as rawdata:
    result = chardet.detect(rawdata.read(100000))
result
churn = pd.read_csv(churny, encoding='ISO-8859-1')
#churn.shape()
churn.head()<jupyter_output><empty_output><jupyter_text>As said understanding the relation of things, people and business metrics all plays a big to understand those small obvious patterns  before the core exploratory phase,thats why  some of the attributes  have almost no impact into our results such as Client Name, Client ID,  Serial Number,... as long as we have an index(counts) it will serve as an ID.<jupyter_code>#drop unnecessary attributes and display new dataset
churn = churn.drop(["S/N","ClientName", "ClientID", "MerchantID", "TerminalID" ], axis =1)

#Describe how big is out dataset helps to understand how big will be our analysis and requirements.
print("Rows : ",churn.shape[0])
print("Columns  : ",churn.shape[1])
#I check if there is any NaN values that can bring biased scenario, all column attributes should return false to verify this 
churn.isnull().any()
#count our unique values without duplication of same figure

print ("\nUnique values :  \n",churn.nunique())<jupyter_output>Unique values :  
 Address            3
Tenure             8
BusinessType       3
BusinessAge       62
Balance           36
HasMB              2
AverageTrans    1368
HasOtherAcc        2
churned            2
dtype: int64<jupyter_text>How huge are our attributes? in the meantime it is quite easier to spot that Address, tenure, BusinessType, Has MobileBanking, HasOtherAccount in same Bank and Churned are categorical attributes that can corresponds to (yes/no) or 1/0 to define its state. so the rest of the attributes should be continuous attributes.<jupyter_code>#what are our data types
churn.dtypes
#Mean=> the are a lot of average calculations in statistics so i used mean the check the average possibility of attributtes to impact the situation
churn.groupby(['churned']).mean()<jupyter_output><empty_output><jupyter_text>The above Mean calculation is one of the ways to understand the average of our data, this helps us to investigate into our study whenever it has high or low measures to help to avoid biases. the difference between churned and retained are not that remarkable to give us a big picture of our study so any case can appear. But to note that:* Customers with low Balance tend to churn,reasonable!* on average older customers are the most to churn,questionable?* customers with high Average Transaction are churning because probably, they are getting attracted by other banks offers which increase their wealth or with reduced bank charges* tenure,Has Mobile Banking  and having another type of account in same bank mean are not explicitly helping in this case to hightlight anything different. Before exploring farther into our dataset,  we can see attributes correlation with churn as we have seen that mean results are not so informative, so i first change dummies variables which are Address and BusinessType in our dataset so that we can get to define correlation of attributes to churn way easy.<jupyter_code>#Let's convert all the categorical variables into dummy variables
df = pd.get_dummies(churn)
df.head()
plt.figure(figsize=(10,4))
df.corr()['churned'].sort_values(ascending = False).plot(kind='bar')<jupyter_output><empty_output><jupyter_text>Let's check the exact correlation with figures<jupyter_code>plt.figure(figsize = (20,10))
sns.heatmap((df.loc[:, ['Tenure','Balance','HasMB','HasOtherAcc','churned','BusinessAge','BusinessType_Enterprise','BusinessType_Individual','BusinessType_coperative','Address_Ikeja','Address_Ajah', 'Address_Lekki']]).corr(),
            annot=True,linewidths=.5);<jupyter_output><empty_output><jupyter_text>The above heatmap easily depict how different features correlate among themselves (including against the target feature: "Churned").Trends from the heatmap visualization:* All features have a weak or strong correlation with the target (Thus we are considering all of them for our model)* Tenure, BusinessType, Address, Balance are the features with significant correlation. **Data Exploration**This section is the core part of understanding the problem and channel it to the right features, as said before we need to establish possible relations in our attributes inorder to secure the best predictions.Note: **Churned ** will always play the role of a target<jupyter_code>labels = 'Churned', 'Retained'
sizes = [df.churned[df['churned']==1].count(), df.churned[df['churned']==0].count()]
explode = (0, 0.1)
fig1, ax1 = plt.subplots(figsize=(10, 8))
ax1.pie(sizes, explode=explode, labels=labels, autopct='%1.1f%%',
        shadow=True, startangle=90)
ax1.axis('equal')
plt.title("POS Clients Churn", size = 20)
plt.show()
# Passing labels and values
lab = churn["churned"].value_counts().keys().tolist()
val = churn["churned"].value_counts().values.tolist()

trace = go.Pie(labels = lab ,
               values = val ,
               marker = dict(colors =  [ 'royalblue' ,'lime'],
                             line = dict(color = "white",
                                         width =  0.3)
                            ),
               rotation = 90,
               hoverinfo = "label+value+text",
               hole = .2
              )
layout = go.Layout(dict(title = "POS Clients Churn",
                        plot_bgcolor  = "rgb(243,243,243)",
                        paper_bgcolor = "rgb(243,243,243)",
                       )
                  )

data = [trace]
fig = go.Figure(data = data,layout = layout)
py.iplot(fig)<jupyter_output><empty_output><jupyter_text>**47.2% Churned,52.8% were retained**This is a loss over a certain time yet there is no clear factor for churning maybe it is the nature of the business better be alert. but also it gives hope because the default test dataset is always 25% which will help to punish gradually. **Head to Head attributes relations**Categorical attributes and Continuous attributes on impacting Churned, here is not about the counts but a specifically churned and unchurned vs the service.<jupyter_code>#Categorical attirbutes churn rate
fig, axs = plt.subplots(2, 2, figsize=(15, 8))
sns.countplot(x= churn.Address, hue = 'churned' ,data=churn, ax =axs[0][0])
sns.countplot(x=churn.BusinessType, hue = 'churned' ,data=churn, ax=axs[1][0])
sns.countplot(x=churn.HasMB, hue = 'churned' ,data=churn, ax=axs[0][1])
sns.countplot(x=churn.HasOtherAcc, hue = 'churned' ,data=churn, ax=axs[1][1])
plt.ylabel('count')<jupyter_output><empty_output><jupyter_text>What does Categorical attributes Highlights:* Address location can determine the success of your business and can be a great tool to know how to play with your market as Ajah show a huge number of customers with high churn, while others shows reasonable retention.* Apparently it is possible that customers who use the mobile banking service and has other account type in the same bank have low churn level* Enterprise businesses (large scale) have less churn than other business types; a promotional deal to consider other business types to improve retention*  Clients who use the bank's mobile banking service churn lesser while there's no visual difference for those who have other bank account with the bank * **Continous attributes churn rate**For continuous attribute i will have to normalize its values in order to compare its churn it won't be possible to plot Balance and age in the same plot yet they have very different figures.<jupyter_code>fig, axarr = plt.subplots(2, 2, figsize=(15, 8))
sns.boxplot(y='Balance',x = 'churned', hue = 'churned',data = churn, ax=axarr[0][0])
sns.boxplot(y='AverageTrans',x = 'churned', hue = 'churned',data = churn , ax=axarr[0][1])
sns.boxplot(y='Tenure',x = 'churned', hue = 'churned',data = churn, ax=axarr[1][0])
sns.boxplot(y='BusinessAge',x = 'churned', hue = 'churned',data = churn, ax=axarr[1][1])
#sns.boxplot(y='Address',x = 'churned', hue = 'churned',data = churn, ax=axarr[2][0])
#sns.boxplot(y='BusinessType',x = 'churned', hue = 'churned',data = churn, ax=axarr[2][1])<jupyter_output><empty_output><jupyter_text>As seen before the mean and correlation maps outcomes relate to these continuous results. but they are more detailed in terms of churned and unchurned distribution with estimated numbers .Over all conclusion is that all attributes have its impact to the performance for instance Balance and Average Transaction are functions of Tenure(that is the number of years your business has been using POS with the bank) the more you get older the more the relation of a customer and a bank become stronger as sign of loyalty. **Feature Engineering and Preparation** We are going to create new features from what we have and based on the relationship of attributes and prepare the existing ones to be ready to predict our next client possibility to churn and these stage is normally standardize head to head attributes as decided, for balance and Average Transaction this is quantitative relation then we will find its ratioNote that df dataframe is been used because categorical attributes was converted  to dummy variables in other to achieve an excellent process.<jupyter_code>df['BalanceAverageTransRatio'] = df.Balance/(df.AverageTrans)
df['TenureOverBusinessAge'] = df.Tenure/(df.BusinessAge)
df['AverageTransOverBusinessAge'] = df.AverageTrans/(df.BusinessAge)
df.head()

con_v=['AverageTrans','BusinessAge','Tenure','Balance','BalanceAverageTransRatio','TenureOverBusinessAge','AverageTransOverBusinessAge']
minVec = df[con_v].min().copy()
maxVec = df[con_v].max().copy()
df[con_v] = (df[con_v]-minVec)/(maxVec-minVec)
df.head()<jupyter_output><empty_output><jupyter_text>****Model fitting and selection****For the model fitting, I will try out the following* Logistic regression in the primal space and with different kernels* SVM Support Vector Machine Our dataset have 47.2% on churned customers this means we will try to predict. Hence we will use train_split model which deliver 25% test set and 75% traning set.because it covers the churned figure which prevent biases inside the model itself.<jupyter_code>'''For the one hot variables, we change 0 to -1 so that the models can capture a negative relation 
where the attribute in inapplicable instead of 0'''
df.loc[df.HasMB == 0, 'HasMB'] = -1
df.loc[df.HasOtherAcc == 0, 'HasOtherAcc'] = -1
df.head()
# One hot encode the categorical variables
lst = ['BusinessType_Enterprise','BusinessType_Individual','BusinessType_coperative','Address_Ajah',
      'Address_Ikeja','Address_Lekki']
remove = list()
for i in lst:
    if (df[i].dtype == np.str or df[i].dtype == np.object):
        for j in df_train[i].unique():
            df[i+'_'+j] = np.where(df[i] == j,1,-1)
        remove.append(i)
df = df.drop(remove, axis=1)
df.head()
# Create Train & Test Data
from sklearn.model_selection import train_test_split
y = df['churned'].values
x = df.drop(columns = ['churned'])
x_train, x_test, y_train, y_test = train_test_split(x, y,random_state=0)
pip install xgboost
# Support functions
from sklearn.preprocessing import PolynomialFeatures
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import GridSearchCV
from scipy.stats import uniform

# Fit models
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
#from sklearn.ensemble import RandomForestClassifier
from xgboost import XGBClassifier

# Scoring functions
from sklearn.metrics import accuracy_score
from sklearn.metrics import classification_report
from sklearn.metrics import roc_auc_score
from sklearn.metrics import roc_curve<jupyter_output><empty_output><jupyter_text>**Logistic Regression**<jupyter_code># Function to give best model score and parameters
def best_model(model):
    print(model.best_score_)    
    print(model.best_params_)
    print(model.best_estimator_)
def get_auc_scores(y_actual, method,method2):
    auc_score = roc_auc_score(y_actual, method); 
    fpr_df, tpr_df, _ = roc_curve(y_actual, method2); 
    return (auc_score, fpr_df, tpr_df)
# Running logistic regression model
from sklearn.linear_model import LogisticRegression
from sklearn import metrics
model = LogisticRegression()
result = model.fit(x_train, y_train)
prediction_test = model.predict(x_test)
print (metrics.accuracy_score(y_test, prediction_test))# Print the prediction accuracy
# getting the weights of all the variables on regression model
weights = pd.Series(model.coef_[0],
                 index=x.columns.values)
weights.sort_values()[-13:].plot(kind = 'barh')
weights.sort_values(ascending = False)

# Fit primal logistic regression
param_grid = {'C': [0.1,0.5,1,10,50,100], 'max_iter': [250], 'fit_intercept':[True],'intercept_scaling':[1],
              'penalty':['l2'], 'tol':[0.00001,0.0001,0.000001]}
log_primal_Grid = GridSearchCV(LogisticRegression(solver='lbfgs'),param_grid, cv=10, refit=True, verbose=0)
log_primal_Grid.fit(df.loc[:, df.columns != 'churned'],df.churned)
best_model(log_primal_Grid)
# Fit logistic regression with degree 2 polynomial kernel
param_grid = {'C': [0.1,10,50], 'max_iter': [300,500], 'fit_intercept':[True],'intercept_scaling':[1],'penalty':['l2'],
              'tol':[0.0001,0.000001]}
poly2 = PolynomialFeatures(degree=2)
df_train_pol2 = poly2.fit_transform(df.loc[:, df.columns != 'churned'])
log_pol2_Grid = GridSearchCV(LogisticRegression(solver = 'liblinear'),param_grid, cv=5, refit=True, verbose=0)
log_pol2_Grid.fit(df_train_pol2,df.churned)
best_model(log_pol2_Grid)<jupyter_output>0.8583700970562285
{'C': 0.1, 'fit_intercept': True, 'intercept_scaling': 1, 'max_iter': 300, 'penalty': 'l2', 'tol': 0.0001}
LogisticRegression(C=0.1, class_weight=None, dual=False, fit_intercept=True,
                   intercept_scaling=1, l1_ratio=None, max_iter=300,
                   multi_class='auto', n_jobs=None, penalty='l2',
                   random_state=None, solver='liblinear', tol=0.0001, verbose=0,
                   warm_start=False)<jupyter_text>**Support Vecor Machine (SVM)**<jupyter_code>from sklearn.svm import SVC
from sklearn.metrics import classification_report, confusion_matrix
model.svm = SVC(kernel='linear') 
model.svm.fit(x_train,y_train)
preds = model.svm.predict(x_test)
metrics.accuracy_score(y_test, preds)# Print the prediction accuracy
from sklearn.neighbors import KNeighborsClassifier
classifiers = [
    KNeighborsClassifier(5),    
]
# iterate over classifiers
for item in classifiers:
    classifier_name = ((str(item)[:(str(item).find("("))]))
    print (classifier_name)
    # Create classifier, train it and test it.
    clf = item
    clf.fit(x_train, y_train)
    pred = clf.predict(x_test)
    score = clf.score(x_test, y_test)
    print (round(score,3),"\n", "- - - - - ", "\n") # Print the prediction accuracy
    

# Fit SVM with RBF Kernel
param_grid = {'C': [0.5,100,150], 'gamma': [0.1,0.01,0.001],'probability':[True],'kernel': ['rbf']}
SVM_grid = GridSearchCV(SVC(), param_grid, cv=3, refit=True, verbose=0)
SVM_grid.fit(df.loc[:, df.columns != 'churned'],df.churned)
best_model(SVM_grid)
# Fit SVM with pol kernel
param_grid = {'C': [0.5,1,10,50,100], 'gamma': [0.1,0.01,0.001],'probability':[True],'kernel': ['poly'],'degree':[2,3] }
SVM_grid = GridSearchCV(SVC(), param_grid, cv=3, refit=True, verbose=0)
SVM_grid.fit(df.loc[:, df.columns != 'churned'],df.churned)
best_model(SVM_grid)<jupyter_output>0.868622595877001
{'C': 0.5, 'degree': 2, 'gamma': 0.1, 'kernel': 'poly', 'probability': True}
SVC(C=0.5, break_ties=False, cache_size=200, class_weight=None, coef0=0.0,
    decision_function_shape='ovr', degree=2, gamma=0.1, kernel='poly',
    max_iter=-1, probability=True, random_state=None, shrinking=True, tol=0.001,
    verbose=False)<jupyter_text>**Fit best Models**<jupyter_code># Fit primal logistic regression
log_primal = LogisticRegression(C=100, class_weight=None, dual=False, fit_intercept=True,intercept_scaling=1, max_iter=250, multi_class='auto',n_jobs=None, 
                                penalty='l2', random_state=None, solver='lbfgs',tol=1e-05, verbose=0, warm_start=False)
log_primal.fit(df.loc[:, df.columns != 'churned'],df.churned)
# Fit logistic regression with pol 2 kernel
poly2 = PolynomialFeatures(degree=2)
df_train_pol2 = poly2.fit_transform(df.loc[:, df.columns != 'churned'])
log_pol2 = LogisticRegression(C=10, class_weight=None, dual=False, fit_intercept=True,intercept_scaling=1, max_iter=300, multi_class='auto', n_jobs=None, 
                              penalty='l2', random_state=None, solver='liblinear',tol=0.0001, verbose=0, warm_start=False)
log_pol2.fit(df_train_pol2,df.churned)
# Fit SVM with RBF Kernel
SVM_RBF = SVC(C=100, cache_size=200, class_weight=None, coef0=0.0, decision_function_shape='ovr', degree=3, gamma=0.1, kernel='rbf', max_iter=-1, probability=True, 
              random_state=None, shrinking=True,tol=0.001, verbose=False)
SVM_RBF.fit(df.loc[:, df.columns != 'churned'],df.churned)
# Fit SVM with Pol Kernel
SVM_POL = SVC(C=100, cache_size=200, class_weight=None, coef0=0.0,  decision_function_shape='ovr', degree=2, gamma=0.1, kernel='poly',  max_iter=-1,
              probability=True, random_state=None, shrinking=True, tol=0.001, verbose=False)
SVM_POL.fit(df.loc[:, df.columns != 'churned'],df.churned)<jupyter_output><empty_output><jupyter_text>**Review best model fit accuracy : Keen interest is on the performance in predicting 1's (Clients who churn)** **Logistics Regression Predict**<jupyter_code>print(classification_report(df.churned, log_primal.predict(df.loc[:, df.columns != 'churned'])))
print(classification_report(df.churned,  log_pol2.predict(df_train_pol2)))<jupyter_output>precision    recall  f1-score   support

           0       0.89      0.93      0.91       723
           1       0.91      0.87      0.89       646

    accuracy                           0.90      1369
   macro avg       0.90      0.90      0.90      1369
weighted avg       0.90      0.90      0.90      1369<jupyter_text>**SVM Predict**<jupyter_code>print(classification_report(df.churned,  SVM_RBF.predict(df.loc[:, df.columns != 'churned'])))
print(classification_report(df.churned,  SVM_POL.predict(df.loc[:, df.columns != 'churned'])))
y = df.churned
X = df.loc[:, df.columns != 'churned']
X_pol2 = df_train_pol2
auc_log_primal, fpr_log_primal, tpr_log_primal = get_auc_scores(y, log_primal.predict(X),log_primal.predict_proba(X)[:,1])
auc_log_pol2, fpr_log_pol2, tpr_log_pol2 = get_auc_scores(y, log_pol2.predict(X_pol2),log_pol2.predict_proba(X_pol2)[:,1])
auc_SVM_RBF, fpr_SVM_RBF, tpr_SVM_RBF = get_auc_scores(y, SVM_RBF.predict(X),SVM_RBF.predict_proba(X)[:,1])
auc_SVM_POL, fpr_SVM_POL, tpr_SVM_POL = get_auc_scores(y, SVM_POL.predict(X),SVM_POL.predict_proba(X)[:,1])
#auc_RF, fpr_RF, tpr_RF = get_auc_scores(y, RF.predict(X),RF.predict_proba(X)[:,1])
#auc_XGB, fpr_XGB, tpr_XGB = get_auc_scores(y, XGB.predict(X),XGB.predict_proba(X)[:,1])
plt.figure(figsize = (12,6), linewidth= 1)
plt.plot(fpr_log_primal, tpr_log_primal, label = 'log primal Score: ' + str(round(auc_log_primal, 5)))
plt.plot(fpr_log_pol2, tpr_log_pol2, label = 'log pol2 score: ' + str(round(auc_log_pol2, 5)))
plt.plot(fpr_SVM_RBF, tpr_SVM_RBF, label = 'SVM RBF Score: ' + str(round(auc_SVM_RBF, 5)))
plt.plot(fpr_SVM_POL, tpr_SVM_POL, label = 'SVM POL Score: ' + str(round(auc_SVM_POL, 5)))
#plt.plot(fpr_RF, tpr_RF, label = 'RF score: ' + str(round(auc_RF, 5)))
#plt.plot(fpr_XGB, tpr_XGB, label = 'XGB score: ' + str(round(auc_XGB, 5)))
plt.plot([0,1], [0,1], 'k--', label = 'Random: 0.5')
plt.xlabel('False positive rate')
plt.ylabel('True positive rate')
plt.title('ROC Curve')
plt.legend(loc='best')
#plt.savefig('roc_results_ratios.png')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11500355534486845, 'fraction_numerical': 0.02398672671249111, 'mean_word_length': 5.084799538505913, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 58, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '417879', 'license': 'MIT', 'n_tokens_mistral': 7244, 'n_tokens_neox': 6781, 'n_words': 2012, 'path': 'py docs/Prediction 2.ipynb', 'repo_name': 'Glitzabeth/POS-Churn-Prediction', 'size': '21095'}