<jupyter_start><jupyter_text>Churn Prediction using AMLWorkbenchThis notebook will introduce the use of the churn dataset to create churn prediction models. The dataset used to ingest is from SIDKDD 2009 competition. The dataset consists of heterogeneous noisy data (numerical/categorical variables) from French Telecom company Orange and is anonymized.We will use the .dprep file created from the datasource wizard.<jupyter_code>import dataprep
from dataprep.Package import Package
import pickle

from sklearn.naive_bayes import GaussianNB
from sklearn.tree import DecisionTreeClassifier, export_graphviz
from sklearn.ensemble import RandomForestClassifier
import pandas as pd
import numpy as np
import csv
from sklearn.metrics import accuracy_score
from sklearn.cross_validation import train_test_split
from sklearn.preprocessing import LabelEncoder<jupyter_output>C:\Users\mikher\AppData\local\AmlWorkbench\Python\lib\site-packages\sklearn\cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
  "This module will be removed in 0.20.", DeprecationWarning)<jupyter_text>Read DataWe first retrieve the data as a data frame using .dprep that we created using the datasource wizard. Print the top few lines using head()<jupyter_code>#Local via package
with Package.open_package('CATelcoCustomerChurnTrainingSample.dprep') as pkg:
    df = pkg.dataflows[0].get_dataframe()

# Blob via package
#with Package.open_package('CATelcoCustomerChurnTrainingBlobSample.dprep') as pkg:
#    df = pkg.dataflows[0].get_dataframe()
    
# if this one fails check to see that you are NOT exporting file to local/blob
# storage because it will fail to overwrite

# Local via CSV file
# df = pd.read_csv("./data/CATelcoCustomerChurnTrainingSample.csv")
df.head(5)
df.columns.get_loc("churn")<jupyter_output><empty_output><jupyter_text>Encode ColumnsConvert categorical variable into dummy/indicator variables using pandas.get_dummies. In addition, we will need to change the column names to ensure there are no multiple columns with the same name<jupyter_code>columns_to_encode = list(df.select_dtypes(include=['category','object']))
print(columns_to_encode)
for column_to_encode in columns_to_encode:
    dummies = pd.get_dummies(df[column_to_encode])
    one_hot_col_names = []
    for col_name in list(dummies.columns):
        one_hot_col_names.append(column_to_encode + '_' + col_name)
    dummies.columns = one_hot_col_names
    df = df.drop(column_to_encode, axis=1)
    df = df.join(dummies)

print("Encoded columns:")
print(df.columns)<jupyter_output>['age_bin', 'customersuspended', 'education', 'gender', 'homeowner', 'maritalstatus', 'noadditionallines', 'occupation', 'state', 'usesinternetservice', 'usesvoiceservice']
Encoded columns:
Index(['age', 'annualincome', 'calldroprate', 'callfailurerate', 'callingnum',
       'customerid', 'monthlybilledamount', 'numberofcomplaints',
       'numberofmonthunpaid', 'numdayscontractequipmentplanexpiring',
       'penaltytoswitch', 'totalminsusedinlastmonth', 'unpaidbalance',
       'percentagecalloutsidenetwork', 'totalcallduration', 'avgcallduration',
       'churn', 'age_bin_0-19', 'age_bin_0-39', 'age_bin_20-39',
       'age_bin_20-59', 'age_bin_40-59', 'age_bin_40-79', 'age_bin_60-79',
       'age_bin_60-99', 'customersuspended_No', 'customersuspended_Yes',
       'education_Bachelor or equivalent', 'education_High School or below',
       'education_Master or equivalent', 'education_PhD or equivalent',
       'gender_Female', 'gender_Male', 'homeowner_No', 'homeowner_Yes',
       'mar[...]<jupyter_text>ModelingFirst, we will build a Gaussian Naive Bayes model using GaussianNB for churn classification. Naive Bayes methods are a set of supervised learning algorithms based on applying Bayes theorem with the “naive” assumption of independence between every pair of features.In addition, we will also build a decision tree classifier for comparison:- min_samples_split=20 requires 20 samples in a node for it to be split- random_state=99 to seed the random number generator  Investigate important featuresNot all features might be important. Try to find the features that influence the most<jupyter_code>from sklearn.ensemble import ExtraTreesClassifier

train, test = train_test_split(df, test_size = 0.3, random_state=98)
Y = train['churn'].values
X = train.drop('churn', 1)
X = X.values

model_feature_test = ExtraTreesClassifier()
model_feature_test.fit(X, Y)
features_ranked = model_feature_test.feature_importances_
print("Feature importance:\n" + str(features_ranked))

#>>> a
#array([9, 4, 4, 3, 3, 9, 0, 4, 6, 0])
ind = np.argpartition(features_ranked, -20)[-20:]
print("Top Feature Index:\n" + str(ind))
#array([1, 5, 8, 0])
print(features_ranked[ind])

top_feature_names = train.columns[ind]
print("Top feature names:\n" + str(top_feature_names))
#array([4, 9, 6, 9])

#Unlike argsort, this function runs in linear time in the worst case, but the returned indices are not sorted, as can be seen from the result of evaluating a[ind]. If you need that too, sort them afterwards:

#>>> ind[np.argsort(a[ind])]
#array([1, 8, 5, 0])


#sorted_features = sorted(range(len(features_ranked)), 
#                         key=lambda i: features_ranked[i], reverse=True) 
df.columns.get_loc("churn")
len(df.columns)<jupyter_output><empty_output><jupyter_text>Train on only top features<jupyter_code>top_feature_names<jupyter_output><empty_output><jupyter_text>Naive Bayes model<jupyter_code>model = GaussianNB()

# top_feature_names.append(['churn'])
dfChurn = df[['churn']]
dfFeatures = df[top_feature_names]
dfTopFeaturesChurn = dfFeatures.join(dfChurn)

train, test = train_test_split(dfTopFeaturesChurn, test_size = 0.3, random_state=98)

target = train['churn'].values
train = train.drop('churn', 1)
train = train.values
model.fit(train, target)

Y_test_val = test['churn'].values
X_test_val = test.drop('churn', 1).values
predicted = model.predict(X_test_val)
print("Naive Bayes Classification Accuracy", accuracy_score(Y_test_val, predicted))
print("#Expected churn: " + str(sum(Y_test_val[Y_test_val[:] > 0])))
print("#Predcited churn: " + str(sum(predicted[predicted[:] > 0])))
exp_inp = pd.Categorical(Y_test_val, categories=[0,1])
pred_inp = pd.Categorical(predicted, categories=[0,1])
print(pd.crosstab(exp_inp, pred_inp, colnames=["Predicted"], rownames=['Actual']))<jupyter_output>Predicted     0   1
Actual             
0          2665  49
1           282   4<jupyter_text>ROC Curve Naive Bayes<jupyter_code>from sklearn.metrics import roc_curve
from sklearn.metrics import roc_auc_score
from sklearn.metrics import precision_recall_curve
import matplotlib.pyplot as plt
predicted_prob = model.predict_proba(X_test_val)
# TASK: Rita ROC curve
# confidence_nb = predicted_prob.max(axis=1)
fpr, tpr, thresholds = roc_curve(Y_test_val, predicted_prob[:, 1])
roc_auc_nb = roc_auc_score(Y_test_val, predicted_prob[:, 1])
print("AUC:" + str(roc_auc_nb))

plt.figure(figsize=(10,10))
plt.title('Receiver operating characteristic example')
plt.plot(fpr, tpr, color='darkorange', lw = 2, label='ROC curve (area = %0.2f)' % roc_auc_nb)
plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')
plt.legend(loc="lower right")
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.0])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.show()<jupyter_output>AUC:0.505733028946<jupyter_text>Precision and recallHigh recall means that we manage to pick most of the positive examples (churn)Low precision means that we also pick a lot of false positive for the churn whichmeans that a high recall and a low precision catches most of the true churn at thecost of getting many false churn at the same time. Low precision = not so preciseSee: https://medium.com/@klintcho/explaining-precision-and-recall-c770eb9c69e9 for a good explanation.The chart below calculates the precision and recall for the different thresholds of the probability of our predicions.<jupyter_code>precision, recall, thresholds_pr = precision_recall_curve(Y_test_val, predicted_prob[:,1])
plt.figure(figsize=(10,10))
plt.title('Precision Recall curve')
plt.plot(precision, recall, color='darkorange', lw = 2, label='ROC curve (area = %0.2f)' % roc_auc_nb)
plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')
plt.legend(loc="lower right")
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.0])
plt.xlabel('precision')
plt.ylabel('recall')
plt.show()
print("Precision:\n" + str(precision))
print("Recall:\n" + str(recall))

print("\nIf we want to catch every churn the precision would be only " + "{:.2%}".format(precision[0]))
print("I.e. we would catch a huge number of false positive at the same time.")
print("The other way around. If we want only True positive churn (no false) we would get a recall of " + "{:.2%}".format(recall[6]))
print("Depending of the business case you can chose different thresholds")<jupyter_output><empty_output><jupyter_text>Tree model<jupyter_code>help(ExtraTreesClassifier)
modelxt = ExtraTreesClassifier()
# modelxt = RandomForestClassifier()

# top_feature_names.append(['churn'])
dfChurn = df[['churn']]
dfFeatures = df[top_feature_names]
dfTopFeaturesChurn = dfFeatures.join(dfChurn)

train, test = train_test_split(dfTopFeaturesChurn, test_size = 0.3, random_state=98)

target = train['churn'].values
train = train.drop('churn', 1)
train = train.values
modelxt.fit(train, target)

Y_test_val = test['churn'].values
X_test_val = test.drop('churn', 1).values
predicted = modelxt.predict(X_test_val)
print("ExtraTreesClassifier acc:", accuracy_score(Y_test_val, predicted))<jupyter_output>ExtraTreesClassifier acc: 0.978<jupyter_text>Confusion matrix<jupyter_code>print("#Expected churn: " + str(sum(Y_test_val[Y_test_val[:] > 0])))
print("#Predcited churn: " + str(sum(predicted[predicted[:] > 0])))
exp_inp = pd.Categorical(Y_test_val, categories=[0,1])
pred_inp = pd.Categorical(predicted, categories=[0,1])
print(pd.crosstab(exp_inp, pred_inp, colnames=["Predicted"], rownames=['Actual']))<jupyter_output>Predicted     0    1
Actual              
0          2714    0
1            66  220<jupyter_text>Confusion matrix results example    Predicted     0    1    Actual                  0          2708    4    1            75  213We predicted 2708+75 as non-churnWe predicted 4 non-churn wrongly as churnWe predicted 217 as churn (4 was non-churn of these)We did not catch 75 churn that was predcited as non-churn  ROC Curve ExtraTreesClassifier<jupyter_code>from sklearn.metrics import roc_curve
from sklearn.metrics import roc_auc_score
from sklearn.metrics import precision_recall_curve
import matplotlib.pyplot as plt
predicted_prob = modelxt.predict_proba(X_test_val)
# TASK: Rita ROC curve
# confidence_nb = predicted_prob.max(axis=1)
fpr, tpr, thresholds = roc_curve(Y_test_val, predicted_prob[:, 1])
roc_auc_nb = roc_auc_score(Y_test_val, predicted_prob[:, 1])
print("AUC:" + str(roc_auc_nb))

plt.figure(figsize=(10,10))
plt.title('Receiver operating characteristic example')
plt.plot(fpr, tpr, color='darkorange', lw = 2, label='ROC curve (area = %0.2f)' % roc_auc_nb)
plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')
plt.legend(loc="lower right")
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.0])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.show()<jupyter_output>AUC:0.945764942027<jupyter_text>Precision and recallHigh recall means that we manage to pick most of the positive examples (churn)Low precision means that we also pick a lot of false positive for the churn whichmeans that a high recall and a low precision catches most of the true churn at thecost of getting many false churn at the same time. Low precision = not so preciseSee: https://medium.com/@klintcho/explaining-precision-and-recall-c770eb9c69e9 for a good explanation.The chart below calculates the precision and recall for the different thresholds of the probability of our predicions.<jupyter_code>precision, recall, thresholds_pr = precision_recall_curve(Y_test_val, predicted_prob[:,1])
plt.figure(figsize=(10,10))
plt.title('Precision Recall curve')
plt.plot(precision, recall, color='darkorange', lw = 2, label='ROC curve (area = %0.2f)' % roc_auc_nb)
plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')
plt.legend(loc="lower right")
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.0])
plt.xlabel('precision')
plt.ylabel('recall')
plt.show()
print("Precision:\n" + str(precision))
print("Recall:\n" + str(recall))

print("\nIf we want to catch every churn the precision would be only " + "{:.2%}".format(precision[0]))
print("I.e. we would catch a huge number of false positive at the same time.")
print("The other way around. If we want only True positive churn (no false) we would get a recall of " + "{:.2%}".format(recall[6]))
print("Depending of the business case you can chose different thresholds")

dfTopFeaturesChurn.columns.get_loc("churn")
dfFeatures
dt = DecisionTreeClassifier(min_samples_split=20, random_state=982)
dt.fit(train, target)
predicted = dt.predict(test)
print("Decision Tree Classification Accuracy", accuracy_score(expected, predicted))
x = np.array([2, 4, 6])
y = np.array([4, 16, 36])

plt.figure(figsize=(10,10))
plt.title('X Y curve')
# plt.plot(precision, recall, color='darkorange', lw = 2, label='ROC curve (area = %0.2f)' % roc_auc_nb)
# plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')
# plt.legend(loc="lower right")
plt.xlim([0.0, 36.0])
plt.ylim([0.0, 36.0])
plt.xlabel('X')
plt.ylabel('Y')
plt.plot(x, y)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1214457130390517, 'fraction_numerical': 0.02472547451094466, 'mean_word_length': 5.889779559118236, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 52, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '219033', 'license': 'MIT', 'n_tokens_mistral': 4912, 'n_tokens_neox': 4540, 'n_words': 1263, 'path': 'ChurnAnalytics.ipynb', 'repo_name': 'blendax/churn', 'size': '13751'}