<jupyter_start><jupyter_text>1. Reading Data<jupyter_code>##importing the dataset
churn_data = pd.read_csv("churn_data.csv")
churn_data.head()
## importing the customer data
customer_data = pd.read_csv("customer_data.csv")
customer_data.head()
## importing  the internet data
internet_data = pd.read_csv("internet_data.csv")
internet_data.head()
## merging all the dataframes above on customerID
df1 = pd.merge(churn_data,customer_data,how="inner",on="customerID")
telecom = pd.merge(df1,internet_data,how="inner",on="customerID")
telecom.head()<jupyter_output><empty_output><jupyter_text>2. Data Understanding<jupyter_code>##getting the shape
telecom.shape
telecom.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 7043 entries, 0 to 7042
Data columns (total 21 columns):
customerID          7043 non-null object
tenure              7043 non-null int64
PhoneService        7043 non-null object
Contract            7043 non-null object
PaperlessBilling    7043 non-null object
PaymentMethod       7043 non-null object
MonthlyCharges      7043 non-null float64
TotalCharges        7043 non-null object
Churn               7043 non-null object
gender              7043 non-null object
SeniorCitizen       7043 non-null int64
Partner             7043 non-null object
Dependents          7043 non-null object
MultipleLines       7043 non-null object
InternetService     7043 non-null object
OnlineSecurity      7043 non-null object
OnlineBackup        7043 non-null object
DeviceProtection    7043 non-null object
TechSupport         7043 non-null object
StreamingTV         7043 non-null object
StreamingMovies     7043 non-null object
dtypes: float64(1), int64(2), obj[...]<jupyter_text>We observe 0 missing data<jupyter_code>## lets see the dtypes
telecom.dtypes
telecom.describe()<jupyter_output><empty_output><jupyter_text>3. Data Preparation Convert variables which have data like Yes/No to 0/1<jupyter_code>## list of variables to map 
var_list = ['PhoneService','PaperlessBilling','Churn','Partner','Dependents']

## defining the binary_map function
def binary_map(x):
    return x.map({'Yes':1,"No":0})

## applying that function to the housing list
telecom[var_list] = telecom[var_list].apply(binary_map)
telecom.head()<jupyter_output><empty_output><jupyter_text>create dummies for categorical variables with multiple levels<jupyter_code>dummy1 = pd.get_dummies(telecom[['Contract','PaymentMethod','gender','InternetService']],drop_first=True)


#concat with the orginal
telecom = pd.concat([telecom,dummy1],axis=1)

##dropping the older variables
telecom.drop(columns=['Contract','PaymentMethod','gender','InternetService'],inplace=True)

telecom.head()
##creating dummies for other variables

##dummies for MultipleLines
ml = pd.get_dummies(telecom["MultipleLines"],prefix="MultipleLines")
##removing the No service, thats why we didnot use drop_first=true
ml1 = ml.drop("MultipleLines_No phone service",axis = 1)
## addind to the master dataframe
telecom = pd.concat([telecom,ml1],axis = 1)
## Phone service already tells if there is phone service or no, so we can drop that vategory

##dummies for OnlineSecurity
os = pd.get_dummies(telecom["OnlineSecurity"],prefix="OnlineSecurity")
##removing the No service, thats why we didnot use drop_first=true
os1 = os.drop("OnlineSecurity_No internet service",axis = 1)
## addind to the master dataframe
telecom = pd.concat([telecom,os1],axis = 1)


##dummies for OnlineBackup
ob = pd.get_dummies(telecom["OnlineBackup"],prefix="OnlineBackup")
##removing the No service, thats why we didnot use drop_first=true
ob1 = ob.drop("OnlineBackup_No internet service",axis = 1)
## addind to the master dataframe
telecom = pd.concat([telecom,ob1],axis = 1)


##dummies for DeviceProtection
dp = pd.get_dummies(telecom["DeviceProtection"],prefix="DeviceProtection")
##removing the No service, thats why we didnot use drop_first=true
dp1 = dp.drop("DeviceProtection_No internet service",axis = 1)
## addind to the master dataframe
telecom = pd.concat([telecom,dp1],axis = 1)


##dummies for TechSupport
ts = pd.get_dummies(telecom["TechSupport"],prefix="TechSupport")
##removing the No service, thats why we didnot use drop_first=true
ts1 = ts.drop("TechSupport_No internet service",axis = 1)
## addind to the master dataframe
telecom = pd.concat([telecom,ts1],axis = 1)


##dummies for StreamingTV
st = pd.get_dummies(telecom["StreamingTV"],prefix="StreamingTV")
##removing the No service, thats why we didnot use drop_first=true
st1 = st.drop("StreamingTV_No internet service",axis = 1)
## addind to the master dataframe
telecom = pd.concat([telecom,st1],axis = 1)


##dummies for StreamingMovies
sm = pd.get_dummies(telecom["StreamingMovies"],prefix="StreamingMovies")
##removing the No service, thats why we didnot use drop_first=true
sm1 = sm.drop("StreamingMovies_No internet service",axis = 1)
## addind to the master dataframe
telecom = pd.concat([telecom,sm1],axis = 1)
## remove the redundant/orginal columns
telecom.drop(columns=["MultipleLines","OnlineSecurity","OnlineBackup","DeviceProtection","TechSupport","StreamingTV","StreamingMovies"],inplace=True)
##print the final shape
telecom.shape
telecom.TotalCharges.dtype
## charges should be a number,  lets use the below method to handle nans as well
telecom.TotalCharges = telecom.TotalCharges.convert_objects(convert_numeric=True)
telecom.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 7043 entries, 0 to 7042
Data columns (total 32 columns):
customerID                               7043 non-null object
tenure                                   7043 non-null int64
PhoneService                             7043 non-null int64
PaperlessBilling                         7043 non-null int64
MonthlyCharges                           7043 non-null float64
TotalCharges                             7032 non-null float64
Churn                                    7043 non-null int64
SeniorCitizen                            7043 non-null int64
Partner                                  7043 non-null int64
Dependents                               7043 non-null int64
Contract_One year                        7043 non-null uint8
Contract_Two year                        7043 non-null uint8
PaymentMethod_Credit card (automatic)    7043 non-null uint8
PaymentMethod_Electronic check           7043 non-null uint8
PaymentMethod_Mailed check        [...]<jupyter_text>Checking for Outliers as they affect regression models<jupyter_code>num_telecom = telecom[['tenure','MonthlyCharges','SeniorCitizen','TotalCharges']]
num_telecom.describe(percentiles=[0.25,0.5,0.75,0.90,0.95,0.99])<jupyter_output><empty_output><jupyter_text>No outliers!!<jupyter_code>#Check the missing values
telecom.isnull().sum()
## drop those 11 records
telecom = telecom[~telecom.TotalCharges.isnull()]
telecom.isnull().sum()<jupyter_output><empty_output><jupyter_text>Train-Test Split<jupyter_code>from sklearn.model_selection import train_test_split

## create data, remove target and index variables from train data
X = telecom.drop(columns=['Churn','customerID'])
y = telecom['Churn']
X_train,X_test,y_train,y_test = train_test_split(X,y,train_size=0.7,test_size=0.3,random_state=100)<jupyter_output><empty_output><jupyter_text>Feature Scaling<jupyter_code>from sklearn.preprocessing import StandardScaler
## lets look at the data
telecom.head()
scaler = StandardScaler()
X_train[['tenure','MonthlyCharges','TotalCharges']] = scaler.fit_transform(X_train[['tenure','MonthlyCharges','TotalCharges']])
X_train.head()
##churn rate
telecom.Churn.sum()/telecom.shape[0] * 100<jupyter_output><empty_output><jupyter_text>Looking at the Correlations<jupyter_code>plt.figure(figsize=(20,10))
sns.heatmap(telecom.corr(),annot=True)
## Dropping highly correlated variables, we see a good cluster of highly correlated values in the below variable. Business context will help more here
vars_to_drop = ["MultipleLines_No","OnlineSecurity_No","OnlineBackup_No","DeviceProtection_No","StreamingTV_No","StreamingMovies_No","TechSupport_No"]

X_test = X_test.drop(columns=vars_to_drop)
X_train = X_train.drop(columns=vars_to_drop)<jupyter_output><empty_output><jupyter_text>Running the first training model<jupyter_code>import statsmodels.api as sm
## logistic regression model
X_train_sm = sm.add_constant(X_train)
logm1 = sm.GLM(y_train,X_train_sm,family=sm.families.Binomial())
logm1.fit().summary()<jupyter_output><empty_output><jupyter_text>If we see p-values above, we see there are lot of larger p-values , so there is a good scope to eliminate a lot of features  RFE<jupyter_code>from sklearn.linear_model import LogisticRegression
logreg = LogisticRegression()
from sklearn.feature_selection import RFE
rfe = RFE(logreg,15)
rfe = rfe.fit(X_train,y_train)
rfe.support_
list(zip(X_train.columns,rfe.support_,rfe.ranking_))
col = X_train.columns[rfe.support_]
##excluded columns
X_train.columns[~rfe.support_]<jupyter_output><empty_output><jupyter_text>Assessing the model with StatsModels<jupyter_code>X_train_sm = sm.add_constant(X_train[col]) ## using the selected features
logm2 = sm.GLM(y_train,X_train_sm,family=sm.families.Binomial())
res = logm2.fit()
res.summary()<jupyter_output><empty_output><jupyter_text>All the p values are les than 0.05<jupyter_code>## lets get the predicted values on the train set
y_train_pred = res.predict(X_train_sm)
y_train_pred.head()
## convert y_train_pred to np array
y_train_pred = y_train_pred.values.reshape(-1)<jupyter_output><empty_output><jupyter_text>Creating a dataframe with the churn and the probability of the churn<jupyter_code>y_train_pred_final = pd.DataFrame({"Churn":y_train.values,'Churn_Prob':y_train_pred})
y_train_pred_final["CustID"] = y_train.index
y_train_pred_final.head()<jupyter_output><empty_output><jupyter_text>Creating a new column predicted which will have 1 if >0.5 else 0<jupyter_code>y_train_pred_final['predicted'] = y_train_pred_final.Churn_Prob.apply(lambda x: 1 if x > 0.5 else 0)
y_train_pred_final.head()
from sklearn import metrics
#confusion matrix
confusion = metrics.confusion_matrix(y_train_pred_final.Churn,y_train_pred_final.predicted)
confusion
# lets print accuracy score 
print(metrics.accuracy_score(y_train_pred_final.Churn,y_train_pred_final.predicted))<jupyter_output>0.8082080455099553<jupyter_text>Checking VIFs<jupyter_code>## importing the VIF fnction from sklearn
from statsmodels.stats.outliers_influence import variance_inflation_factor
# Create a dataframe that will contain the names of all the feature variables and their respective VIFs
vif = pd.DataFrame()
vif['Features'] = X_train[col].columns
vif['VIF'] = [variance_inflation_factor(X_train[col].values, i) for i in range(X_train[col].shape[1])]
vif['VIF'] = round(vif['VIF'], 2)
vif = vif.sort_values(by = "VIF", ascending = False)
vif
## lets drop PhoneService as it has VIF > 5
col = col.drop('PhoneService',1)
col
## retrain after dropping
X_train_sm = sm.add_constant(X_train[col]) ## using the selected features
logm3 = sm.GLM(y_train,X_train_sm,family=sm.families.Binomial())
res = logm3.fit()
res.summary()<jupyter_output><empty_output><jupyter_text>Now multiples lines has higher p value > 0.05<jupyter_code>## lets get the predicted values on the train set
y_train_pred = res.predict(X_train_sm)
## convert y_train_pred to np array
y_train_pred = y_train_pred.values.reshape(-1)
y_train_pred
y_train_pred_final["Churn_Prob"] = y_train_pred
y_train_pred_final.head()
y_train_pred_final['predicted'] = y_train_pred_final.Churn_Prob.apply(lambda x: 1 if x > 0.5 else 0)
y_train_pred_final.head()
#confusion matrix
confusion = metrics.confusion_matrix(y_train_pred_final.Churn,y_train_pred_final.predicted)
confusion
# lets print accuracy score 
print(metrics.accuracy_score(y_train_pred_final.Churn,y_train_pred_final.predicted))<jupyter_output>0.8051605038602194<jupyter_text>Accuracy didnot drop much  Let's look at VIF again<jupyter_code># Create a dataframe that will contain the names of all the feature variables and their respective VIFs
vif = pd.DataFrame()
vif['Features'] = X_train[col].columns
vif['VIF'] = [variance_inflation_factor(X_train[col].values, i) for i in range(X_train[col].shape[1])]
vif['VIF'] = round(vif['VIF'], 2)
vif = vif.sort_values(by = "VIF", ascending = False)
vif
## lets drop TotalCharges as it has VIF > 5
col = col.drop('TotalCharges',1)
col
## retrain after dropping
X_train_sm = sm.add_constant(X_train[col]) ## using the selected features
logm4 = sm.GLM(y_train,X_train_sm,family=sm.families.Binomial())
res = logm4.fit()
res.summary()
## lets get the predicted values on the train set
y_train_pred = res.predict(X_train_sm)
## convert y_train_pred to np array
y_train_pred = y_train_pred.values.reshape(-1)
y_train_pred
y_train_pred_final["Churn_Prob"] = y_train_pred
y_train_pred_final.head()
y_train_pred_final['predicted'] = y_train_pred_final.Churn_Prob.apply(lambda x: 1 if x > 0.5 else 0)
y_train_pred_final.head()
#confusion matrix
confusion = metrics.confusion_matrix(y_train_pred_final.Churn,y_train_pred_final.predicted)
confusion
# lets print accuracy score 
print(metrics.accuracy_score(y_train_pred_final.Churn,y_train_pred_final.predicted))
# Create a dataframe that will contain the names of all the feature variables and their respective VIFs
vif = pd.DataFrame()
vif['Features'] = X_train[col].columns
vif['VIF'] = [variance_inflation_factor(X_train[col].values, i) for i in range(X_train[col].shape[1])]
vif['VIF'] = round(vif['VIF'], 2)
vif = vif.sort_values(by = "VIF", ascending = False)
vif<jupyter_output><empty_output><jupyter_text>All VIFs are below 5!!  Metrics beyond simply accuracy<jupyter_code>TP = confusion[1,1]
TN = confusion[0,0]
FP = confusion[0,1]
FN = confusion[1,0]
# Lets look at the sensitivity
TP/float(TP+FN)<jupyter_output><empty_output><jupyter_text>Sensitivity here tells us that how many churners amongst that total actual churners were predicted<jupyter_code># lets look at the specificity
TN/float(TN + FP)
## lets look at the false positive rate, which tells number of churners predicted who aren't actaully one
FP/float(FP+TN)
## lets look at the false negative rate, which tells number of non churners who are actually actaully churners
FN/float(FN + TP)
## positive predictive rate
TP/float(TP+FP)
## negative predictive rate
TN/float(TN+FN)<jupyter_output><empty_output><jupyter_text>Drawing the ROC Curve<jupyter_code># Defining the function to plot the ROC curve
def draw_roc( actual, probs ):
    fpr, tpr, thresholds = metrics.roc_curve( actual, probs,
                                              drop_intermediate = False )
    auc_score = metrics.roc_auc_score( actual, probs )
    plt.figure(figsize=(5, 5))
    plt.plot( fpr, tpr, label='ROC curve (area = %0.2f)' % auc_score )
    plt.plot([0, 1], [0, 1], 'k--')
    plt.xlim([0.0, 1.0])
    plt.ylim([0.0, 1.05])
    plt.xlabel('False Positive Rate or [1 - True Negative Rate]')
    plt.ylabel('True Positive Rate')
    plt.title('Receiver operating characteristic example')
    plt.legend(loc="lower right")
    plt.show()

    return None

# Calling the function
draw_roc(y_train_pred_final.Churn, y_train_pred_final.Churn_Prob)<jupyter_output><empty_output><jupyter_text>Finding the right cutoff<jupyter_code>## lets create columns of predictions with different cutoffs
numbers = [float(x)/10 for x in range(10)]

for i in numbers:
    y_train_pred_final[i] = y_train_pred_final.Churn_Prob.apply(lambda x: 1 if x>i else 0)

y_train_pred_final.head()
## lets calculate accuracy,sensitivity,specificity of the model at each cutoff probability
cutoff_df = pd.DataFrame(columns=['prob','accuracy','sensi(TPR)','speci','FPR'])
num = [.0,.1,.2,.3,.4,.5,.6,.7,.8,.9]

for i in num:
    cm1 = metrics.confusion_matrix(y_train_pred_final.Churn,y_train_pred_final[i])
    total1 = sum(sum(cm1)) ## get the whole count
    accuaracy = (cm1[0,0]+cm1[1,1])/total1
    sensi = cm1[1,1]/(cm1[1,0]+cm1[1,1])
    speci = cm1[0,0]/(cm1[0,0]+cm1[0,1])
    FPR = 1 - speci
    
    cutoff_df.loc[i] = [i,accuaracy,sensi,speci,FPR]
cutoff_df
cutoff_df.plot.line(x='prob',y=["accuracy","sensi(TPR)","speci","FPR"])<jupyter_output><empty_output><jupyter_text>at 0.3 we have less FPR and high TPR. Both sensi and specifi along with accuracy are high we can choose any cutoff based on the metrics we are focussing by looking at the graph<jupyter_code># Creating new column 'predicted' with 1 if Churn_Prob > 0.3 else 0
y_train_pred_final['predicted'] = y_train_pred_final.Churn_Prob.map(lambda x: 1 if x > 0.3 else 0)
# lets print accuracy score 
print(metrics.accuracy_score(y_train_pred_final.Churn,y_train_pred_final.predicted))
confusion = metrics.confusion_matrix(y_train_pred_final.Churn,y_train_pred_final.predicted)
confusion
# Lets look at the sensitivity
TP = confusion[1,1]
TN = confusion[0,0]
FP = confusion[0,1]
FN = confusion[1,0]
TP/float(TP+FN)<jupyter_output><empty_output><jupyter_text>Now we see that sensitivity went up from 57 to 77 which means now, we can identify churners more precisely  Precision and Recall<jupyter_code>precision = TP/float(TP+FP)
recall = TP/float(TP+FN)
print("precision is: ",precision)
print("recall is: ",recall)
### using sklearn for precision or recall
from sklearn.metrics import precision_score,recall_score,precision_recall_curve

precision_score(y_train_pred_final.Churn,y_train_pred_final.predicted)
recall_score(y_train_pred_final.Churn,y_train_pred_final.predicted)<jupyter_output><empty_output><jupyter_text>Precision-Recall Tradeoff<jupyter_code>p,r,thresholds = precision_recall_curve(y_train_pred_final.Churn,y_train_pred_final.Churn_Prob)
print(p,r,thresholds)
plt.plot(thresholds,p[:-1],'g')
plt.plot(thresholds,r[:-1],'r')
plt.show()<jupyter_output><empty_output><jupyter_text>Acc to precision recall view, its around 0.42. But we will go with sensitivity-specificity view  Evaluating on Test Set<jupyter_code>##scale the features
X_test[['tenure','MonthlyCharges','TotalCharges']] = scaler.transform(X_test[['tenure','MonthlyCharges','TotalCharges']])
## use the selected features only
X_test = X_test[col]
X_test.head()
X_test_sm = sm.add_constant(X_test)
y_test_pred = res.predict(X_test_sm)
y_test_pred.head()
y_test_pred_final = pd.DataFrame(columns=["y_test_actual","y_test_probs","y_test_predicted"])
## using the sensi-speci cutoff of 0.3
y_test_pred_final["y_test_actual"] = y_test
y_test_pred_final['y_test_probs'] = y_test_pred
y_test_pred_final['y_test_predicted'] = y_test_pred_final.y_test_probs.apply(lambda x: 1 if x>0.3 else 0)
y_test_pred_final.head()
## lets look at the accuracy
metrics.accuracy_score(y_test_pred_final.y_test_actual,y_test_pred_final.y_test_predicted)<jupyter_output><empty_output><jupyter_text>Good accuracy compared to the train set<jupyter_code>confusion = metrics.confusion_matrix(y_test_pred_final.y_test_actual,y_test_pred_final.y_test_predicted)
confusion
# Lets look at the sensitivity
TP = confusion[1,1]
TN = confusion[0,0]
FP = confusion[0,1]
FN = confusion[1,0]
TP/float(TP+FN)
## lets look at specificity
TN/float(TN+FP)<jupyter_output><empty_output><jupyter_text>we got decent sensitivity and specificity values for the cut off of 0.3 (we can use prec-recall cutoff of 0.42 as well jic)  Methodology Followed Data cleaning and preparation- Combining three dataframes- Handling categorical variables    - Mapping categorical variables to integers    - Dummy variable creation- Handling missing values Test-train split and scaling Model Building- Feature elimination based on correlations- Feature selection using RFE (Coarse Tuning)- Manual feature elimination (using p-values and VIFs) Model Evaluation- Accuracy- Sensitivity and Specificity- Optimal cut-off using ROC curve- Precision and Recall Predictions on the test set<jupyter_code>1050/1400
1190/1400
1050/1450<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10815719114160319, 'fraction_numerical': 0.02502143974171417, 'mean_word_length': 5.895304347826087, 'pattern_counts': {'":': 2, '<': 115, '<?xml version=': 0, '>': 126, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '29', 'dir': 'jupyter-structured-clean-dedup', 'id': '500558', 'license': 'MIT', 'n_tokens_mistral': 7136, 'n_tokens_neox': 6711, 'n_words': 1575, 'path': 'Telecom Churn.ipynb', 'repo_name': 'sameerg07/TelecomChurn', 'size': '19823'}