<jupyter_start><jupyter_text>Pearson CorrelationCorrelation is the standardized covariance, i.e the covariance of x and y divided by the standard deviation of x and y. We can use this to check multicolinearity and also the direction and strength of the corelation.![pic](https://wikimedia.org/api/rest_v1/media/math/render/svg/f76ccfa7c2ed7f5b085115086107bbe25d329cec)<jupyter_code>corr=round(df[['tenure','MonthlyCharges','TotalCharges']].corr(),2)
plt.figure(figsize=(5,5))
mask=np.triu(np.ones_like(corr,dtype=np.bool))
cmap=sns.diverging_palette(220,10,as_cmap=True)
sns.heatmap(corr,square=True,annot=False,mask=mask,cmap=cmap,linewidths=0.5,cbar_kws={'shrink':0.5})
plt.show()<jupyter_output><empty_output><jupyter_text>We can observe that TotalCharges , Monthly Charges and tenure are **highly correlated** and there's a tendency that we will maintain Total Charges. We can consider dropping tenure, since TotalChargers = tenure * MonthlyChargers  Modelling<jupyter_code># Baseline
df.Churn.value_counts(normalize=True).round(2)<jupyter_output><empty_output><jupyter_text>Our baseline model is giving us an accuracy of 73%. It means that if we assign every row to class '0', we will be 73% correct<jupyter_code># Prepare Predictor and Target Variables
X = df.drop(['Churn'],axis=1)
y = df['Churn']
# Train-test-split with 20% test size and stratify set to yes
# as we want the train and test set to be balanced
X_train, X_test, y_train, y_test = train_test_split(
     X, y, test_size=0.2, random_state=42,stratify=y)<jupyter_output><empty_output><jupyter_text>Logistic Regression  Processing pipeline<jupyter_code># define the numerical and categorical columns
numerical =['MonthlyCharges','TotalCharges','tenure']
categorical= [ col for col in df.columns if col not in numerical and col!='Churn' and col!='tenure' ]

# all numeric columns will be standardized
numeric_transformer = Pipeline(steps=[
    ('scaler', StandardScaler())])

# all categorical columns will be One-Hot-Encoded
categorical_transformer = Pipeline(steps=[
    ('onehot', OneHotEncoder(handle_unknown='ignore'))])

# we will process numerical and categorical columns in parallel
preprocessor = ColumnTransformer(
    transformers=[
        ('cat_multi', categorical_transformer, categorical),
        ('num', numeric_transformer, numerical)])<jupyter_output><empty_output><jupyter_text>We will be using pipelines and ColumnTransformer so as to assist us with better workflow and especially for deployment.The purpose of the pipeline is to assemble several steps that can be cross-validated together while setting different parameters.  RFECV RFE is a wrapper-type feature selection algorithm. RFE works by searching for a subset of features by starting with all features in the training dataset and successfully removing features until the desired number remains.This is achieved by fitting the given machine learning algorithm used in the core of the model, ranking features by importance, discarding the least important features, and re-fitting the model. This process is repeated until a specified number of features remains.RFECV means we include Cross Validation when we fit  **F1 scoring** F1 scoring is used for RFECV feature selection because F1 Score keeps a balance between Precision and Recall.AUC- ROC vs F1 Score In general, the ROC is used for many different levels of thresholds and thus it has many F score values. F1 score is applicable for any particular point on the ROC curve.You may think of it as a measure of precision and recall at a particular threshold value whereas AUC is the area under the ROC curve. For F score to be high, both precision and recall should be high.When you have a data imbalance between positive and negative samples, you should always use F1-score because of ROC averages over all possible thresholds.![pic](https://lh4.googleusercontent.com/kGNsnAScbuhUL2JXNXmoKQhhi8N6P7yT1yrAt_KEaPet_1XiptjcutS6Xi83Ja1EbIJbeEjDYFVJI9fQwJVAtl31o4CJIQEtNvcdeB2VMiVYBxyULgaXO1JMUc-F2CcDn_Wlpd_R)<jupyter_code># fit RFECV using LogReg to obtain the best features

estimator = LogisticRegression(max_iter=1000, tol=0.1,random_state=42)
pipe_rfecv = Pipeline(steps=[('preprocessor', preprocessor),
                      ('rfecv',RFECV(estimator, step=1, cv=5,scoring='f1',n_jobs=-1))])
pipe_rfecv.fit(X_train,y_train)
# Plot number of features VS. cross-validation scores
plt.figure(figsize=(15, 6))
plt.xlabel("Number of features selected")
plt.ylabel("Cross validation score (f1_score)")
plt.plot(range(1, len(pipe_rfecv['rfecv'].grid_scores_) + 1), pipe_rfecv['rfecv'].grid_scores_)
plt.show()
print("Optimal number of features : %d" % pipe_rfecv['rfecv'].n_features_)
# features that should be selected based on the boolean below
pipe_rfecv['rfecv'].support_
# obtain the categorical columns that are One Hot Encoded from pipe_rfecv
cat = pipe_rfecv['preprocessor'].transformers_[0][1]['onehot']\
                         .get_feature_names(categorical)
cat
# obtain numerical columns
numerical
# combine both categorical and numeric horizontally
cat_numeric = np.hstack([cat, numerical])

# use mask to filter 
cat_numeric[pipe_rfecv['rfecv'].support_]
#manually filter out the important columns as they are dummified
numerical =['MonthlyCharges','TotalCharges','tenure']
categorical = ['SeniorCitizen','Dependents','MultipleLines','InternetService','OnlineSecurity','OnlineBackup',\
               'DeviceProtection','TechSupport','StreamingTV','StreamingMovies','Contract','PaperlessBilling','PaymentMethod']
# we re-initialize the numerical and categorical columns
# previously was all columns before RFECV
preprocessor = ColumnTransformer(
    transformers=[
        ('cat_multi', categorical_transformer, categorical),
        ('num', numeric_transformer, numerical)])<jupyter_output><empty_output><jupyter_text>k-Fold Cross Validation **Cross-validation** is primarily used in applied machine learning to estimate the skill of a machine learning model on unseen data. That is, to use a limited sample in order to estimate how the model is expected to perform in general when used to make predictions on data not used during the training of the model.If we set cv = 5, it means we will iterate 4 times with 5 portions divided into 4 Training Data and 1 Test Data as per illustration below. ![pic](https://miro.medium.com/max/523/1*C5FJt_NH1BWJrFSvw_6jtw.png)<jupyter_code>def eval_model(model_name,parameter):
    '''This function takes in a Pipeline and its parameters. It prints the validation auc-roc score, best parameters,
    score on training data, recall score of model, precision score of model, accuracy score of model,
    ,confusion matrix,roc-auc chart. 
    
    And important parameters in dictionary format for model comparison purposes'''
    
    # Gridsearch with CV based on the model's parameter, metric used is 'roc_auc', utilizing all the CPU cores
    search = GridSearchCV(pipe,parameter, cv=5, scoring = 'f1',verbose=1,n_jobs=-1)
    search = search.fit(X_train,y_train)
    
    # Dump model for future use or deployment 
    pickle.dump(search, open('data/{}_model.pkl'.format(model_name), 'wb'))

    #Best score ROC-AUC from GridSearchCV
    best_score = search.best_score_.round(3)
    print(f'CV F1 Score: {best_score}')
      
    #Best Parameters from GridSearchCV
    best_params = search.best_params_
    print(f'Best Params: {best_params}')
    
    #save Best model from GridSearchCV
    best_model = search.best_estimator_ 
    
    #Make predictions using best model on test data
    y_pred = search.predict(X_test)
    
    #Plot confusion matrix ( normalize and non-normalize)
    plot_confusion_matrix(search.best_estimator_, X_test, y_test,normalize='true',display_labels=['No Churn','Churn']);
    plot_confusion_matrix(search.best_estimator_, X_test, y_test,display_labels=['No Churn','Churn'])
    
    #Plot roc auc curve
    plot_roc_curve(search.best_estimator_, X_test, y_test);
    
    # Print Test metrics : Accuracy, Precision, Recall
    accuracy = accuracy_score(y_test,y_pred).round(3)
    print(f'Test Accuracy: {accuracy}')
    precision = precision_score(y_test,y_pred).round(2)
    print(f'Test Precision:  {precision}')
    recall = recall_score(y_test,y_pred).round(2)
    print(f'Test Recall: {recall}')
    f1 = f1_score(y_test,y_pred, average='micro').round(2)
    print(classification_report(y_test,y_pred,target_names=['No Churn','Churn']))
    
    model_dict = {}
    model_dict['val_f1'] = best_score
    model_dict['test_scores'] = ' '
    model_dict['f1'] = f1
    model_dict['accuracy'] = accuracy
    model_dict['precision'] = precision
    model_dict['recall'] = recall
    model_dict['best_params'] = [best_params]
    tn, fp, fn, tp = confusion_matrix(y_test, y_pred, labels=[0,1]).ravel()
    model_dict['cost_benefit'] = (80 * tp * 11 ) +\
                                 (65 * tn * 12 ) +\
                                 (65 * fp * 11 ) +\
                                 (-80 * fn * 12 ) 
    print(model_dict['cost_benefit'])
    # return the best model
    return search , model_dict 
# we will process the columns then we will do logreg modelling
pipe = Pipeline(steps=[('preprocessor', preprocessor),
                      ('logreg',LogisticRegression(max_iter=1000, tol=0.1,random_state=42))])

# parameters we will grid search
logreg_param = {
    'logreg__C': [1e-2, 1e-1, 1, 5, 10,20,30,40,50],
    'logreg__penalty' : ['l1','l2'],
    'logreg__solver' : ['lbfgs', 'liblinear','saga'],
    'logreg__class_weight' : ['balanced', None]
}

# run model evaluation
logreg,logreg_dict = eval_model('logreg',logreg_param)<jupyter_output>Fitting 5 folds for each of 108 candidates, totalling 540 fits<jupyter_text>Random forest<jupyter_code>categorical= [ col for col in df.columns if col not in numerical and col!='Churn' and col!='tenure' ]
# all numeric columns will be standardized
numeric_transformer = Pipeline(steps=[
    ('scaler', StandardScaler())])
# all categorical columns will be One-Hot-Encoded
categorical_transformer = Pipeline(steps=[
    ('onehot', OneHotEncoder(handle_unknown='ignore'))])
# we will process numerical and categorical columns in parallel
preprocessor = ColumnTransformer(
    transformers=[
        ('cat_multi', categorical_transformer, categorical),
        ('num', numeric_transformer, numerical)])
# fit RFECV using LogReg to obtain the best features
estimator = RandomForestClassifier(random_state=42)
pipe_rfecv = Pipeline(steps=[('preprocessor', preprocessor),
                      ('rfecv',RFECV(estimator, step=1, cv=5,scoring='f1_micro',n_jobs=-1))])
pipe_rfecv.fit(X_train,y_train)
# Plot number of features VS. cross-validation scores
plt.figure(figsize=(15, 6))
plt.xlabel("Number of features selected")
plt.ylabel("Cross validation score (nb of correct classifications)")
plt.plot(range(1, len(pipe_rfecv['rfecv'].grid_scores_) + 1), pipe_rfecv['rfecv'].grid_scores_)
plt.show()
print("Optimal number of features : %d" % pipe_rfecv['rfecv'].n_features_)
pipe_rfecv['rfecv'].support_
cat = pipe_rfecv['preprocessor'].transformers_[0][1]['onehot']\
                         .get_feature_names(categorical)
numerical
cat_numeric = np.hstack([cat, numerical])
cat_numeric[pipe_rfecv['rfecv'].support_]
#manually filter out the  important columns as the columns are dummified
numerical =['MonthlyCharges','TotalCharges','tenure']
categorical = ['gender','SeniorCitizen','Partner','Dependents','MultipleLines','InternetService','OnlineSecurity','OnlineBackup',
               'DeviceProtection','TechSupport','StreamingTV','StreamingMovies','Contract','PaperlessBilling','PaymentMethod']
# we will process numerical and categorical columns in parallel
preprocessor = ColumnTransformer(
    transformers=[
        ('cat_multi', categorical_transformer, categorical),
        ('num', numeric_transformer, numerical)])
pipe = Pipeline(steps=[('preprocessor', preprocessor),
                      ('rf',RandomForestClassifier(random_state=42))])

rf_param = {
    'rf__n_estimators': [175,200,225],
    'rf__max_depth' : [7],
    'rf__min_samples_split' : [2],
    'rf__min_samples_leaf' : [2],
    'rf__oob_score' : ['True'],
    'rf__class_weight' : ['balanced', 'balanced_subsample',None]
}
rf,rf_dict = eval_model('rf',rf_param)
pipe.fit(X,y)
feat = pipe['preprocessor'].transformers_[0][1]['onehot']\
                    .get_feature_names(categorical)
feat = np.concatenate((feat, numerical))
pd.DataFrame(data=rf.estimator[1].feature_importances_,index = feat)
# display top 10 features
feat_importances = pd.Series(rf.estimator[1].feature_importances_ ,index=feat)
feat_importances.nlargest(10).plot(kind='barh');<jupyter_output><empty_output><jupyter_text>SVChttps://medium.com/all-things-ai/in-depth-parameter-tuning-for-svc-758215394769 * **C** is the penalty parameter of the error term. It controls the trade off between smooth decision boundary and classifying the training points correctly.* **gamma** is a parameter for non linear hyperplanes. The higher the gamma value it tries to exactly fit the training data set* **kernel** parameters selects the type of hyperplane used to separate the data. Using ‘linear’ will use a linear hyperplane (a line in the case of 2D data). ‘rbf’ and ‘poly’ uses a non linear hyper-plane<jupyter_code>categorical= [ col for col in df.columns if col not in numerical and col!='Churn' and col!='tenure' ]
# all numeric columns will be standardized
numeric_transformer = Pipeline(steps=[
    ('scaler', StandardScaler())])
# all categorical columns will be One-Hot-Encoded
categorical_transformer = Pipeline(steps=[
    ('onehot', OneHotEncoder(handle_unknown='ignore'))])
# we will process numerical and categorical columns in parallel
preprocessor = ColumnTransformer(
    transformers=[
        ('cat_multi', categorical_transformer, categorical),
        ('num', numeric_transformer, numerical)])
estimator = LinearSVC()
pipe_rfecv = Pipeline(steps=[('preprocessor', preprocessor),
                      ('rfecv',RFECV(estimator, step=1, cv=5,scoring='f1_micro',n_jobs=-1))])
pipe_rfecv.fit(X_train,y_train)
pipe_rfecv['rfecv'].support_
cat = pipe_rfecv['preprocessor'].transformers_[0][1]['onehot']\
                         .get_feature_names(categorical)
numerical
cat_numeric = np.hstack([cat, numerical])
cat_numeric[pipe_rfecv['rfecv'].support_]
#manually filter out the  important columns as the columns are dummified
numerical =['MonthlyCharges','tenure']
categorical = ['PhoneService','MultipleLines','InternetService','OnlineSecurity','OnlineBackup','DeviceProtection',
               'TechSupport','StreamingTV','StreamingMovies','Contract','PaymentMethod']
svc_param = {
    'SVC__C':[1,5,6,7,8,910], #It controls the trade off between smooth decision boundary and classifying the training points correctly.
    'SVC__class_weight' : ['balanced',None]
}
pipe = Pipeline(steps=[('preprocessor', preprocessor),
                      ('SVC',LinearSVC(random_state=42,verbose=True))])
svc,svc_dict= eval_model('svc',svc_param)<jupyter_output>Fitting 5 folds for each of 12 candidates, totalling 60 fits<jupyter_text>Comparison between the 3 models<jupyter_code>pd.DataFrame([logreg_dict,rf_dict,svc_dict],\
            index=['LogReg', 'RF', 'LinearSVC']).T<jupyter_output><empty_output><jupyter_text>Cost Benefit Analysis<jupyter_code># Median Monthly Charges for Churn
df[df['Churn']==1][['MonthlyCharges']].median() #Approx $80
# Median Monthly Charges for  No Churn
df[df['Churn']==0][['MonthlyCharges']].median() #Approx $65<jupyter_output><empty_output><jupyter_text>From the above, we can see that 'No Churn' customers has lesser subscription than 'Churn' customers.We will select model based on **f1 validation score** and from here we can see that Logistic Regression has the best value of 0.639 and we verified using the Cost_Benefit analysis which is based on 1 calendar year. With LogReg, we are able to have an estimated Sales Revenue of **$973,485** **True Positive** =  correctly predict Churn  Total Revenue = $80 *  tp * 11Remarks : We provide free 1 month subscription to entice customer to stay but change to contract 1 year**True Negative** = correctly predict No Churn  Total Revenue = $65 *  tn * 12 Remarks :  Business as usual **False Positive** = incorrectly predict Churn **Type 1 Error** Total Revenue = $65 *  fp * 11Remarks : Not as bad as FN as we only lose 1 month of revenueWe provide free 1 month subscription to entice customer to stay but change to contract 1 year**False Negative** = incorrectly predict No Churn **Type 2 Error**Total Revenue = -$80 *  fn * 12 Remarks :Total revenue loss :(**Note :**If we go by the baseline/naive way/predict all no churn, we will totally lose at least 380 x 12 x $79 = $360,240.<jupyter_code>numerical =['MonthlyCharges','TotalCharges','tenure']
categorical = ['SeniorCitizen','Dependents','MultipleLines','InternetService','OnlineSecurity','OnlineBackup',\
               'DeviceProtection','TechSupport','StreamingTV','StreamingMovies','Contract','PaperlessBilling','PaymentMethod']
preprocessor = ColumnTransformer(
    transformers=[
        ('cat_multi', categorical_transformer, categorical),
        ('num', numeric_transformer, numerical)])


pipe = Pipeline(steps=[('preprocessor', preprocessor),
                      ('logreg',LogisticRegression(max_iter=1000, tol=0.1,random_state=42))])

# 
logreg_param = {
    'logreg__C': [1e-2, 1e-1, 1, 5, 10,20,30,40,50],
    'logreg__penalty' : ['l1','l2'],
    'logreg__solver' : ['lbfgs', 'liblinear','saga'],
    'logreg__class_weight' : ['balanced', None]
}

# run model evaluation
logreg,logreg_dict = eval_model('logreg',logreg_param)


cat_column = logreg.best_estimator_.named_steps["preprocessor"].transformers_[0][1]['onehot'].get_feature_names(categorical).tolist()
num_column = ['MonthlyCharges','TotalCharges','tenure','intercept']

column_name= cat_column  + num_column<jupyter_output><empty_output><jupyter_text>Factors that contributed to Churn (+ve)If the customer is on a **Month_to_Month** contract the tendency will be **twice as likely** since they can leave for another company easily. At the opposite side of the spectrum, it's tenure and contract of two years.If the probability of success is 50%, then the odds of success are 1.If the probability of success is 100%, then the odds of success are $\infty$<jupyter_code>intercept = np.exp(logreg.best_estimator_.named_steps['logreg'].intercept_).round(2).tolist()
# convert logit(log-odds) to odds by np.exp() for interpretation purposes
# IF we need to convert to probability we need to odds/( 1+ odds)
coef=np.exp(logreg.best_estimator_.named_steps['logreg'].coef_).round(2).tolist()[0]
coef_intercept = coef + intercept
coef=np.exp(logreg.best_estimator_.named_steps['logreg'].coef_[0]).round(2)
odds = pd.DataFrame(coef_intercept,index=column_name,columns=["coefficient_odds"]).sort_values('coefficient_odds',ascending=False)
odds
# to display in deployment
odds.to_csv('data\odds.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11285744322960471, 'fraction_numerical': 0.017609335576114382, 'mean_word_length': 5.358622994652406, 'pattern_counts': {'":': 0, '<': 47, '<?xml version=': 0, '>': 47, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '415171', 'license': 'CNRI-Python', 'n_tokens_mistral': 6181, 'n_tokens_neox': 5779, 'n_words': 1656, 'path': '.ipynb_checkpoints/2_Modelling-checkpoint.ipynb', 'repo_name': 'andrewng88/telco_churn', 'size': '19024'}