<jupyter_start><jupyter_text>Final Model (6sense)IntroductionFindings        Finding 1     Finding 2     Data Pre-processing         Merge Table     Feature Engineering     Data Visualization     Building Model         KNN     Logistic Regression SVM -->    Decision TreeRandom Forest -->    AdaBoost    XGBoost    LightGBM Classifier    ROC Curves     Conclusion        Introduction> This notebook is for training the 6sense data through different models (such as Random Froest, KNN, XGBoost and so on) and find the best fit model through several variables including call disposition, best day of week to call and best hour to call, predicting the success or not if a customer would pick up phone calls so that the company can get engagement with prospective customers.<jupyter_code>#import library
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import itertools
%matplotlib inline
from sklearn.model_selection import train_test_split,cross_val_score,KFold,cross_val_predict
from sklearn.metrics import accuracy_score, classification_report, precision_score, recall_score,confusion_matrix,precision_recall_curve,roc_curve
from sklearn.feature_selection import RFE
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import ExtraTreesClassifier,RandomForestClassifier,AdaBoostClassifier,GradientBoostingClassifier
from sklearn.neighbors  import KNeighborsClassifier
from sklearn import svm,tree
#import data
calls = pd.read_csv('calls.csv')
companies = pd.read_csv('companies.csv')
events = pd.read_csv('events.csv')
opportunies = pd.read_csv('opportunities.csv')
people = pd.read_csv('people.csv')<jupyter_output><empty_output><jupyter_text>Findings > There are two new findings based on our dataset. Finding 1: The most activities (194 times) company did not pick up 5 phone calls.<jupyter_code>#sorted sum of the events for grouped id
event_counts_by_id = events.groupby('contact_id').count().sort_values('date', ascending=False).reset_index().drop(['date','activity_type'], axis=1)
event_counts_by_id
#the call response with most events company
calls.query("contact_id == 7505295075260161764")<jupyter_output><empty_output><jupyter_text>Finding 2: Scatterplot Display  Important Step: Mapping the Positive Variables<jupyter_code>calls.call_disposition.value_counts()
#if positive responses return 1 in call_disposition column
def get_pos(x):
    if x == 'MC - Meaningful Conversation':
        return 1
    elif x == 'MC - Meeting Confirmation':
        return 1
    elif x == 'MC - MQL/QMA':
        return 1
    elif x == 'MC - No Need':
        return 1
    elif x == 'MC - No Authority':
        return 1
    elif x == 'MC - No Budget':
        return 1
    elif x == 'MC - No Timeline':
        return 1
    elif x == 'MC - No Technical Fit':
        return 1
    elif x == 'MC - Call Back':
        return 1
    else:
        return 0
#sum up the positive responses to call_disposition
calls_temp = calls.copy()
calls_temp.call_disposition = calls_temp.call_disposition.apply(get_pos)
calls_temp = calls_temp.drop('date', axis=1).groupby('contact_id').sum().reset_index()
calls_temp.head()
# value counts for each positive responses
calls_temp['call_disposition'].value_counts()
#find out the relationship between call response and events times
#combine call and events table
combined = calls_temp.merge(event_counts_by_id, on='contact_id', how='left')
combined = combined.dropna()
combined
combined['call_disposition'].value_counts()
combined[combined['call_disposition'] > 0].shape[0]
combined.shape[0]
combined[combined['call_disposition'] > 0].shape[0] / combined.shape[0]<jupyter_output><empty_output><jupyter_text>We have 8647 positive responses over 22251 calls. Only 38.9% rows are meaningful conversation.<jupyter_code># calls.query("call_disposition == 'MC - Meaningful Conversation'")
#draw scatterplot 
plt.figure(figsize=(12,8))
plt.scatter(combined.activity_action, combined.call_disposition)
plt.title('The Relationship Between Counts of Events and Positive Response')
plt.xlabel('Counts of Events (regardless of type)')
plt.ylabel('Counts of Positive Responses')
plt.show()<jupyter_output><empty_output><jupyter_text>Based on the scatterplot:1. Most companies have many counts of events and no positive responses.2. Only few of the companies have high positive responses.  Data Pre-processingThe data from 6sense:calls.csv: a timeline of outgoing sales calls and the disposition of those callsevents.csv: any activities that we have on record taking place before the phone calls were madecompanies.csv: the industry and employee count of the companiespeople.csv: the people who were called, along with their job level and function and the ID of the company they work foropportunities.csv: the date an opportunity was generated for a contact> In this part, we will gather three parts of data  1. Calls table    Add weekday columns.    Map positive response of phone calls and calculate the sum.    Find out max and min duration of each call date.<jupyter_code>type(calls.date[0])
#change the date from string to datetime
new_calls = calls.copy()
new_calls['date'] = pd.to_datetime(new_calls['date'], format='%Y-%m-%d')
new_calls
#add weekday column (Monday is 0 and Sunday is 6) [this tbl is not unique id]
weekday = []
for i in new_calls['date']:
    weekday.append(i.weekday())
new_calls['weekday'] = weekday   
new_calls
#positive responses
def get_pos(x):
    if x == 'MC - Meaningful Conversation':
        return 1
    elif x == 'MC - Meeting Confirmation':
        return 1
    elif x == 'MC - MQL/QMA':
        return 1
    elif x == 'MC - No Need':
        return 1
    elif x == 'MC - No Authority':
        return 1
    elif x == 'MC - No Budget':
        return 1
    elif x == 'MC - No Timeline':
        return 1
    elif x == 'MC - No Technical Fit':
        return 1
    elif x == 'MC - Call Back':
        return 1
    else:
        return 0
#sum of the positive responses with unique id
new_calls_disposition = new_calls.copy()
new_calls_disposition.call_disposition = new_calls_disposition.call_disposition.apply(get_pos)
new_calls_disposition = new_calls_disposition.drop(['date', 'weekday'], axis=1).groupby('contact_id').sum().sort_values('call_disposition', ascending=False).reset_index()
new_calls_disposition.head()
#get the most frequency of weekday with unique id
new_calls_weekday = new_calls.copy()
new_calls_weekday = new_calls_weekday.drop(['date', 'call_disposition'], axis=1).groupby(['contact_id']).agg(lambda x:x.value_counts().index[0])
new_calls_weekday.head()
#find out max and min duration of each call date 
min_diff = []
max_diff = []
unique_contact = new_calls["contact_id"].unique()
for i in unique_contact:
    max_diff = np.append(max_diff, max(new_calls[new_calls["contact_id"] == i]["date"].sort_values(ascending = True).diff().fillna(pd.Timedelta(seconds=0)).dt.days))
    min_diff = np.append(min_diff, min(new_calls[new_calls["contact_id"] == i]["date"].sort_values(ascending = True).diff().fillna(pd.Timedelta(seconds=0)).dt.days))
max_min_call = pd.DataFrame({'contact_id' : new_calls["contact_id"].unique(), 'min_diff_call' : min_diff, 'max_diff_call' : max_diff})
max_min_call
#duration of call (last day call - first day call)
new_calls_time = new_calls.groupby(['contact_id']).agg(lambda x: max(x) - min(x)).sort_values('date', ascending=False).reset_index()
new_calls_time
#double check if it is true
new_calls.query('contact_id == 166566594430710')
#merge the new_calls_time and new_calls_disposition
new_calls_merge = new_calls_disposition.merge(new_calls_time, on='contact_id', how='left')
new_calls_merge<jupyter_output><empty_output><jupyter_text>Feature Engineering 2. People table & Companies table    Merge two table based on contact_id.    Apply numerical features to job_level (numerical 0 - Other, 1 - Staff ...)    One-hot encoding based on industry.<jupyter_code>#merge the people and companies with the unique company_id
pplcomp = people.merge(companies, on = ['company_id'])
pplcomp.head(10)
#apply numerical features to job_level (numerical 0 - Other, 1 - Staff ...)
job_level_convert = {"Other" :0, "Staff":1, "Manager":2, "Director":3, "Senior":4, "Vice President":5, "C-Level":6}
pplcomp["job_level"] = pplcomp["job_level"].map(job_level_convert)   
pplcomp.head(10)
#one-hot encoding based on industry (one-hot encoding)
dum_industry = pd.get_dummies(pplcomp["industry"])
pplcomp = pplcomp.drop(["industry"],axis=1).merge(dum_industry,left_index = True,right_index=True)
pplcomp.head(10)
#drop useless columns
new_merge_people_companies = pplcomp.drop(['job_function', 'employee_range', 'company_id'], axis=1)
new_merge_people_companies<jupyter_output><empty_output><jupyter_text>3. Events Table    Find out max and min duration of each event date.    Calculate the duration of events (last day events - first day events)<jupyter_code>new_events = events.copy()
new_events['date'] = pd.to_datetime(new_events['date'], format='%Y-%m-%d')
new_events.head()
#find out max and min duration of call date 
min_diff_events = []
max_diff_events = []
unique_contacts = new_events["contact_id"].unique()
for i in unique_contacts:
    max_diff_events = np.append(max_diff_events, max(new_events[new_events["contact_id"] == i]["date"].sort_values(ascending = True).diff().fillna(pd.Timedelta(seconds=0)).dt.days))
    min_diff_events = np.append(min_diff_events, min(new_events[new_events["contact_id"] == i]["date"].sort_values(ascending = True).diff().fillna(pd.Timedelta(seconds=0)).dt.days))
max_min_events = pd.DataFrame({'contact_id' : new_events["contact_id"].unique(), 'min_diff_events' : min_diff_events, 'max_diff_events' : max_diff_events})
max_min_events
#calculate the duration of events (last day events - first day events)
new_events_time_temp = new_events.groupby(['contact_id']).agg(lambda x: max(x) - min(x)).sort_values('date', ascending=False).reset_index()
new_events_time_temp
new_events_time = new_events_time_temp.merge(combined, on='contact_id').drop('call_disposition', axis=1)
new_events_time<jupyter_output><empty_output><jupyter_text>Merge All Tables<jupyter_code>#merge unique contact_id with these two new table
combine_time = new_calls_merge.merge(new_events_time, on='contact_id',how='left').rename(columns={'date_x':'calls_time', 'date_y':'events_time'})
combine_time = combine_time.merge(new_calls_weekday, on='contact_id',how='left').rename(columns={'weekday_y':'weekday'}).drop('weekday_x', axis=1)
combine_time = combine_time.merge(max_min_call, on='contact_id',how='left')
combine_time = combine_time.merge(max_min_events, on='contact_id',how='left')
combine_time = combine_time.merge(new_merge_people_companies, on='contact_id',how='left')
#set the success to 1, fail to 0
success = []
for i in combine_time['call_disposition']:
    if i > 0:
        success = np.append(success, 1)
    else:
        success = np.append(success, 0)

#new column success
combine_time['success'] = success
combine_time['calls_time'] = pd.to_timedelta(combine_time.calls_time, errors='coerce').dt.days
combine_time['events_time'] = pd.to_timedelta(combine_time.events_time, errors='coerce').dt.days.fillna(0)
combine_time['activity_action'] = combine_time['activity_action'].fillna(0)
combine_time['min_diff_events'] = combine_time['min_diff_events'].fillna(0)
combine_time['max_diff_events'] = combine_time['max_diff_events'].fillna(0)
combine_time = combine_time.drop(['min_diff_events', 'min_diff_call'], axis=1)
combine_time.iloc[:, 8:40] = combine_time.iloc[:, 8:40].fillna(0)<jupyter_output><empty_output><jupyter_text>Feature: Descriptioncall_disposition:  sum of the positive response    calls_time:  duration of all calls (last day call - first day call)    events_time:  duration of all events (last day event - first day event)activity_action:  sum of the event timesweekday:  the most frequent of weekday with each idmax_diff_call:  maximum duration between two consecutive call dates    max_diff_events:  maximum duration between two consecutive event dates    job_level:  numerical features of job_level (0 - Other, 1 - Staff, 2 - Manager, 3 - Director, 4 - Senior, 5 - Vice President, 6 - C-Level)    success:  1 - positive response, 0 - negative responseIndustry (one-hot encoding features):        'Aerospace and Defense', 'Agriculture', 'Apparel', 'Associations',       'Automotive', 'Biotech', 'Business Services', 'Construction',       'Consumer Goods and Services', 'Education', 'Energy and Utilities',       'Financial Services', 'Food and Beverage', 'Furniture', 'Government',       'Hardware', 'Healthcare and Medical', 'Home and Garden',       'Hospitality and Travel', 'Manufacturing', 'Media and Entertainment',       'Mining', 'Pharmaceuticals', 'Printing and Publishing', 'Real Estate',       'Recreation', 'Retail and Distribution', 'Software and Technology',       'Telecommunications', 'Textiles', 'Transportation and Logistics',       'success'<jupyter_code>#merge table
combine_time
combine_time.columns<jupyter_output><empty_output><jupyter_text>Data Visualization<jupyter_code>combine_time_sub = ['calls_time', 'events_time', 'activity_action', 'weekday',\
          'max_diff_call', 'max_diff_events', 'job_level', 'success']
sns.pairplot(combine_time[combine_time_sub],hue='success', height=1.5);<jupyter_output>/Users/ceciliabeauty/opt/anaconda3/lib/python3.8/site-packages/seaborn/distributions.py:369: UserWarning: Default bandwidth for data is 0; skipping density estimation.
  warnings.warn(msg, UserWarning)
/Users/ceciliabeauty/opt/anaconda3/lib/python3.8/site-packages/seaborn/distributions.py:369: UserWarning: Default bandwidth for data is 0; skipping density estimation.
  warnings.warn(msg, UserWarning)<jupyter_text>Key Takeaways from the Pairplot* people with more activity with 6Sense is more likely to be predicted as success* people with more events with 6Sense is more likely to be predicted as success<jupyter_code>#Setting up correlation for our dataframe and passing it to seaborn heatmap function
sns.set(style="white")
df1 = combine_time[combine_time_sub]
corr = df1.corr()
mask = np.zeros_like(corr, dtype=np.bool)
mask[np.triu_indices_from(mask)] = True
f, ax = plt.subplots(figsize=(11, 9))
cmap = sns.diverging_palette(220, 10, as_cmap=True)
sns.heatmap(corr,annot=True, mask=mask, cmap=cmap, vmax=.3, center=0,
            square=True, linewidths=.5, cbar_kws={"shrink": .5});<jupyter_output><empty_output><jupyter_text>Key Takeaways from the Correlation Plot* activity_action and events_time, max_diff_events and acticity_action, are have positive corrlation* max_diff_call and calls_time, max_diff_events and events_time, are have a larger positive correlation* people with more activity with 6Sense is more likely to be predicted as success* people with more events with 6Sense is more likely to be predicted as success  Start to Build Model>     Idea: using different model to predict if we can get positive response from the customer.    Target label: success -- times of positive responses for one contact    Feature: calls_time(the differece between starting time and ending time), events_time, activity_action(times of activities)    Goal: after running the model, we can predict if the customer is positive. If it is positive, then the customer is in our potential customer list.<jupyter_code>X = combine_time.iloc[:, 2:40].values
y = combine_time['success'].values

from sklearn.model_selection import train_test_split
# x_train, x_test, y_train, y_test = train_test_split(x, y, test_size = 0.33, random_state = 42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20,random_state=42, stratify = y)
#The code for the below matrix is taken from sklearn documentation
#Defining the confusion matrix function
def plot_confusion_matrix(cm, classes,
                          normalize=False,
                          title='Confusion matrix',
                          cmap=plt.cm.Blues):
    
   
    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)
    


    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, cm[i, j],
                 horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black")
        
        
    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')
#Using Success and Failure for 0 and 1    
class_names = ['Success','Failure']<jupyter_output><empty_output><jupyter_text>Model Building ProcessWe apply 6 Classification Predictor Algorithms to our predictive model as the list below. Then use 10-fold to get cross validation score and do the model validation.1. KNN2. Logistic Regression3. Decision Tree4. AdaBoost5. XGBoost6. LightGBM Classifier  KNN<jupyter_code># Defining the kNNClassifier with 6 neighbors
knn = KNeighborsClassifier(n_neighbors = 6)
#Fitting the classifier to the training set
knn.fit(X_train,y_train)
print ("kNN Accuracy is %2.2f" % accuracy_score(y_test, knn.predict(X_test)))
#The cross validation score is obtained for kNN using 10 folds
score_knn = cross_val_score(knn, X, y, cv=10).mean()
print("Cross Validation Score = %2.2f" % score_knn)
y_pred= knn.predict(X_test)
print(classification_report(y_test, y_pred))
#Defining the confusion matrix
cm = confusion_matrix(y_test,y_pred)
#Plotting the confusion matrix
plot_confusion_matrix(cm, classes=class_names, title='Confusion matrix')<jupyter_output>kNN Accuracy is 0.77
Cross Validation Score = 0.76
              precision    recall  f1-score   support

         0.0       0.79      0.95      0.87     10105
         1.0       0.41      0.12      0.18      2834

    accuracy                           0.77     12939
   macro avg       0.60      0.54      0.53     12939
weighted avg       0.71      0.77      0.72     12939<jupyter_text>Logistic Regression<jupyter_code>#Logistic Regression Classifier
LR = LogisticRegression(max_iter=7600)
LR.fit(X_train,y_train)
print ("Logistic Accuracy is %2.2f" % accuracy_score(y_test, LR.predict(X_test)))
score_LR = cross_val_score(LR, X, y, cv=10).mean()
print("Cross Validation Score = %2.2f" % score_LR)
y_pred = LR.predict(X_test)
print(classification_report(y_test, y_pred))
# Confusion matrix for LR
cm = confusion_matrix(y_test,y_pred)
plot_confusion_matrix(cm, classes=class_names, title='Confusion matrix')<jupyter_output>Logistic Accuracy is 0.78
Cross Validation Score = 0.78
              precision    recall  f1-score   support

         0.0       0.79      0.99      0.87     10105
         1.0       0.45      0.04      0.07      2834

    accuracy                           0.78     12939
   macro avg       0.62      0.51      0.47     12939
weighted avg       0.71      0.78      0.70     12939<jupyter_text>SVM --><jupyter_code># #SVM Classifier
# SVM = svm.SVC()
# SVM.fit(X_train, y_train)
# print ("SVM Accuracy is %2.2f" % accuracy_score(y_test, SVM.predict(X_test)))
# score_svm = cross_val_score(SVM, X, y, cv=10).mean()
# print("Cross Validation Score = %2.2f" % score_svm)
# y_pred = SVM.predict(X_test)
# print(classification_report(y_test,y_pred))
# #Confusion matrix for SVM
# cm = confusion_matrix(y_test,y_pred)
# plot_confusion_matrix(cm, classes=class_names, title='Confusion matrix')<jupyter_output><empty_output><jupyter_text>Decision Tree<jupyter_code># Decision Tree Classifier
DT = tree.DecisionTreeClassifier(random_state = 0,class_weight="balanced",
    min_weight_fraction_leaf=0.01)
DT = DT.fit(X_train,y_train)
print ("Decision Tree Accuracy is %2.2f" % accuracy_score(y_test, DT.predict(X_test)))
score_DT = cross_val_score(DT, X, y, cv=10).mean()
print("Cross Validation Score = %2.2f" % score_DT)
y_pred = DT.predict(X_test)
print(classification_report(y_test, y_pred))
# Confusion Matrix for Decision Tree
cm = confusion_matrix(y_test,y_pred)
plot_confusion_matrix(cm, classes=class_names, title='Confusion matrix')<jupyter_output>Decision Tree Accuracy is 0.69
Cross Validation Score = 0.69
              precision    recall  f1-score   support

         0.0       0.86      0.72      0.79     10105
         1.0       0.38      0.60      0.46      2834

    accuracy                           0.69     12939
   macro avg       0.62      0.66      0.62     12939
weighted avg       0.76      0.69      0.72     12939<jupyter_text>Random Forest --><jupyter_code># #Random Forest Classifier
# rfc = RandomForestClassifier(n_estimators=1000, max_depth=None, min_samples_split=10,class_weight="balanced")
# rfc.fit(X_train, y_train)
# print ("Random Forest Accuracy is %2.2f" % accuracy_score(y_test, rfc.predict(X_test)))
# score_rfc = cross_val_score(rfc, X, y, cv=10).mean()
# print("Cross Validation Score = %2.2f" % score_rfc)
# y_pred = rfc.predict(X_test)
# print(classification_report(y_test,y_pred ))
# #Confusion Matrix for Random Forest
# cm = confusion_matrix(y_test,y_pred)
# plot_confusion_matrix(cm, classes=class_names, title='Confusion matrix')<jupyter_output><empty_output><jupyter_text>AdaBoost<jupyter_code>#AdaBoost Classifier
ada = AdaBoostClassifier(n_estimators=400, learning_rate=0.1)
ada.fit(X_train,y_train)
print ("AdaBoost Accuracy= %2.2f" % accuracy_score(y_test,ada.predict(X_test)))
score_ada = cross_val_score(ada, X, y, cv=10).mean()
print("Cross Validation Score = %2.2f" % score_ada)
y_pred = ada.predict(X_test)
print(classification_report(y_test,y_pred ))
#Confusion Marix for AdaBoost
cm = confusion_matrix(y_test,y_pred)
plot_confusion_matrix(cm, classes=class_names, title='Confusion matrix')<jupyter_output>AdaBoost Accuracy= 0.78
Cross Validation Score = 0.78
              precision    recall  f1-score   support

         0.0       0.79      0.99      0.88     10105
         1.0       0.50      0.04      0.07      2834

    accuracy                           0.78     12939
   macro avg       0.65      0.51      0.47     12939
weighted avg       0.72      0.78      0.70     12939<jupyter_text>XGBOOST<jupyter_code>from sklearn.metrics import accuracy_score
from xgboost import XGBClassifier

#XGBoost Classifier
xgb = GradientBoostingClassifier(n_estimators=1000,learning_rate=0.01)
xgb.fit(X_train , y_train)
print ("GradientBoost Accuracy= %2.2f" % accuracy_score(y_test,xgb.predict(X_test)))
score_xgb = cross_val_score(xgb, X, y, cv=10).mean()
print("Cross Validation Score = %2.2f" % score_xgb)
y_pred = xgb.predict(X_test) 
print(classification_report(y_test,y_pred))
#Confusion Matrix for XGBoost Classifier
cm_xg = confusion_matrix(y_test,y_pred)
plot_confusion_matrix(cm_xg, classes=class_names, title='Confusion matrix')<jupyter_output>GradientBoost Accuracy= 0.78
Cross Validation Score = 0.78
              precision    recall  f1-score   support

         0.0       0.79      0.98      0.88     10105
         1.0       0.55      0.09      0.15      2834

    accuracy                           0.78     12939
   macro avg       0.67      0.53      0.51     12939
weighted avg       0.74      0.78      0.72     12939<jupyter_text>LIGHTGBM<jupyter_code># pip install lightgbm
#lightgbm Classifier
from lightgbm import LGBMClassifier
# parameters={'n_estimators':[900,1000,1100],
#             'learning_rate':[0.01],
#             'max_depth':[8],
#             'min_child_weight':[1],
#             'subsample':[0.8],
#             'colsample_bytree':[0.3,0.4,0.5]}
lgb = LGBMClassifier(n_estimators=1100, learning_rate=0.01, max_depth=4, min_child_weight=1, subsample=0.8, \
                     colsample_bytree=0.5, num_leaves=100)

lgb.fit(X_train,y_train)
print ("LightGBM Accuracy= %2.2f" % accuracy_score(y_test,lgb.predict(X_test)))
score_lgb = cross_val_score(lgb, X, y, cv=10).mean()
print("Cross Validation Score = %2.2f" % score_lgb)
y_pred = lgb.predict(X_test)
print(classification_report(y_test, y_pred))
#Confusion Marix for lightgbm
cm = confusion_matrix(y_test,y_pred)
plot_confusion_matrix(cm, classes=class_names, title='Confusion matrix')<jupyter_output>LightGBM Accuracy= 0.78
Cross Validation Score = 0.78
              precision    recall  f1-score   support

         0.0       0.80      0.97      0.88     10105
         1.0       0.53      0.12      0.19      2834

    accuracy                           0.78     12939
   macro avg       0.66      0.54      0.53     12939
weighted avg       0.74      0.78      0.73     12939<jupyter_text>ROC Curves> Applying Reciever Operating Characteristic(ROC) Curves to plot the true positive rate and the false postive rates so that we can determine which model works best.<jupyter_code>#Obtaining False Positive Rate, True Positive Rate and Threshold for all classifiers
fpr, tpr, thresholds = roc_curve(y_test, knn.predict_proba(X_test)[:,1])
LR_fpr, LR_tpr, thresholds = roc_curve(y_test, LR.predict_proba(X_test)[:,1])
# SVM_fpr, SVM_tpr, thresholds = roc_curve(y_test, SVM.predict_proba(X_test)[:,1])
DT_fpr, DT_tpr, thresholds = roc_curve(y_test, DT.predict_proba(X_test)[:,1])
# rfc_fpr, rfc_tpr, thresholds = roc_curve(y_test, rfc.predict_proba(X_test)[:,1])
ada_fpr, ada_tpr, thresholds = roc_curve(y_test, ada.predict_proba(X_test)[:,1])
xgb_fpr, xgb_tpr, thresholds = roc_curve(y_test, xgb.predict_proba(X_test)[:,1])
lgb_fpr, lgb_tpr, thresholds = roc_curve(y_test, lgb.predict_proba(X_test)[:,1])
#PLotting ROC Curves for all classifiers
plt.figure(figsize=(12,6))
plt.plot(fpr, tpr, label='KNN' )
plt.plot(LR_fpr, LR_tpr, label='Logistic Regression')
# plt.plot(SVM_fpr, SVM_tpr, label='SVM')
plt.plot(DT_fpr, DT_tpr, label='Decision Tree')
# plt.plot(rfc_fpr, rfc_tpr, label='Random Forest')
plt.plot(ada_fpr, ada_tpr, label='AdaBoost')
plt.plot(xgb_fpr, xgb_tpr, label='GradientBoosting')
plt.plot(lgb_fpr, lgb_tpr, label='LightGBM')
# Plot Base Rate ROC

plt.plot([0,1],[0,1],label='Base Rate')


plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.0])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC Graph')
plt.legend(loc="lower right")
plt.show()<jupyter_output><empty_output><jupyter_text>AdaBoost, XGBOOST, LightGBM are doing their part well with good accuracy scores, less true negatives.  Model Evaluation:  Model Performance Comparison<jupyter_code>clfs= [knn, LR, DT, ada, xgb, lgb]
index =['K-Nearest Neighbors', 'Logistic Regression','Decision Trees',
        'AdaBoost','XGBOOST','LIGHTGBM']
scores=[]

for clf in clfs:
    score = np.mean(cross_val_score(clf,X_test,y_test,cv=5,scoring = 'accuracy'))
    scores = np.append(scores,score)
models = pd.Series(scores,index=index)
models.sort_values(ascending=False)<jupyter_output><empty_output><jupyter_text>Ensemble Voting<jupyter_code>from sklearn.ensemble import VotingClassifier 
from sklearn.linear_model import LogisticRegression 
# from sklearn.svm import SVC 
from sklearn.tree import DecisionTreeClassifier 
from sklearn.datasets import load_iris 
from sklearn.metrics import accuracy_score 
from sklearn.model_selection import train_test_split 
clf_vc = VotingClassifier(estimators=[('AdaBoost', ada),                                       
                                      ('LightGBM', lgb),
                                      ('XGBOOST', xgb),
                                      ('Logistic Regression', LR)],
                          voting='hard',
                          weights=[0.4, 0.3, 0.1, 0.2])

clf_vc = clf_vc.fit(X_train, y_train)
from sklearn.metrics import accuracy_score
print('Final Model Accuracy: %.5f'%(accuracy_score(y_test, clf_vc.predict(X_test))))<jupyter_output>Final Model Accuracy: 0.78221
{'fraction_non_alphanumeric': 0.10972830215983503, 'fraction_numerical': 0.02948518505119207, 'mean_word_length': 4.392508778774873, 'pattern_counts': {'":': 7, '<': 85, '<?xml version=': 0, '>': 98, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 16, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '399071', 'license': 'MIT', 'n_tokens_mistral': 9746, 'n_tokens_neox': 9181, 'n_words': 2287, 'path': 'Final_Model.ipynb', 'repo_name': 'YingluDeng/6sense_model_predict', 'size': '27641'}