<jupyter_start><jupyter_text>Final Project SubmissionPlease fill out:* Student name: <NAME> * Student pace:  full time* Scheduled project review date/time: Tuesday April 28th 3:00PM EST* Instructor name: Abhineet * Blog post URL:  Dataset The dataset is comprised of customer data collected from a syrian based telecom company and their telecom users in the united states.  state - US State customer account, listed in state abbreviation account length 	- Number of months account active area code 	- area code of the userphone number 	- phone number of the user international plan 	- yes/no value for if they pay for international calling plan voice mail plan 	- yes/no value if they pay for voicemail plan number vmail messages 	- Number of voicemails that are active on plan total day minutes 	- number of minutes used per day total day calls 	- number of calls used per day total day charge 	- total charge for used minutes total eve calls 	- number of minutes used during evening billing hours total eve charge 	- total charge of evening time calls total night minutes - total number of night minutes used during night billing hours total night calls 	- total number of calls placed during night billing hours total night charge 	- total charge of night billing hours total intl minutes 	- total initial minutes on the account total intl calls 	- total calls on the account total intl charge 	- total initial charge on the account customer service calls 	- number of times the customer has contacted customer service churn - Boolean value: True/False of customer churn  Loading Libraries<jupyter_code># Your code here - remember to use markdown cells for comments as well!
import pandas as pd 
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

%matplotlib inline 
import warnings
plt.style.use('seaborn')
 
    
data = pd.read_csv('data.csv')
df = pd.DataFrame(data)
df.head(25)<jupyter_output><empty_output><jupyter_text>Looking at the above data we notice a number of columns that really have no measurable catagories. We also notice that our most important variable and the value we are looking to model 'churn', is a binary value. This means through our data cleaning process we will try to isolate and find values that effect this value the most.   Data Cleaning<jupyter_code>df.corr()<jupyter_output><empty_output><jupyter_text>A cursory examination of the table above shows us that there is some correlation between values and churn, however the correlations are not very strong.<jupyter_code># This code is inserted to generate the number of churn values that exist in the dataset. 
pd.value_counts(df['churn'])

# There are a number of object vlaues that are not important to our data set - 
#we can clean these up in the data cleaning process
df.info()
# Generate the shape of our data 
rows,cols = df.shape
print("Number of rows: ", rows)
print("Number of columns: ", cols)
# Quickly checking for null or missing values 
df.isnull().sum()
# Getting a quick look at all the data vs all other data points to see visually 
#if there is any correlation not apparent in the .corr table
stats['range'] = np.subtract(stats['max'], stats['min'], dtype=np.float32)
pd.plotting.scatter_matrix(df, figsize=(10,10));<jupyter_output><empty_output><jupyter_text>As seen from the plots above - visually there does not appear to be any stronger correlation linear or otherwise than what was observed in the correlation table.<jupyter_code># Creating a boxplot to quickly visualize the data sets to see where the values lie within the dataset. 
plt.figure(figsize=(20, 10))
plt.boxplot([df[col] for col in df.columns])
plt.title("Box plot of all columns in dataset")
plt.xticks(range(len(df.columns.values)), df.columns.values)
plt.show()<jupyter_output><empty_output><jupyter_text>This plot confirms our value count line that our data set values are all over the place.   Stripplots<jupyter_code>#Looking at customer churn versus customer service calls as they have the 
#highest correlation. 
sns.stripplot(x='churn', y='customer service calls',
              data=df, dodge=True)

sns.stripplot(x='churn', y='account length', data=df)
sns.stripplot(x='churn', y='total day charge', data=df)
sns.stripplot(x='churn', y='total day minutes', data=df)
sns.stripplot(x='churn', y='total intl charge', data=df)
df.dtypes
# creating a working copy of our dataset in order to preserve the original dataset and create a cleaned version.
df_cleaned = df 
df_cleaned.drop(['area code'], axis=1, inplace=True)
df_cleaned.drop(['total eve calls'], axis=1, inplace=True)
df_cleaned.drop(['total night calls'], axis=1, inplace=True)
df_cleaned.drop(['international plan'], axis=1, inplace=True)
df_cleaned.drop(['voice mail plan'], axis=1, inplace=True)
df_cleaned.drop(['state'], axis=1, inplace=True)
df_cleaned.drop(['phone number'], axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>The above tables were dropped as they either had user account data info like location, phone number, etc. and were not related to the actual data needed to model and predict future churn within the data.<jupyter_code>df_cleaned.head(10)<jupyter_output><empty_output><jupyter_text>The above table is much cleaner - leaving only our one boolean column for churn along with integers that can be modeled accurately.<jupyter_code># Creating a correlation heatmap to verify our hypothesis and identify variables. 
correlation = abs(round(df_cleaned.corr(), 2))
f, ax = plt.subplots(figsize=(15, 15))
sns.heatmap(correlation, annot = True, square=True, cmap =sns.diverging_palette(220, 10, as_cmap=True))
plt.title('Correlation Heatmap',fontweight='bold',fontsize=15)
plt.show();
collinearity = correlation > 0.20
collinearity<jupyter_output><empty_output><jupyter_text>Modeling<jupyter_code>from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn import svm
from sklearn.model_selection import train_test_split, GridSearchCV, cross_val_score

X = df_cleaned[[col for col in df_cleaned.columns if col != 'churn']]
y = df_cleaned['churn']
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=22)
train = pd.concat([X_train, y_train], axis=1)
aggs = train.groupby('churn').agg(['mean', 'std'])
aggs
row = 0
df_cleaned.iloc[row]<jupyter_output><empty_output><jupyter_text>Importing additional libraries<jupyter_code>#Import additional libraries
import xgboost as xgb
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import accuracy_score
# from sklearn.grid_search import GridSearchCV
import warnings
from sklearn.model_selection import GridSearchCV
warnings.filterwarnings('ignore')
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Model choices: The following models were chosen for this project for a few reasons. In computational complexity and optimization the no free lunch theorem is a result that states that for certain types of mathematical problems, the computational cost of finding a solution, averaged over all problems in the class, is the same for any solution method. No solution therefore offers a "short cut". In computing, there are circumstances in which the outputs of all procedures solving a particular type of problem are statistically identical.Thus multiple models are chosen to compare against eachother in order to dertermine the most effective or simmilarly effective models.Models:     K Nearest Neighbors,     Random Forest,    XGBoost  XGBoost<jupyter_code>#Split data into tests and sets
labels = df_cleaned['churn']
labels_removed_df = df_cleaned.drop('churn', axis=1, inplace=False)
scaler = StandardScaler()
scaled_df = scaler.fit_transform(labels_removed_df)
X_train, X_test, y_train, y_test = train_test_split(scaled_df, labels)

clf = xgb.XGBClassifier()
clf.fit(X_train, y_train)
training_preds = clf.predict(X_train)
val_preds = clf.predict(X_test)
training_accuracy = accuracy_score(y_train, training_preds)
val_accuracy = accuracy_score(y_test, val_preds)

print("Training Score: {:.5}%".format(training_accuracy * 100))
print("Test Score: {:.5}%".format(val_accuracy * 100))
param_grid = {
    "learning_rate": [0.1],
    'max_depth': [6],
    'min_child_weight': [10],
    'subsample': [ 0.7],
    'n_estimators': [5, 30, 100, 250],
}



grid_clf = GridSearchCV(clf, param_grid, scoring='accuracy', cv=None, n_jobs=1)
grid_clf.fit(scaled_df, labels)

best_parameters = grid_clf.best_params_

print("Using paramater grid, the model was optimized with the following optimal parameters: ")
for param_name in sorted(best_parameters.keys()):
    print("%s: %r" % (param_name, best_parameters[param_name]))

training_preds = grid_clf.predict(X_train)
val_preds = grid_clf.predict(X_test)
training_accuracy = accuracy_score(y_train, training_preds)
val_accuracy = accuracy_score(y_test, val_preds)

print("")
print("Training Score: {:.4}%".format(training_accuracy * 100))
print("Test Score: {:.4}%".format(val_accuracy * 100))<jupyter_output>Grid Search found the following optimal parameters: 
learning_rate: 0.1
max_depth: 6
min_child_weight: 10
n_estimators: 100
subsample: 0.7

Training Score: 94.04%
Test Score: 94.84%<jupyter_text>KNN<jupyter_code>from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn import svm
from sklearn.model_selection import train_test_split, GridSearchCV, cross_val_score

X_train, X_test, y_train, y_test = train_test_split(df_cleaned, labels, test_size=0.25, random_state=42)
scaler = StandardScaler()
scaled_data_train = scaler.fit_transform(X_train)
scaled_data_test = scaler.transform(X_test)
scaled_df_train = pd.DataFrame(scaled_data_train, columns=df_cleaned.columns)
scaled_df_train.head(25)


# deploy KNeighborsClassifier
clf = KNeighborsClassifier()
clf.fit(scaled_data_train, y_train)
test_preds = clf.predict(scaled_data_test)
# Import metric scoring 
from sklearn.metrics import precision_score, recall_score, accuracy_score, f1_score
def print_metrics(labels, preds):
    print("Training Score: {}".format(accuracy_score(labels, preds)))
    print("Test Score: {}".format(f1_score(labels, preds)))
    
print_metrics(y_test, test_preds)<jupyter_output>Training Score: 0.9976019184652278
Test Score: 0.9919354838709677<jupyter_text>Grid Search<jupyter_code>from sklearn.model_selection import train_test_split, GridSearchCV, cross_val_score
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier
from sklearn.metrics import accuracy_score
y = df_cleaned['churn']
X = df_cleaned.drop('churn', axis=1)
X.head()
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
dt_param_grid = {
    'criterion': ['gini', 'entropy'],
    'max_depth': [None, 2, 3, 4, 5, 6],
    'min_samples_split': [2, 5, 10],
    'min_samples_leaf': [1, 2, 3, 4, 5, 6]
}

trees = 3 * 2 * 6 * 3 * 6
print(f"Grid Search will create {trees} different permutations.")
dt_grid_search = GridSearchCV(dt_clf, dt_param_grid, cv=3, return_train_score=True)

# Fit to the data
dt_grid_search.fit(X_train, y_train)
dt_gs_training_score = np.mean(dt_grid_search.cv_results_['mean_train_score'])

# Mean test score
dt_gs_testing_score = dt_grid_search.score(X_test, y_test)

print(f"Training Score: {dt_gs_training_score :.3%}")
print(f" Test Score: {dt_gs_testing_score :.3%}")<jupyter_output>Training Score: 92.178%
 Test Score: 91.847%
{'fraction_non_alphanumeric': 0.08806361559935708, 'fraction_numerical': 0.015142542932070045, 'mean_word_length': 6.108839446782922, 'pattern_counts': {'":': 1, '<': 47, '<?xml version=': 0, '>': 48, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '629983', 'license': 'RSA-MD', 'n_tokens_mistral': 3855, 'n_tokens_neox': 3655, 'n_words': 1240, 'path': 'student.ipynb', 'repo_name': 'bkdatl/dsc-mod-3-project-v2-1-onl01-dtsc-ft-012120', 'size': '11831'}