<jupyter_start><jupyter_text>Machine Learning with Python - Assigment> Author : <NAME>>> Mar 11, 2020 In this notebook we try to practice all the classification algorithms that we learned in this course.We load a dataset using Pandas library, and apply the following algorithms, and find the best one for this specific dataset by accuracy evaluation methods.Lets first load required libraries:<jupyter_code>import itertools
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.ticker import NullFormatter
import pandas as pd
import numpy as np
import matplotlib.ticker as ticker
from sklearn import preprocessing
%matplotlib inline
# setting the random seed for similar results on each run
np.random.seed(7)<jupyter_output><empty_output><jupyter_text>About dataset This dataset is about past loans. The __Loan_train.csv__ data set includes details of 346 customers whose loan are already paid off or defaulted. It includes following fields:| Field          | Description                                                                           ||----------------|---------------------------------------------------------------------------------------|| Loan_status    | Whether a loan is paid off on in collection                                           || Principal      | Basic principal loan amount at the                                                    || Terms          | Origination terms which can be weekly (7 days), biweekly, and monthly payoff schedule || Effective_date | When the loan got originated and took effects                                         || Due_date       | Since it’s one-time payoff schedule, each loan has one single due date                || Age            | Age of applicant                                                                      || Education      | Education of applicant                                                                || Gender         | The gender of applicant                                                               | Lets download the dataset<jupyter_code>!wget -O loan_train.csv https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/loan_train.csv<jupyter_output>--2020-03-11 14:54:40--  https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/loan_train.csv
Resolving s3-api.us-geo.objectstorage.softlayer.net (s3-api.us-geo.objectstorage.softlayer.net)... 192.168.127.12
Connecting to s3-api.us-geo.objectstorage.softlayer.net (s3-api.us-geo.objectstorage.softlayer.net)|192.168.127.12|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 23101 (23K) [text/csv]
Saving to: ‘loan_train.csv’

100%[======================================>] 23,101      --.-K/s   in 0.002s  

2020-03-11 14:54:40 (11.2 MB/s) - ‘loan_train.csv’ saved [23101/23101]<jupyter_text>Load Data From CSV File<jupyter_code>df = pd.read_csv('loan_train.csv')
df.head()
df.shape<jupyter_output><empty_output><jupyter_text>Convert to date time object<jupyter_code>df['due_date'] = pd.to_datetime(df['due_date'])
df['effective_date'] = pd.to_datetime(df['effective_date'])
df.head()<jupyter_output><empty_output><jupyter_text>Data visualization and pre-processing Let’s see how many of each class is in our data set<jupyter_code>df['loan_status'].value_counts()<jupyter_output><empty_output><jupyter_text>260 people have paid off the loan on time while 86 have gone into collection  Lets plot some columns to underestand data better:<jupyter_code># notice: installing seaborn might takes a few minutes
# !conda install -c anaconda seaborn -y
import seaborn as sns

bins = np.linspace(df.Principal.min(), df.Principal.max(), 10)
g = sns.FacetGrid(df, col="Gender", hue="loan_status", palette="Set1", col_wrap=2)
g.map(plt.hist, 'Principal', bins=bins, ec="k")

g.axes[-1].legend()
plt.show()
bins = np.linspace(df.age.min(), df.age.max(), 10)
g = sns.FacetGrid(df, col="Gender", hue="loan_status", palette="Set1", col_wrap=2)
g.map(plt.hist, 'age', bins=bins, ec="k")

g.axes[-1].legend()
plt.show()<jupyter_output><empty_output><jupyter_text>Pre-processing:  Feature selection/extraction  Lets look at the day of the week people get the loan<jupyter_code>df['dayofweek'] = df['effective_date'].dt.dayofweek
bins = np.linspace(df.dayofweek.min(), df.dayofweek.max(), 10)
g = sns.FacetGrid(df, col="Gender", hue="loan_status", palette="Set1", col_wrap=2)
g.map(plt.hist, 'dayofweek', bins=bins, ec="k")
g.axes[-1].legend()
plt.show()<jupyter_output><empty_output><jupyter_text>Lets look at the Months people get the loan<jupyter_code>df['Month'] = df.due_date.dt.month
bins = np.linspace(df.Month.min()-1, df.Month.max()+1, 10)
g = sns.FacetGrid(df, col="Gender", hue="loan_status", palette="Set1", col_wrap=2)
g.map(plt.hist, 'Month', bins=bins, ec="k")
g.axes[-1].legend()
plt.show()<jupyter_output><empty_output><jupyter_text>there is higher possibility that people pay off the loan in 9th month than paying it in 10th or 11th  **as the observation shows people who get the loan at the end of the week don't pay it off, so lets use Feature binarization to set a threshold values less then day 4**<jupyter_code>df['weekend'] = df['dayofweek'].apply(lambda x: 1 if (x>3)  else 0)
df.head()<jupyter_output><empty_output><jupyter_text>Convert Categorical features to numerical values Lets look at gender:<jupyter_code>df.groupby(['Gender'])['loan_status'].value_counts(normalize=True)<jupyter_output><empty_output><jupyter_text>86 % of female pay there loans while only 73 % of males pay there loan Lets convert male to 0 and female to 1:<jupyter_code>df['Gender'].replace(to_replace=['male','female'], value=[0,1],inplace=True)
df.head()
df['deadline']=df['due_date']-df['effective_date']

df.head()
df['deadline']=df['deadline'].dt.days
df.head(3)<jupyter_output><empty_output><jupyter_text>*based on the tested rusults columns Month and deadline doesn't improve the result so we will not use them in feature sets of training  One Hot Encoding   How about education?<jupyter_code>df.groupby(['education'])['loan_status'].value_counts(normalize=True)<jupyter_output><empty_output><jupyter_text>**There are only two people with Master or Above degress,one of them has paid off the loan and the other one hasn't.As a result keeping 'Master or Above' in our data set wont't help much! so we drop it.**<jupyter_code>len(df[df.education=='Master or Above'])<jupyter_output><empty_output><jupyter_text>We use one hot encoding technique to convert categorical variables to binary variables and append them to the feature Data Frame<jupyter_code>Feature = df[['Principal','terms','age','Gender','weekend','dayofweek']]
Feature = pd.concat([Feature,pd.get_dummies(df['education'])], axis=1)
Feature.drop(['Master or Above'], axis = 1,inplace=True)
Feature.head()<jupyter_output><empty_output><jupyter_text>Feature selection  Our data without label<jupyter_code>X = Feature
X[0:5]<jupyter_output><empty_output><jupyter_text>Our labels<jupyter_code># We encode our labels as 0 and 1 s
y = df['loan_status'].replace(to_replace=['PAIDOFF','COLLECTION'], value=[0,1]).values
y[0:5]<jupyter_output><empty_output><jupyter_text>Normalize Data  Data Standardization give data zero mean and unit variance<jupyter_code>X= preprocessing.StandardScaler().fit_transform(X)
X[0:5]
# import scoring methods
from sklearn.metrics import jaccard_similarity_score
from sklearn.metrics import f1_score
from sklearn.metrics import log_loss
from sklearn.model_selection import cross_val_score
# our data for training (used '_train' just for improving readability)
X_train, y_train=X,y
X_train.shape, y_train.shape
# a dictionary for keeping all scores of the classifiers
trainScores={}<jupyter_output><empty_output><jupyter_text>K Nearest Neighbor(KNN)<jupyter_code>from sklearn.neighbors import KNeighborsClassifier

bestScore=0.0
accList=[]

for k in range(3,12):
    
    clf_knn = KNeighborsClassifier(n_neighbors=k,algorithm='auto')
    
    # using 10 fold cross validation for scoring the classifier's accuracy
    scores = cross_val_score(clf_knn, X, y, cv=10)
    score=scores.mean()
    accList.append(score)
    
    if score > bestScore:
        bestScore=score
        best_clf=clf_knn
        bestK=k
        
print("Best K is :",bestK,"| Cross validation Accuracy :",bestScore)
clf_knn=best_clf
clf_knn.fit(X_train,y_train)
y_pred=best_clf.predict(X_train)
trainScores['KNN-jaccard']=jaccard_similarity_score(y_train, y_pred)
trainScores['KNN-f1-score']=f1_score(y_train, y_pred, average='weighted')  
trainScores
plt.plot(range(3,12),accList)
plt.xlabel('K')
plt.ylabel('CV Accuracy')
plt.show()<jupyter_output><empty_output><jupyter_text>Decision Tree<jupyter_code>from sklearn import tree

clf_tree = tree.DecisionTreeClassifier()
clf_tree = clf_tree.fit(X_train, y_train)

y_pred=clf_tree.predict(X_train)
trainScores['Tree-jaccard']=jaccard_similarity_score(y_train, y_pred)
trainScores['Tree-f1-score']=f1_score(y_train, y_pred, average='weighted')  
trainScores<jupyter_output><empty_output><jupyter_text>Visualizng the tree<jupyter_code>!pip install graphviz
!pip install pydotplus
import graphviz 
import pydotplus

dot_data = tree.export_graphviz(clf_tree, out_file=None, 
                     feature_names=['Principal',
                                    'terms','age',
                                    'Gender',
                                    'weekend',
                                    'Bechalor',
                                    'High School or Below',
                                    'college',
                                    'dayofweek',
                                     #'deadline'
#                                     ,'Month'
                                   ],  
                     class_names='loan_status',  
                     filled=True, rounded=True,  
                     special_characters=True) 

graph = pydotplus.graph_from_dot_data(dot_data)
graph.set_size('"8,8!"')
gvz_graph = graphviz.Source(graph.to_string())

gvz_graph<jupyter_output>Collecting graphviz
  Downloading https://files.pythonhosted.org/packages/f5/74/dbed754c0abd63768d3a7a7b472da35b08ac442cf87d73d5850a6f32391e/graphviz-0.13.2-py2.py3-none-any.whl
Installing collected packages: graphviz
Successfully installed graphviz-0.13.2
Collecting pydotplus
[?25l  Downloading https://files.pythonhosted.org/packages/60/bf/62567830b700d9f6930e9ab6831d6ba256f7b0b730acb37278b0ccdffacf/pydotplus-2.0.2.tar.gz (278kB)
[K     |████████████████████████████████| 286kB 14.5MB/s eta 0:00:01
[?25hRequirement already satisfied: pyparsing>=2.0.1 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from pydotplus) (2.3.1)
Building wheels for collected packages: pydotplus
  Building wheel for pydotplus (setup.py) ... [?25ldone
[?25h  Stored in directory: /home/dsxuser/.cache/pip/wheels/35/7b/ab/66fb7b2ac1f6df87475b09dc48e707b6e0de80a6d8444e3628
Successfully built pydotplus
Installing collected packages: pydotplus
Successfully installed pydotplus-2.0.2<jupyter_text>Support Vector Machine<jupyter_code>y_train=y_train.astype(float)
from sklearn import svm

clf_svm = svm.LinearSVC(random_state=7)
clf_svm.fit(X_train, y_train)  

y_pred=clf_svm.predict(X_train)
trainScores['SVM-jaccard']=jaccard_similarity_score(y_train, y_pred)
trainScores['SVM-f1-score']=f1_score(y_train, y_pred, average='weighted')  
trainScores<jupyter_output><empty_output><jupyter_text>Logistic Regression<jupyter_code>from sklearn.linear_model import LogisticRegression

clf_log = LogisticRegression(random_state=0, solver='lbfgs',
                         multi_class='multinomial')
clf_log.fit(X_train, y_train)

y_pred=clf_log.predict(X_train)
y_proba=clf_log.predict_proba(X_train)

#testScores['LogReg-logLoss']=log_loss(testy, proba)

trainScores['LogReg-jaccard']=jaccard_similarity_score(y_train, y_pred)
trainScores['LogReg-f1-score']=f1_score(y_train, y_pred, average='weighted')  
trainScores['LogReg-logLoss']=log_loss(y_train, y_proba)
trainScores<jupyter_output><empty_output><jupyter_text>Model Evaluation using Test set<jupyter_code>!wget -O loan_test.csv https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/loan_test.csv<jupyter_output>--2020-03-11 14:58:17--  https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/loan_test.csv
Resolving s3-api.us-geo.objectstorage.softlayer.net (s3-api.us-geo.objectstorage.softlayer.net)... 192.168.127.12
Connecting to s3-api.us-geo.objectstorage.softlayer.net (s3-api.us-geo.objectstorage.softlayer.net)|192.168.127.12|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3642 (3.6K) [text/csv]
Saving to: ‘loan_test.csv’

100%[======================================>] 3,642       --.-K/s   in 0s      

2020-03-11 14:58:18 (358 MB/s) - ‘loan_test.csv’ saved [3642/3642]<jupyter_text>Load Test set for evaluation<jupyter_code>test_df = pd.read_csv('loan_test.csv')
test_df.head()
test_df['due_date'] = pd.to_datetime(test_df['due_date'])
test_df['effective_date'] = pd.to_datetime(test_df['effective_date'])

test_df['dayofweek'] = test_df['effective_date'].dt.dayofweek

test_df['weekend'] = test_df['dayofweek'].apply(lambda x: 1 if (x>3)  else 0)

test_df['Gender'].replace(to_replace=['male','female'], value=[0,1],inplace=True)

# test_df['Month'] = test_df.due_date.dt.month 

# test_df['deadline']=test_df['due_date']-test_df['effective_date']
# test_df['deadline']=test_df['deadline'].dt.days

Feature = test_df[['Principal','terms','age','Gender','weekend','dayofweek']]
Feature = pd.concat([Feature,pd.get_dummies(test_df['education'])], axis=1)
Feature.drop(['Master or Above'], axis = 1,inplace=True)

X = Feature

y = test_df['loan_status'].replace(to_replace=['PAIDOFF','COLLECTION'], value=[0,1]).values

testy=y.astype(float)
testX= preprocessing.StandardScaler().fit_transform(X)
testScores={}
knn_pred=clf_knn.predict(testX)
testScores['KNN-jaccard']=jaccard_similarity_score(testy, knn_pred)
testScores['KNN-f1-score']=f1_score(testy, knn_pred, average='weighted')  
tree_pred=clf_tree.predict(testX)
testScores['Tree-jaccard']=jaccard_similarity_score(testy, tree_pred)
testScores['Tree-f1-score']=f1_score(testy, tree_pred, average='weighted')  
svm_pred=clf_svm.predict(testX)
testScores['SVM-jaccard']=jaccard_similarity_score(testy, svm_pred)
testScores['SVM-f1-score']=f1_score(testy, svm_pred, average='weighted')  
log_pred=clf_log.predict(testX)
proba=clf_log.predict_proba(testX)
testScores['LogReg-jaccard']=jaccard_similarity_score(testy, log_pred)
testScores['LogReg-f1-score']=f1_score(testy, log_pred, average='weighted')  
testScores['LogReg-logLoss']=log_loss(testy, proba)
trainScores
testScores
# Features used
list(Feature.columns)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13641267700606877, 'fraction_numerical': 0.03169251517194875, 'mean_word_length': 4.414749908725812, 'pattern_counts': {'":': 0, '<': 95, '<?xml version=': 0, '>': 104, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 4, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 29, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '426152', 'license': 'MIT', 'n_tokens_mistral': 5529, 'n_tokens_neox': 5047, 'n_words': 995, 'path': 'IBM - Data Science Professional Certificate/Machine Learning with Python-IBM/Final Project - Loan Data.ipynb', 'repo_name': 'ngngocsonan2610/coursera', 'size': '14842'}