<jupyter_start><jupyter_text>This dataset is aimed at the case of customer default payments in Taiwan. From the perspective of risk management, the result of predictive accuracy of the estimated probability of default will be more valuable than the binary result of classification - credible or not credible clients. Because the real probability of default is unknown, this study presented the novel Sorting Smoothing Method to estimate the real probability of default.<jupyter_code>#required imports
import numpy as np
import pandas as pd
import seaborn as sns<jupyter_output><empty_output><jupyter_text>After installing the necessary libraries, proceed to download the data. Since reading the excel file won't create headers by default, we added two more operations to substitute the columns.<jupyter_code>#loading the data
dataset = pd.read_excel("https://archive.ics.uci.edu/ml/machine-learning-databases/00350/default%20of%20credit%20card%20clients.xls",header=1)
#dataset.columns = dataset.iloc[0]

dataset.head(10)
dataset.info()
dataset.drop(['ID'], axis=1,inplace=True)<jupyter_output><empty_output><jupyter_text>In the following, you can take a look into the dataset.<jupyter_code>dataset.head(10)
dataset.describe()
dataset.info()
# Sort by dtype 
dep_var=['default payment next month']

categorical_vars = ['SEX','EDUCATION','MARRIAGE','PAY_0','PAY_2','PAY_3','PAY_4','PAY_5','PAY_6']

discrete_vars=['LIMIT_BAL','AGE']

continuous_vars =['BILL_AMT1','BILL_AMT2','BILL_AMT3','BILL_AMT4','BILL_AMT5','BILL_AMT6','PAY_AMT1','PAY_AMT2','PAY_AMT3','PAY_AMT4','PAY_AMT5','PAY_AMT6' ]
#check on missing data
dataset.isna()
#Quantitative features
dataset.loc[:, dep_var + discrete_vars + continuous_vars].describe()<jupyter_output><empty_output><jupyter_text>Data Preprocessing Build a classifier by using decision tree and calculate the confusion matrix. Try different hyper-parameters (at least two) and discuss the result.<jupyter_code>import matplotlib.pyplot as plt
from scipy import stats
from sklearn.datasets import load_wine
from sklearn.tree import DecisionTreeClassifier
from IPython.display import SVG
from graphviz import Source
from sklearn import tree
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
#set up the features and target
X = dataset.drop(['default payment next month'],axis=1)
y = dataset['default payment next month']

#split the dataset into train and test dataset
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
tree_clf = DecisionTreeClassifier(max_depth=2, criterion='entropy')
tree_clf.fit(X_train, y_train)
y_pred1=tree_clf.predict(X_test)
#graph = Source(tree.export_graphviz(tree_clf, out_file=None,
        #class_names=y, filled = True))
#display(SVG(graph.pipe(format='svg')))
from sklearn.metrics import plot_confusion_matrix
from sklearn.metrics import confusion_matrix
confusion_matrix(y_test, y_pred1,normalize='true')
print(accuracy_score(y_test, y_pred1))
tree_clf2 = DecisionTreeClassifier(max_depth=5, criterion='gini')
tree_clf2.fit(X_train, y_train)
y_pred2=tree_clf2.predict(X_test)
confusion_matrix(y_test, y_pred2,normalize='true')
print(accuracy_score(y_test, y_pred2))<jupyter_output>0.8194666666666667<jupyter_text>Conclusion - The second set of parameters imporved TP from 0.94 to 0.96. build the decision tree by RandomizedSearchCV over hyper-parameters. Compare the results.<jupyter_code>from scipy.stats import randint
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import RandomizedSearchCV

# Setup the parameters 
param_dist = {"max_depth": [5, None],
              "max_features": randint(1, 9),
              "min_samples_leaf": randint(1, 9),
              "criterion": ["gini", "entropy"]}

# Instantiate a Decision Tree classifier: tree
tree = DecisionTreeClassifier()

# Instantiate the RandomizedSearchCV object: tree_cv
tree_cv = RandomizedSearchCV(tree, param_dist, cv=5)

# Fit it to the data
tree_cv.fit(X_train,y_train)

# Print the tuned parameters and score
print("Tuned Decision Tree Parameters: {}".format(tree_cv.best_params_))
print("Best score is {}".format(tree_cv.best_score_))
y_pred3=tree_cv.predict(X_test)
confusion_matrix(y_test, y_pred3,normalize='true')
print(accuracy_score(y_test, y_pred3))<jupyter_output>0.8028<jupyter_text>Concludion - the result has been improved. The RandomizedSearchCV imporves TP from 0.94 to 0.95. Build the same classifier by using following ensemble models. For each of these models calculate accuracy and at least for two in the list below, plot the learning curves.* Random Forest * AdaBoost* Extra Trees Classifier * Gradient Boosted Trees<jupyter_code>from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import VotingClassifier
from sklearn.ensemble import AdaBoostClassifier
from sklearn.ensemble import ExtraTreesClassifier
from sklearn.ensemble import GradientBoostingClassifier


rnd_clf = RandomForestClassifier()
ada_clf = AdaBoostClassifier(
    #DecisionTreeClassifier(max_depth=1), n_estimators=200,
    #algorithm="SAMME.R", learning_rate=0.5
    )
et_clf = ExtraTreesClassifier()
gb_clf = GradientBoostingClassifier()

voting_clf = VotingClassifier(
    estimators=[('aba', ada_clf), ('rf', rnd_clf), ('et', et_clf), ('gb', gb_clf) ],
    voting='hard')

voting_clf.fit(X_train, y_train)

from sklearn.metrics import accuracy_score
for clf in (ada_clf, rnd_clf, et_clf, gb_clf, voting_clf):
    clf.fit(X_train, y_train)
    y_pred4 = clf.predict(X_test);
    print(clf.__class__.__name__, accuracy_score(y_test, y_pred4));
confusion_matrix(y_test, y_pred4,normalize='true')<jupyter_output><empty_output><jupyter_text>plot the learning curves<jupyter_code>from sklearn.linear_model import LinearRegression
from sklearn.pipeline import Pipeline
#print(__doc__)

import numpy as np
import matplotlib.pyplot as plt
from sklearn.naive_bayes import GaussianNB
#from sklearn.svm import SVC
#from sklearn.datasets import load_digits
from sklearn.model_selection import learning_curve
from sklearn.model_selection import ShuffleSplit


def plot_learning_curve(estimator, title, X, y, axes=None, ylim=None, cv=None,
                        n_jobs=None, train_sizes=np.linspace(.1, 1.0, 5)):
    if axes is None:
        _, axes = plt.subplots(1, 3, figsize=(20, 5))

    axes[0].set_title(title)
    if ylim is not None:
        axes[0].set_ylim(*ylim)
    axes[0].set_xlabel("Training examples")
    axes[0].set_ylabel("Score")

    train_sizes, train_scores, test_scores, fit_times, _ = \
        learning_curve(estimator, X, y, cv=cv, n_jobs=n_jobs,
                       train_sizes=train_sizes,
                       return_times=True)
    train_scores_mean = np.mean(train_scores, axis=1)
    train_scores_std = np.std(train_scores, axis=1)
    test_scores_mean = np.mean(test_scores, axis=1)
    test_scores_std = np.std(test_scores, axis=1)
    fit_times_mean = np.mean(fit_times, axis=1)
    fit_times_std = np.std(fit_times, axis=1)

    # Plot learning curve
    axes[0].grid()
    axes[0].fill_between(train_sizes, train_scores_mean - train_scores_std,
                         train_scores_mean + train_scores_std, alpha=0.1,
                         color="r")
    axes[0].fill_between(train_sizes, test_scores_mean - test_scores_std,
                         test_scores_mean + test_scores_std, alpha=0.1,
                         color="g")
    axes[0].plot(train_sizes, train_scores_mean, 'o-', color="r",
                 label="Training score")
    axes[0].plot(train_sizes, test_scores_mean, 'o-', color="g",
                 label="Cross-validation score")
    axes[0].legend(loc="best")

    # Plot n_samples vs fit_times
    axes[1].grid()
    axes[1].plot(train_sizes, fit_times_mean, 'o-')
    axes[1].fill_between(train_sizes, fit_times_mean - fit_times_std,
                         fit_times_mean + fit_times_std, alpha=0.1)
    axes[1].set_xlabel("Training examples")
    axes[1].set_ylabel("fit_times")
    axes[1].set_title("Scalability of the model")

    # Plot fit_time vs score
    axes[2].grid()
    axes[2].plot(fit_times_mean, test_scores_mean, 'o-')
    axes[2].fill_between(fit_times_mean, test_scores_mean - test_scores_std,
                         test_scores_mean + test_scores_std, alpha=0.1)
    axes[2].set_xlabel("fit_times")
    axes[2].set_ylabel("Score")
    axes[2].set_title("Performance of the model")

    return plt


fig, axes = plt.subplots(3, 2, figsize=(10, 15))

X=X_train 
y=y_train

title = "Learning Curves (Naive Bayes)"
# Cross validation with 100 iterations to get smoother mean test and train
# score curves, each time with 20% data randomly selected as a validation set.
cv = ShuffleSplit(n_splits=100, test_size=0.2, random_state=0)

estimator = GradientBoostingClassifier()
plot_learning_curve(estimator, title, X, y, axes=axes[:, 0], ylim=(0.7, 1.01),
                    cv=cv, n_jobs=4)

title = r"Learning Curves (SVM, RBF kernel, $\gamma=0.001$)"
# SVC is more expensive so we do a lower number of CV iterations:
cv = ShuffleSplit(n_splits=10, test_size=0.2, random_state=0)
estimator = ExtraTreesClassifier()
plot_learning_curve(estimator, title, X, y, axes=axes[:, 1], ylim=(0.7, 1.01),
                    cv=cv, n_jobs=4)

plt.show()<jupyter_output><empty_output><jupyter_text>Discuss and compare the results for the all past three questions. * How does changing hyperparms effect model performance? * Why do you think certain models performed better/worse? * How does this performance line up with known strengths/weakness of these models? By toning the hyperparms, the TP and FN gets improved.<jupyter_code>print(confusion_matrix(y_test, y_pred1,normalize='true'))
print(confusion_matrix(y_test, y_pred2,normalize='true'))
print(confusion_matrix(y_test, y_pred3,normalize='true'))
print(confusion_matrix(y_test, y_pred4,normalize='true'))<jupyter_output><empty_output><jupyter_text>The max depth increased from 2 to 5 from model 1 to model 5. This improved the TP of the model.  The RandomizedSearchCV and Voting on model 3 and model 4 also improved the performance of the model. The TP rate is increased while the FN is decreased. These two models have strength in finding a better result. RandomizedSearchCV toning the parameters and find the best result. The strength of the RandomizedSearchCV will improve the accuracy of the model. However, it could lead to an overfitting problem. Meanwhile, voting ( hard voting ) finds a model that results in the best score among multiple models. However, it could be restricted by the parameters set up in each of the models.  RandomizedSearchCV may result in a low variance but in high bias.  GB has a less trainning time; Randomforest uses more time.  The tradeoff between recall and precision should be considered as well. This should be included in model selections.<jupyter_code>Random Forest 
Pros-Low bias
Cons-more time to run the model
AdaBoost
Pros - Reduce the variance by assign a weight to each of instance. 
Cons - takes a long time to run. It may leads to overfitting problem. 
Extra Trees Classifier
Pros- exterme low bias
Cons- takes a long time to run
Gradient Boosted Trees
Pros- Low variance as it reduce variance
Cons-takes more time to run when the error is significant.<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09473684210526316, 'fraction_numerical': 0.01760283060592658, 'mean_word_length': 5.0719656283566055, 'pattern_counts': {'":': 4, '<': 35, '<?xml version=': 0, '>': 35, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '456726', 'license': 'MIT', 'n_tokens_mistral': 3852, 'n_tokens_neox': 3634, 'n_words': 1029, 'path': 'customer default payments in Taiwan - Decision Trees and Ensemble Models in classification and regression.ipynb', 'repo_name': 'zoe-xu-1234/Data-Science-Profile', 'size': '11305'}