<jupyter_start><jupyter_text>__Assignment 5__1. [Import](Import)1. [Assignment 5](Assignment-5)    1. [Train decision tree](Train-decision-tree)    1. [Improve the model](Improve-the-model)    1. [Review results](Review-results)  Import<jupyter_code>import pandas as pd
import numpy as np
from sklearn import tree

import warnings

warnings.simplefilter("ignore")
np.set_printoptions(threshold=np.inf, suppress=True)
from IPython.core.display import display, HTML

display(HTML("<style>.container { width:88% !important; }</style>"))<jupyter_output><empty_output><jupyter_text>Assignment 5Assignment Content:Launch the “DecisionTreesAssignment.ipynb” file in Jupyter Notebook and complete the assignment. Don’t forget to include tae.data in your directory or change the code to read this input file. Namely, build a decision tree classifier and test the classifier on previous and future data. Write a one page report explaining the following:- How the decision tree classifier works- Thoroughly explain the code (i.e. your implementation of the decision tree classifier)- Try to get as high accuracy as possible on the future data.- Is it possible to have both high training/previous and test/future accuracy? Why or why not?- Play around with the sklearn DecisionTreeClassifier library, tweaking the various decision tree parameters and report on the results of your findingsDeliverables:- Completed DecisionTreesAssignment.ipynb code which compiles, runs, and gives fairly good training and test accuracy (they don’t need to be 100%, but justify your answer in the report)- Report (as docx, pdf, or any other popular text format)<jupyter_code># Dataset used: https://archive.ics.uci.edu/ml/datasets/Teaching+Assistant+Evaluation

ta_eval_raw = pd.read_csv("s3://tdp-ml-datasets/misc/tae.data")

# Used for sampling (previous data) and (future data)
total_sample_num = ta_eval_raw.shape[0]

# in order to simulate what would happen in real life scenarios, we choose
# 50% of this dataset at random to be our previous (training) data

previous_data_to_train_on = ta_eval_raw.sample(
    int(0.5 * total_sample_num), replace=True
)

# review size of dataset
print("Full dataset dimensions: {0}".format(ta_eval_raw.shape))
# convert data into numpy arrays
X = previous_data_to_train_on.iloc[:, :-1].values
y = previous_data_to_train_on.iloc[:, -1].values.reshape(-1, 1)

print("X dimensions: {0}".format(X.shape))
print("y dimensions: {0}".format(y.shape))
# TODO: Build your decision tree model here (keep the name of your decision tree consistent throughout the program)
# Hint: Read the sklearn DecisionTreeClassifier docs:
# http://scikit-learn.org/stable/modules/generated/sklearn.DecisionTreeClassifier.html
# Hint: You may also find it useful to visualize the tree in Graphviz as we've done in the lecture
decision_tree = DecisionTreeClassifier()
decision_tree.fit(X, y)<jupyter_output><empty_output><jupyter_text>Train decision tree<jupyter_code># in order to simulate what would happen in real life scenarios, we choose
# 50% of this dataset at random to be our future (test) data
future_data_we_will_test_on = ta_eval_raw.sample(
    int(0.5 * total_sample_num), replace=True
)

# true values so that we can compare how good our classifier that we built performs
true_value_training = list(previous_data_to_train_on.iloc[:, 5])
true_value_future = list(future_data_we_will_test_on.iloc[:, 5])

# using Decision Tree to predict the labels i.e. 1, 2, or 3 for training and test data
prediction_on_trainingdata = decision_tree.predict(
    previous_data_to_train_on.iloc[:, :-1]
)
prediction_on_futuredata = decision_tree.predict(
    future_data_we_will_test_on.iloc[:, :-1]
)

print(
    "Data We Trained On Prediction Accuracy: {0}%".format(
        sum(true_value_training == prediction_on_trainingdata)
        / len(prediction_on_trainingdata)
    )
)
print(
    "Future Data Prediction Accuracy: {0}%".format(
        sum(true_value_future == prediction_on_futuredata)
        / len(prediction_on_futuredata)
    )
)<jupyter_output>Data We Trained On Prediction Accuracy: 0.9733333333333334%
Future Data Prediction Accuracy: 0.6533333333333333%<jupyter_text>Improve the model<jupyter_code># use sklearn's train_test_split to create train/test sets
# create a test sample that is 50% of the sample just like the workflow above
from sklearn.model_selection import train_test_split

X = ta_eval_raw.iloc[:, :-1].values
y = ta_eval_raw.iloc[:, -1].values

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1)

print("X_train dimensions: {0}".format(X_train.shape))
print("X_test dimensions: {0}".format(X_test.shape))
print("y_train dimensions: {0}".format(y_train.shape))
print("y_test dimensions: {0}".format(y_test.shape))
# baseline
decision_tree = DecisionTreeClassifier(random_state=1)
decision_tree.fit(X_train, y_train)

y_pred_train = decision_tree.predict(X_train)
y_pred_test = decision_tree.predict(X_test)

print("Baseline performance: \n")

print(
    "Data We Trained On Prediction Accuracy: {0}%".format(
        sum(y_train == y_pred_train) / len(y_pred_train)
    )
)
print(
    "Future Data Prediction Accuracy: {0}%".format(
        sum(y_test == y_pred_test) / len(y_pred_test)
    )
)
# setup parameter grid for grid search
parameters = {
    "criterion": ["gini", "entropy"],
    "min_samples_split": range(2, 10),
    "max_depth": range(2, 10),
    "max_leaf_nodes": range(2, 30),
}
print(parameters)
# perform grid search with 10 fold cross validation using the parameter grid above
# this ran in less than 30 seconds on my local machine
from sklearn.model_selection import GridSearchCV

decision_tree = DecisionTreeClassifier(random_state=1)
grid_search = GridSearchCV(
    estimator=decision_tree, param_grid=parameters, cv=10, refit=True
)
grid_search.fit(X_train, y_train)
# display best parameters chosen by grid search
print("Best parameters: {0}".format(grid_search.best_params_))<jupyter_output>Best parameters: {'criterion': 'entropy', 'max_depth': 6, 'max_leaf_nodes': 10, 'min_samples_split': 2}<jupyter_text>Review results<jupyter_code># generate predictions using best model and review accuracies
y_pred_train = grid_search.predict(X_train)
y_pred_test = grid_search.predict(X_test)

print("Grid search model performance: \n")

print(
    "Data We Trained On Prediction Accuracy: {0}%".format(
        sum(y_train == y_pred_train) / len(y_pred_train)
    )
)
print(
    "Future Data Prediction Accuracy: {0}%".format(
        sum(y_test == y_pred_test) / len(y_pred_test)
    )
)
# review best model as determined by GridSearchCV
grid_search.best_estimator_<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09970014992503748, 'fraction_numerical': 0.01664167916041979, 'mean_word_length': 5.521016617790812, 'pattern_counts': {'":': 4, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '259228', 'license': 'MIT', 'n_tokens_mistral': 2219, 'n_tokens_neox': 2122, 'n_words': 635, 'path': 'academics/MachineLearningPrinciples/Assignment05_ML.ipynb', 'repo_name': 'sudhu26/data-science-portfolio', 'size': '6670'}