<jupyter_start><jupyter_text>HW4 Trees, Forests, and Bag of WordsOfficial instructions:https://www.cs.tufts.edu/comp/135/2020f/hw4.htmlThis is the *starter code* notebook.<jupyter_code>import numpy as np
import pandas as pd
import os
import sys
import time
import sklearn.tree
import sklearn.linear_model
import sklearn.metrics
import sklearn.ensemble
# From the HW4 starter code
from pretty_print_sklearn_tree import pretty_print_sklearn_tree
# Plotting utils
import matplotlib
import matplotlib.pyplot as plt

%matplotlib inline
plt.style.use('seaborn') # pretty matplotlib plots

import seaborn as sns
sns.set('notebook', font_scale=1.25, style='whitegrid')<jupyter_output><empty_output><jupyter_text>Load all data from train/valid/test<jupyter_code># TODO fix to path on your local system
DATA_DIR = os.path.join("../data_product_reviews/")<jupyter_output><empty_output><jupyter_text>Load training<jupyter_code>x_tr_df = pd.read_csv(os.path.join(DATA_DIR, 'x_train.csv.zip'))
y_tr_df = pd.read_csv(os.path.join(DATA_DIR, 'y_train.csv'))
x_tr_NF = np.minimum(x_tr_df.values, 1.0).copy()
y_tr_N = y_tr_df.values[:,0].copy()

print("Training data")
print("x_tr_NF.shape: %s" % str(x_tr_NF.shape))
print("y_tr_N.shape : %s" % str(y_tr_N.shape))
print("mean(y_tr_N) : %.3f" % np.mean(y_tr_N))<jupyter_output>Training data
x_tr_NF.shape: (6346, 7729)
y_tr_N.shape : (6346,)
mean(y_tr_N) : 0.500<jupyter_text>Load validation set<jupyter_code>x_va_df = pd.read_csv(os.path.join(DATA_DIR, 'x_valid.csv.zip'))
y_va_df = pd.read_csv(os.path.join(DATA_DIR, 'y_valid.csv'))

x_va_TF = np.minimum(x_va_df.values, 1.0).copy()
y_va_T = y_va_df.values[:,0].copy()

print("Validation data")
print("x_va_TF.shape: %s" % str(x_va_TF.shape))
print("y_va_T.shape : %s" % str(y_va_T.shape))
print("mean(y_va_T) : %.3f" % np.mean(y_va_T))<jupyter_output>Validation data
x_va_TF.shape: (792, 7729)
y_va_T.shape : (792,)
mean(y_va_T) : 0.490<jupyter_text>Load test set<jupyter_code>x_te_df = pd.read_csv(os.path.join(DATA_DIR, 'x_test.csv.zip'))
y_te_df = pd.read_csv(os.path.join(DATA_DIR, 'y_test.csv'))

x_te_TF = np.minimum(x_te_df.values, 1.0).copy()
y_te_T = y_te_df.values[:,0].copy()

print("Heldout Test data")
print("x_te_TF.shape: %s" % str(x_te_TF.shape))
print("y_te_T.shape : %s" % str(y_te_T.shape))
print("mean(y_te_T) : %.3f" % np.mean(y_te_T))<jupyter_output>Heldout Test data
x_te_TF.shape: (793, 7729)
y_te_T.shape : (793,)
mean(y_te_T) : 0.515<jupyter_text>Load vocabulary as a list of strings<jupyter_code>vocab_list = x_tr_df.columns.tolist()
for word in vocab_list[:8]:
    print(word)
print("...")
for word in vocab_list[-8:]:
    print(word)<jupyter_output>good
great
time
book
don't
work
i_have
read
...
never_get
i'd_like
loves_it
an_author
nomin
could_give
bad_but
gap<jupyter_text>Pack training and validation sets into big arrays (so we can use sklearn's hyperparameter search tools)<jupyter_code>xall_LF = np.vstack([x_tr_NF, x_va_TF])
yall_L = np.hstack([y_tr_N, y_va_T])
valid_indicators_L = np.hstack([
    -1 * np.ones(y_tr_N.size), # -1 means never include this example in any test split
    0  * np.ones(y_va_T.size), #  0 means include in the first test split (we count starting at 0 in python)
    ])
# Create splitter object using Predefined Split
# Will be used later by all hyperparameter searches

my_splitter = sklearn.model_selection.PredefinedSplit(valid_indicators_L)<jupyter_output><empty_output><jupyter_text>Problem 1: Decision Trees 1A: Train a simple tree with depth 3<jupyter_code>simple_tree = sklearn.tree.DecisionTreeClassifier(
    max_depth=3, min_samples_split=2, min_samples_leaf=1, criterion='gini')<jupyter_output><empty_output><jupyter_text>**Fit the tree** **TODO Train on the training set** in the next coding cell<jupyter_code>simple_tree.fit(None, None) #TODO<jupyter_output><empty_output><jupyter_text>**Print Tree** Use a helper function from the starter code<jupyter_code>pretty_print_sklearn_tree(simple_tree, feature_names=vocab_list)<jupyter_output><empty_output><jupyter_text>1B : Find best Decision Tree with grid search<jupyter_code># Construct the default predictor
# Any hyperparameters here may be overridden by the hyperparameter grid

tree = sklearn.tree.DecisionTreeClassifier(
    criterion='gini', min_samples_split=2, min_samples_leaf=1)
tree_hyperparameter_grid_by_name = dict(
    max_depth=[2, 8, 32, 128],
    min_samples_leaf=[1, 3, 9],
    random_state = [101],
    )<jupyter_output><empty_output><jupyter_text>**TODO Build the Grid Search** in the next coding cellHint: See Lab for Grid Search: https://github.com/tufts-ml-courses/comp135-20f-assignments/blob/master/labs/day13_HyperparameterSearch.ipynbKey Function: sklearn.model_selection.GridSearchCVKey Requirements:* Provide the above tree_hyperparameter_grid_by_name dictionary as the set of hyperparameters to search* Set scoring='balanced_accuracy', since our target metric is balanced accuracy* Set cv=my_splitter so you can use the predefined split we defined earlier.* Set return_train_score=True (we want training set scores as well as test set scores)* Set refit=False (we only want fits on x_tr not on x_all)<jupyter_code>tree_grid_searcher = None # TODO<jupyter_output><empty_output><jupyter_text>Do the search!<jupyter_code>start_time_sec = time.time()
tree_grid_searcher.fit(xall_LF, yall_L)
elapsed_time_sec = time.time() - start_time_sec<jupyter_output><empty_output><jupyter_text>Build dataframe of resultsMove the results of grid search into a nice pandas data frame.<jupyter_code>tree_search_results_df = pd.DataFrame(tree_grid_searcher.cv_results_).copy()
print("Grid search of %3d configurations done after %6.1f sec" % (
    tree_search_results_df.shape[0], elapsed_time_sec))<jupyter_output><empty_output><jupyter_text>Display search resultsThis block will make a pretty printed table of the results of your grid search<jupyter_code>pd.set_option('precision', 4)
tree_keys = ['param_max_depth', 'param_min_samples_leaf']
tree_search_results_df.sort_values(tree_keys, inplace=True)
tree_search_results_df[tree_keys + ['mean_train_score', 'mean_test_score', 'rank_test_score', 'mean_fit_time']]
print("Printing a dict of the best hyperparameters")
print(tree_grid_searcher.best_params_)<jupyter_output>Printing a dict of the best hyperparameters<jupyter_text>Build the best decision tree **TODO Build the Best Tree** in the next coding cellThis is necessary so you have the specific best performing tree in your workspace.Although you fit many trees in the search, they were not stored, so we need to recreate the best one.Hint: Just feed the best hyperparameters as keyword args to construct the tree. Or see the lab about grid search.<jupyter_code>best_tree = tree.set_params(None) #TODO
best_tree.fit(x_tr_NF, y_tr_N)<jupyter_output><empty_output><jupyter_text>Interpret the best decision tree<jupyter_code>best_tree.tree_
pretty_print_sklearn_tree(best_tree, feature_names=vocab_list)<jupyter_output><empty_output><jupyter_text>Problem 2: Random forest<jupyter_code>forest = sklearn.ensemble.RandomForestClassifier(
    criterion='gini', min_samples_split=2, random_state=101)<jupyter_output><empty_output><jupyter_text>2A: Best Random Forest via grid searchFollow the instructions and using what you learn in 1B to finish this step.This block might take 2-10 minutes. (Takes about 2 min on staff Macbook laptops.)If yours runs significantly longer, try this out on Google Colab instead.<jupyter_code>forest = sklearn.ensemble.RandomForestClassifier(
    n_estimators=125,
    criterion='gini',
    max_depth=15,
    min_samples_split=2,
    min_samples_leaf=1)

forest_hyperparameter_grid_by_name = dict(
    max_features=[3, 10, 33, 100, 333],
    max_depth=[16, 32],
    min_samples_leaf=[1],
    n_estimators=[125],
    random_state=[101],
    )
# TODO construct a GridSearchCV object like you did above.

forest_searcher = None<jupyter_output><empty_output><jupyter_text>Do the search!<jupyter_code>#TODO<jupyter_output><empty_output><jupyter_text>Build dataframe of results<jupyter_code>#TODO<jupyter_output><empty_output><jupyter_text>Display search results<jupyter_code>#TODO<jupyter_output><empty_output><jupyter_text>Build the best random forest using the best hyperparameters found in 2B and train it.This is necessary so you have the specific best performing forest in your workspace.<jupyter_code>#TODO<jupyter_output><empty_output><jupyter_text>2B & Figure 2 : Feature Importances Access the **feature_importances_** attribute of your trained forest to get a score for each term in our vocabulary.A higher value of this score indicates the feature is "more important".In one panel of Figure 2, display a list of the top 10 vocabulary words of your best forest with highest feature importance.In another panel of Figure 2, display a list of 10 randomly chosen terms that have close-to-zero feature importance (use anything with importance less than 0.00001).  Figure 2.2**Sample Output** (Feel free to print all words and organize them in any software)|**Important Words**|**Unimportant Words**||:-:|:-:||I1 |  U1  ||I2 |  U2  ||I3 |  U3  ||I4 |  U4  ||I5 |  U5  ||I6 |  U6  ||I7 |  U7  ||I8 |  U8  ||I9 |  U9  ||I0 |  U0  |  Problem 3: Comparison of models 3A Implementation **TODO**:Selecting the best logistic regression with L1 penaltyTable 3 in Report<jupyter_code>lasso = sklearn.linear_model.LogisticRegression(
    penalty='l1', solver='saga', random_state=101)
lasso_hyperparameter_grid_by_name = dict(
    C=np.logspace(-4, 4, 9),
    max_iter=[20, 40], # sneaky way to do "early stopping" 
                       # we'll take either iter 20 or iter 40 in training process, by best valid performance
    )<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10558302430984755, 'fraction_numerical': 0.021219612690564483, 'mean_word_length': 6.428462127008416, 'pattern_counts': {'":': 0, '<': 92, '<?xml version=': 0, '>': 92, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '254686', 'license': 'MIT', 'n_tokens_mistral': 3626, 'n_tokens_neox': 3442, 'n_words': 858, 'path': 'hw4/hw4.ipynb', 'repo_name': 'ypark12/comp135-20f-assignments', 'size': '9708'}