<jupyter_start><jupyter_text>Part 1: Decision Trees With Grid SearchLook at the sklearn [DecisionTreeClassifier documentation](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html). For each hyperparameter:* Decide if it will impact the results or exclusively impact the computational speed.    * **Only the `ccp_alpha` value exclusively impacts computational speed.**    * However, some values don't always make sense...    * For example, you'll only want to use `class_weight` unless you have imbalanced classes in your dataset [(learn more here)](https://machinelearningmastery.com/cost-sensitive-decision-trees-for-imbalanced-classification/)* Pick at least 3 values for each hyperparameter that you'd like to include in your grid search.    * If there are fewer than 3 valid choices, include all valid choices.    * Carefully consider the choices, try to predict the impact of this hyperparameter on the outcome.     * Does this hyperparameter interact with others?    See my choices of values below. **criterion** {“gini”, “entropy”}, default=”gini”> Typically you'll end up with similar results using either of these. Gini is a bit faster to compute than entropy. Gini is a measure of "purity" while entropy is a measure of information/randomness. With Gini we are saying we want to maximize each split for homogeniety, with entropy we are maximizing the amount of "information gained" by performing the split in terms of reducing the randomness in the node. [Read more](https://towardsdatascience.com/gini-index-vs-information-entropy-7a7e4fed3fcb)**splitter** {“best”, “random”}, default=”best”> Best is most commonly used, and what almost everyone means when they describe a decision tree (including the lesson that this homework is based on...). Random will select still look at evrey feature but randomly select one split point per feature, then choose the best feature. I don't suggest it TBH — if you want to do something like this use a random forest instead. Using random will potentially avoid overfitting, but it will likely also decrease performance overall. **max_depth** int, default=None> The maximum depth of the tree. If None, then nodes are expanded until all leaves are pure or until all leaves contain less than min_samples_split samples. Shallow trees won't overfit as much, but at the risk of underfitting. Very shallow trees will only be allowed a few splits, whereas deeper trees will be allowed more. As the number of classess increases you may wish to increase the max_depth to allow your tree enough capacity to split out the classes. For example, if you have 10 classes, but a max depth of 3 (root->mid->leaf) then you'll have 4 leaf nodes, representing only 4 out of your possible 10 classes.**min_samples_split** int or float, default=2> The minimum number of samples required to split an internal node. Larger numbers here will smooth out the model by forcing your leaf nodes to represent a larger sub-sample of the data. This is especially true in regresssion trees, since you'll use the mean of that sub-sample as your prediction. However, as a consequence, with large numbers you might end up with "impure" leaf nodes that really ought to be split further.         **min_weight_fraction_leaf** float, default=0.0> The minimum weighted fraction of the sum total of weights (of all the input samples) required to be at a leaf node. Samples have equal weight when sample_weight is not provided. This is similar to min_samples_split but more useful when you have imbalanced classes and are using class_weight.**max_features** int, float or {“auto”, “sqrt”, “log2”}, default=None> Artificially limit the number of features that can be considered at each split point. If you choose a value less than the number of features available that number of features will be randomly sampled per split. This is very common in random forests, and in a single decision tree it CAN have a regularizing effect but I don't generally recommend it.  This will also speed up the computational time by reduing the overall search space.**random_state** int, RandomState instance, default=None> This can impact the model when there are ties for best split, or when you're using random feature selection / max_features. For deterministic results, use it. If you don't need totally deterministic results, you can ignore it. **max_leaf_nodes** int, default=None> Grow a tree with max_leaf_nodes in best-first fashion. Best nodes are defined as relative reduction in impurity. If None then unlimited number of leaf nodes. An alternative way to perform regularization / stop the model from overfitting. Different from others above because it will cause the tree to grow "best split first" rather than just building the tree layer by layer.         **min_impurity_decrease** float, default=0.0> A node will be split if this split induces a decrease of the impurity greater than or equal to this value. If splitting a node can't improve our metric (gini/entropy) by at least this much then a node won't be split. This will help us avoid overfitting, and it will smooth out our model somewhat, similar to min_samples_split.**class_weight** dict, list of dict or “balanced”, default=None> Weights associated with classes in the form {class_label: weight}. If None, all classes are supposed to have weight one. For multi-output problems, a list of dicts can be provided in the same order as the columns of y. See this link for more: **ccp_alpha** non-negative float, default=0.0> Complexity parameter used for Minimal Cost-Complexity Pruning. The subtree with the largest cost complexity that is smaller than ccp_alpha will be chosen. By default, no pruning is performed. See Minimal Cost-Complexity Pruning [(learn more here)](https://machinelearningmastery.com/cost-sensitive-decision-trees-for-imbalanced-classification/)<jupyter_code>import numpy as np
import pandas as pd
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import GridSearchCV


# Load the data
heart_dataset = pd.read_csv('../../datasets/uci-heart-disease/heart.csv')

# Split the data into input and labels
labels = heart_dataset['target']
input_data = heart_dataset.drop(columns=['target'])

# Note, we don't split the data. GridSearchCV will automatically apply 5-fold cross validation by default.


# Note, leaving out class_weights because our data is pretty balanced. 
tuned_parameters = {
    'criterion': ['gini', 'entropy'], # Not sure what will win out here TBH. Maybe gini.
    'splitter': ['best', 'random'], # best will be best, 
    'max_depth': [None, 5, 10], # None is probably best, allow the tree to grow as much as it needs
    'min_samples_split': [2, 10, 30], # 2 gives the model the most power to overfit, but I think it will win.
    'max_features':['auto', 'sqrt', 'log2'], # auto will probably be best.
    'max_leaf_nodes': [None, 4, 20], # I suspect 4 will be quite bad, and None will win
    'min_impurity_decrease': [0, 0.01, 0.5, 1, 5] # I added several b/c I'm curious. I think 0 will be best, relying on other params to smooth things.
}

# These two lines will result in every possible combo of the above paramters to be fit and scored
# which can take a LONG TIME with large datasets.
clf = DecisionTreeClassifier()
grid_tree = GridSearchCV(clf, tuned_parameters)
grid_tree.fit(input_data, labels) 

print("Best parameters set found on development set:")
print()
print(grid_tree.best_params_, grid_tree.best_score_)
print()
print("Grid scores on development set:")
print()
means = grid_tree.cv_results_['mean_test_score']
stds = grid_tree.cv_results_['std_test_score']



for mean, std, params in zip(means, stds, grid_tree.cv_results_['params']):
    print("%0.3f (+/-%0.03f) for %r"
          % (mean, std * 2, params))
sorted_performances = pd.DataFrame({'mean_accuracy': means, 'std': stds, 'params': grid_tree.cv_results_['params']})
sorted_performances = sorted_performances.sort_values('mean_accuracy', ascending=False)

from pprint import pprint
# top 5
print("\n=====top 5=====\n")
for index, row in sorted_performances.head(5).iterrows():
    print(row['mean_accuracy'])
    pprint(row['params'])
    
# bottom 5
print("\n=====bottom 5=====\n")
for index, row in sorted_performances.tail(5).iterrows():
    print(row['mean_accuracy'])
    pprint(row['params'])
# It's kind of hard to see patterns this way... lets look at what 
# choices were represented in each of our top 5s
choices = {}
for index, row in sorted_performances.head(5).iterrows():
    for key, value in row['params'].items():
        if not choices.get(key):
            choices[key] = []
        choices[key].append(value)
        
print("Top 5 hyperparmeters")
pprint(choices)
choices = {}
for index, row in sorted_performances.tail(5).iterrows():
    for key, value in row['params'].items():
        if not choices.get(key):
            choices[key] = []
        choices[key].append(value)
        
print("Top 5 hyperparmeters")
pprint(choices)<jupyter_output>Top 5 hyperparmeters
{'criterion': ['entropy', 'entropy', 'entropy', 'entropy', 'entropy'],
 'max_depth': [None, None, None, None, 10],
 'max_features': ['sqrt', 'sqrt', 'sqrt', 'sqrt', 'log2'],
 'max_leaf_nodes': [4, 4, 4, 4, 20],
 'min_impurity_decrease': [5, 5, 5, 5, 5],
 'min_samples_split': [2, 2, 10, 10, 30],
 'splitter': ['best', 'random', 'best', 'random', 'random']}<jupyter_text>Any Patterns?What I see in the above is:high values in `min_samples_split` perform poorly. `max_features` perform better when limited somehow vs unlimited (auto). Enforcing a `min_impurity_decrease` seems to hurt us, especially when using higher values. splitter value is quite mixed in both. Bonus Challenges* Get the best performing model (look at the `best_estimator` property in the `GridSearchCV` documentation) and examine the `feature_importances_` of that `DecisionTree`. What are these, and what are they telling us?* Create another `GridSearchCV` but this time include more metrics to score the model.    * List of all built in metrics can be [found here](https://scikit-learn.org/stable/modules/classes.htmlclassification-metrics). I suggest using accuracy, precision, and recall.    * A demo of using multiple metrics can be [found here](https://scikit-learn.org/stable/auto_examples/model_selection/plot_multi_metric_evaluation.htmlsphx-glr-auto-examples-model-selection-plot-multi-metric-evaluation-py)    * If you were building a real world medical screening system, how would you decide to value and balance these different metrics?        * [Read more about precision and recall](https://deepai.org/machine-learning-glossary-and-terms/precision-and-recall)        * Double bonus: can you make your GridSearch compute specificity as well? [Hint](https://stackoverflow.com/questions/47704133/how-to-define-specificity-as-a-callable-scorer-for-model-evaluation)<jupyter_code># Get the best estimator's feature importances.
# These tell us which features gave the model the most information as it split.
best_model = grid_tree.best_estimator_
feat_importances = pd.Series(best_model.feature_importances_, index=input_data.columns)
feat_importances.sort_values().plot(kind='barh', figsize=(10,10))
# Multi Metric CV
from sklearn.metrics import accuracy_score, precision_score, recall_score, make_scorer

scoring = {
    'accuracy': make_scorer(accuracy_score),
    'precision': make_scorer(precision_score),
    'recall': make_scorer(recall_score),
    'specificity': make_scorer(recall_score,pos_label=0) # Note, this is the double bonus. Not too bad!
}

tuned_parameters = {
    'criterion': ['gini', 'entropy'], # Not sure what will win out here TBH. Maybe gini.
    'splitter': ['best', 'random'], # best will be best, 
    'max_depth': [None, 5, 10], # None is probably best, allow the tree to grow as much as it needs
    'min_samples_split': [2, 10, 30], # 2 gives the model the most power to overfit, but I think it will win.
    'max_features':['auto', 'sqrt', 'log2'], # auto will probably be best.
    'max_leaf_nodes': [None, 4, 20], # I suspect 4 will be quite bad, and None will win
    'min_impurity_decrease': [0, 0.01, 0.5, 1, 5] # I added several b/c I'm curious. I think 0 will be best, relying on other params to smooth things.
}


clf = DecisionTreeClassifier()

# refit=false basically means we aren't choosing one estimator as the "best" one
# We do this because we have multi-metric scoring and it's no longer clear which is "best"
grid_tree = GridSearchCV(clf, tuned_parameters, scoring=scoring, refit=False) 
grid_tree.fit(input_data, labels) 

grid_tree.cv_results_



# Plus, check out this trick making the DF a bit easier to work with
results = pd.DataFrame({
    'accuracy': grid_tree.cv_results_['mean_test_accuracy'],
    'precision': grid_tree.cv_results_['mean_test_precision'],
    'recall': grid_tree.cv_results_['mean_test_recall'],
    'specificity': grid_tree.cv_results_['mean_test_specificity'],
})

params_df = pd.DataFrame(grid_tree.cv_results_['params'], columns=grid_tree.cv_results_['params'][0].keys())
all_results = pd.concat([results, params_df], axis=1)

# Lots of interesting analysis could be done on this dataframe now to learn
# about the trained models...
all_results<jupyter_output><empty_output><jupyter_text>Part 2, Random ForestsThe only new hyperparamters are:**n_estimators** > the number of trees to build. More is "better" but you'll hit a point of diminishing returns as bagging and feature selection can only create so much variance. **bootstrap** > Whether or not to use bagging. Generally you want this, otherwise your only source of variance is feature selection. **oob_score**> When bagging, you can use "out of bag" samples as a form of validation data, rather than splitting out a validation set explicitly. If your number of estimators is fairly high, you may not have many out of bag samples. We're using a validation set anyway, so we won't use this.**n_jobs**> Speed impacting only, we can build these trees in parallel and this value controls how many processes to spin up in that parallel creation process.**warm_start**> Only relevant for retraining an existing random forest — warm start can be used to add new estimators when calling .fit more than once on the same model. We won't use this either.Also note splitter is no longer available, best is always used for RFs.<jupyter_code>from sklearn.ensemble import RandomForestClassifier

# Load the data
heart_dataset = pd.read_csv('../../datasets/uci-heart-disease/heart.csv')

# Split the data into input and labels
labels = heart_dataset['target']
input_data = heart_dataset.drop(columns=['target'])

# Note, we don't split the data. GridSearchCV will automatically apply 5-fold cross validation by default.


# Note, leaving out class_weights because our data is pretty balanced. 
tuned_parameters = {
    'n_estimators': [10, 50, 100, 200], # 10 is too few, I guess 200 not tons better than 100
    'bootstrap': [True, False], # I guess false will generally perform worse
    'n_jobs': [-1],  # always using max number of jobs
    'criterion': ['gini', 'entropy'], # Not sure what will win out here TBH. Maybe gini.
    'max_depth': [None, 5, 10], # None is probably best, allow the tree to grow as much as it needs
    'min_samples_split': [2, 10, 30], # 2 gives the model the most power to overfit, but I think it will win.
    'max_features':['sqrt', 'log2'], # auto will probably be best.
    'max_leaf_nodes': [None, 4, 20], # I suspect 4 will be quite bad, and None will win
}

# These two lines will result in every possible combo of the above paramters to be fit and scored
# which can take a LONG TIME with large datasets.
clf = RandomForestClassifier()
grid_tree = GridSearchCV(clf, tuned_parameters)
grid_tree.fit(input_data, labels) 

print("Best parameters set found on development set:")
print()
print(grid_tree.best_params_, grid_tree.best_score_)
print()
print("Grid scores on development set:")
print()
means = grid_tree.cv_results_['mean_test_score']
stds = grid_tree.cv_results_['std_test_score']

for mean, std, params in zip(means, stds, grid_tree.cv_results_['params']):
    print("%0.3f (+/-%0.03f) for %r"
          % (mean, std * 2, params))
results = pd.DataFrame({'mean_accuracy': means, 'std': stds})
params_df = pd.DataFrame(grid_tree.cv_results_['params'], columns=grid_tree.cv_results_['params'][0].keys())
all_results = pd.concat([results, params_df], axis=1)
sorted_performances = all_results.sort_values('mean_accuracy', ascending=False)

sorted_performances.head(5)
sorted_performances.tail(5)<jupyter_output><empty_output><jupyter_text>Any Patterns?A surprise in the top 5, an `n_estimators = 10`, yet ALL the worst ones are the same. very interesting. It's the clear outlier though, the rest of the best are 100 or 200.Interesting that `max_leaf_nodes = 4` is 3/5 in the top 5, and None in 4/5 of the worst... Using "weak learners" in RF is common, and this is possibly an example of that. Having too many strong learners reduces the value of ensembling.  Note, part 2 bonuses:The code changes are identical for part 2 and part 1, so refer to part 1 for the answers.  Part 3: Comparing ResultsWrite a few paragraphs summarizing the differences you've found with this dataset and these two types of model. While writing, consider the following:* Is one of these two model types better "on average"?    * **the best RFs are better than the best DTs*** Consider the best and worse performing models of each type:    * Are the worst performing models similarly bad, or is one much better?    * What about the best performing, are they similarly good or is one much better?* Compare the training times for each type.    * Was one much faster than the other?        * **RF is much more expensive to train**    * Which hyperparameters could a major impact on time to train?        * **n_estimators, n_jobs, n_features are all big ones for training times**    * Imagine your dataset became much larger, could the time-to-train become a serious problem?        * **absolutely****Bonus challenge*** Compute and compare the 'feature importances' for each of your best performing models (the best performing DT and best performing RF).    * Did both models consider the same features to be "most important"?    * What about "least important"?    **Looking below, the distributions are actually very different. Less domination by any one feature, and the order is totally different. This isn't too surprising since the whole construction of an RF forces "non-optimal" choices, so we'd expect a flatter distribution as well as a different order as a result.**<jupyter_code># Grabbing the RF importances...
best_model = grid_tree.best_estimator_
feat_importances = pd.Series(best_model.feature_importances_, index=input_data.columns)
feat_importances.sort_values().plot(kind='barh', figsize=(10,10))
# Just for fun, I made a few charts related to the top 50 performing RFs and the relative commonality 
# of the hyperparameters.
from matplotlib import pyplot as plt
top_50 = sorted_performances.head(50)
top_50 = top_50.drop(columns=['mean_accuracy', 'std'])

for col in top_50.columns:
    top_50[col].value_counts(dropna=False).plot.pie()
    print(col)
    plt.show()<jupyter_output>bootstrap
{'fraction_non_alphanumeric': 0.09088078060933202, 'fraction_numerical': 0.010743758758498988, 'mean_word_length': 4.784449114380066, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 33, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '327855', 'license': 'Unlicense', 'n_tokens_mistral': 5600, 'n_tokens_neox': 5292, 'n_words': 2408, 'path': 'lessons/06-decision-trees-random-forests/04s-exercise-solution.ipynb', 'repo_name': 'Tebs-Lab/intro-python-for-machine-learning', 'size': '19267'}