<jupyter_start><jupyter_text>Decision trees and random forests<jupyter_code>import numpy as np
import pandas as pd

from sklearn import model_selection as ms, tree, ensemble

%matplotlib inline
WHITES_URL = 'http://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-white.csv'<jupyter_output><empty_output><jupyter_text>Read in the Wine Quality dataset.<jupyter_code>whites = pd.read_csv(WHITES_URL, sep=';')<jupyter_output><empty_output><jupyter_text>Define a new variable 'good_quality' for whites with quality >= 7.<jupyter_code>whites['good_quality'] = whites.quality >= 7<jupyter_output><empty_output><jupyter_text>Prepare the data.<jupyter_code>X = whites.drop(['quality', 'good_quality'], axis=1)
y = whites['good_quality']<jupyter_output><empty_output><jupyter_text>Decision trees Train a decision tree.<jupyter_code>tree1 = tree.DecisionTreeClassifier()
tree1.fit(X, y)<jupyter_output><empty_output><jupyter_text>Export the tree for plotting.<jupyter_code>tree.export_graphviz(tree1, 'tree1.dot', feature_names=X.columns)<jupyter_output><empty_output><jupyter_text>If you have [Graphviz](http://www.graphviz.org) installed, run `dot -Tpng tree1.dot -o tree1.png`.Alternatively, use [WebGraphviz](http://www.webgraphviz.com/). Define stratified folds for cross-validation.<jupyter_code>ten_fold_cv = ms.StratifiedKFold(n_splits=10, shuffle=True)<jupyter_output><empty_output><jupyter_text>Compute average AUC across folds.<jupyter_code>aucs = ms.cross_val_score(tree.DecisionTreeClassifier(), X, y, scoring='roc_auc', cv=ten_fold_cv)
np.mean(aucs)<jupyter_output><empty_output><jupyter_text>Train a decision tree by limiting:* the maximum number of questions (depth)* the minimum number of samples in each leaf<jupyter_code>tree2 = tree.DecisionTreeClassifier(max_depth=2, min_samples_leaf=50)
tree2.fit(X, y)<jupyter_output><empty_output><jupyter_text>Export the tree for plotting.<jupyter_code>tree.export_graphviz(tree2, 'tree2.dot', feature_names=X.columns)<jupyter_output><empty_output><jupyter_text>Investigate importances of predictors (the higher, the more important).<jupyter_code>tree2.feature_importances_<jupyter_output><empty_output><jupyter_text>Random forests Train a random forest with 20 decision trees.<jupyter_code>rf1 = ensemble.RandomForestClassifier(n_estimators=20)
rf1.fit(X, y)<jupyter_output><empty_output><jupyter_text>Investigate importances of predictors (the higher, the more important).<jupyter_code>rf1.feature_importances_<jupyter_output><empty_output><jupyter_text>Evaluate performance through cross-validation.<jupyter_code>aucs = ms.cross_val_score(ensemble.RandomForestClassifier(n_estimators=20),
                          X, y, scoring='roc_auc', cv=ten_fold_cv)
np.mean(aucs)<jupyter_output><empty_output><jupyter_text>What happens when we increase the number of trees?<jupyter_code>for n_trees in [2, 5, 10, 20, 50, 100]:
    aucs = ms.cross_val_score(
        ensemble.RandomForestClassifier(n_estimators=n_trees), X, y, scoring='roc_auc', cv=ten_fold_cv)
    print('{:>3} trees: mean AUC {:.2%}'.format(n_trees, np.mean(aucs)))<jupyter_output>2 trees: mean AUC 79.81%
  5 trees: mean AUC 86.76%
 10 trees: mean AUC 89.47%
 20 trees: mean AUC 91.44%
 50 trees: mean AUC 92.17%
100 trees: mean AUC 92.55%
{'fraction_non_alphanumeric': 0.11962160512664022, 'fraction_numerical': 0.023191943851083308, 'mean_word_length': 8.182072829131652, 'pattern_counts': {'":': 0, '<': 60, '<?xml version=': 0, '>': 63, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '391684', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 1250, 'n_tokens_neox': 1168, 'n_words': 216, 'path': '12_decision_trees_and_random_forests/notebooks/01_decision_trees_and_random_forests.ipynb', 'repo_name': 'JoseHJBlanco/ga-data-science', 'size': '3277'}