<jupyter_start><jupyter_text>Exercises Chapter 6 - Decision Trees  Exercise 1 What is the approximate depth of a Decision Tree trained (without restrictions)on a training set with one million instances? $\log_2(1000000)$<jupyter_code>import numpy as np
np.log2(1000000)<jupyter_output><empty_output><jupyter_text>Exercise 2 Is a node’s Gini impurity generally lower or greater than its parent’s? Is it generally lower/greater, or always lower/greater? The Gini impurity decreases at each level  Exercise 3 If a Decision Tree is overfitting the training set, is it a good idea to try decreasingmax_depth ? Decrease max_depth to provide more regularization.  Exercise 4 If a Decision Tree is underfitting the training set, is it a good idea to try scalingthe input features? Scaling has no relevance for Decision Trees  Exercise 5 If it takes one hour to train a Decision Tree on a training set containing 1 millioninstances, roughly how much time will it take to train another Decision Tree on atraining set containing 10 million instances? ???  Exercise 6 If your training set contains 100,000 instances, will setting presort=True speedup training? Probably not, as pre-sorting will take longer than fitting directly  Exercise 7 Train and fine-tune a Decision Tree for the moons dataset by following thesesteps:a. Use make_moons(n_samples=10000, noise=0.4) to generate a moons dataset.b. Use train_test_split() to split the dataset into a training set and a test set.c. Use grid search with cross-validation (with the help of the GridSearchCVclass) to find good hyperparameter values for a DecisionTreeClassifier .Hint: try various values for max_leaf_nodes .d. Train it on the full training set using these hyperparameters, and measureyour model’s performance on the test set. You should get roughly 85% to 87%accuracy.<jupyter_code>from sklearn.datasets import make_moons
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.tree import DecisionTreeClassifier
X, y = make_moons(n_samples=10000, noise=0.4)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
param_grid = {'max_leaf_nodes':[3, 4, 5, 6, 10],
              'max_depth':[None, 5, 10], 
              'min_samples_leaf':[1,2, 3,5,10],
              'min_samples_split':[2, 3, 4, 8]}
clf = DecisionTreeClassifier()
grid = GridSearchCV(clf, param_grid, n_jobs=-1)
_ = grid.fit(X_train, y_train)
grid.best_estimator_
y_pred = grid.predict(X_test)
print(accuracy_score(y_pred, y_test))<jupyter_output>0.854<jupyter_text>Exercise 8 Grow a forest by following these steps:a. Continuing the previous exercise, generate 1,000 subsets of the training set, each containing 100 instances selected randomly. Hint: you can use Scikit-Learn’s ShuffleSplit class for this.b. Train one Decision Tree on each subset, using the best hyperparameter values found in the previous exercise. Evaluate these 1,000 Decision Trees on the test set. Since they were trained on smaller sets, these Decision Trees will likely perform worse than the first Decision Tree, achieving only about 80% accuracy.c. Now comes the magic. For each test set instance, generate the predictions of the 1,000 Decision Trees, and keep only the most frequent prediction (you can use SciPy’s mode() function for this). This approach gives you majority-vote predictions over the test set.d. Evaluate these predictions on the test set: you should obtain a slightly higher accuracy than your first model (about 0.5 to 1.5% higher). Congratulations, you have trained a Random Forest classifier!<jupyter_code>import numpy as np

from scipy.stats import mode

from sklearn.base import clone
from sklearn.model_selection import ShuffleSplit
shuffler = ShuffleSplit(n_splits=1000, train_size=100, test_size=1)
trees = []
for idx in shuffler.split(X_train):
    trees.append(clone(grid.best_estimator_))
    trees[-1].fit(X_train[idx[0],],y_train[idx[0],])
scores = []
for tree in trees:
    y_pred = tree.predict(X_test)
    scores.append(accuracy_score(y_pred, y_test))
print('Average accuracy score is {:6.4f} with a standard deviation of {:6.4f}'.format(np.mean(scores), np.std(scores)))
predictions = np.zeros((1000, len(y_test)))
for i,tree in enumerate(trees):
    predictions[i,:] = tree.predict(X_test)
preds_rf = np.apply_along_axis(mode, 0, predictions)
y_pred_rf = preds_rf[0,:,:].squeeze()
print('Accuracy Score of Random Forest: {:6.4f}'.format(accuracy_score(y_pred_rf, y_test)))<jupyter_output>Accuracy Score of Random Forest: 0.8555
{'fraction_non_alphanumeric': 0.0760941279964812, 'fraction_numerical': 0.02793050362876622, 'mean_word_length': 5.414668547249647, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '449070', 'license': 'Apache-2.0', 'n_tokens_mistral': 1455, 'n_tokens_neox': 1323, 'n_words': 547, 'path': 'exercises/exercises_ch6.ipynb', 'repo_name': 'pbr142/handson-ml', 'size': '4547'}