<jupyter_start><jupyter_text>Introduction to Ensembling*Adapted from Chapter 8 of [An Introduction to Statistical Learning](http://www-bcf.usc.edu/~gareth/ISL/)*Let's pretend that instead of building a single model to solve a classification problem, you created **five independent models**, and each model was correct 70% of the time. If you combined these models into an "ensemble" and used their majority vote as a prediction, how often would the ensemble be correct?Let's simulate it to find out!<jupyter_code>import numpy as np

# set a seed for reproducibility
np.random.seed(1234)

# generate 1000 random numbers (between 0 and 1) for each model, representing 1000 observations
mod1 = np.random.rand(1000)
mod2 = np.random.rand(1000)
mod3 = np.random.rand(1000)
mod4 = np.random.rand(1000)
mod5 = np.random.rand(1000)

# each model independently predicts 1 (the "correct response") if random number was at least 0.3
preds1 = np.where(mod1 > 0.3, 1, 0)
preds2 = np.where(mod2 > 0.3, 1, 0)
preds3 = np.where(mod3 > 0.3, 1, 0)
preds4 = np.where(mod4 > 0.3, 1, 0)
preds5 = np.where(mod5 > 0.3, 1, 0)

# print the first 20 predictions from each model
print preds1[:20]
print preds2[:20]
print preds3[:20]
print preds4[:20]
print preds5[:20]
# add the predictions together
sum_of_preds = preds1 + preds2 + preds3 + preds4 + preds5

# ensemble predicts 1 (the "correct response") if at least 3 models predict 1
ensemble_preds = np.where(sum_of_preds >=3 , 1, 0)

# print the ensemble's first 20 predictions
print ensemble_preds[:20]
# how accurate was the ensemble?
ensemble_preds.mean()<jupyter_output><empty_output><jupyter_text>Amazing, right?**Ensemble learning (or "ensembling")** is simply the process of combining several models to solve a prediction problem, with the goal of producing a combined model that is more accurate than any individual model. For **classification** problems, the combination is often done by majority vote. For **regression** problems, the combination is often done by taking an average of the predictions.For ensembling to work well, the individual models must meet two conditions:- Models should be **accurate** (they must outperform random guessing)- Models should be **independent** (their predictions are not correlated with one another)The idea, then, is that if you have a collection of individually imperfect (and independent) models, the "one-off" mistakes made by each model are probably not going to be made by the rest of the models, and thus the mistakes will be discarded when averaging the models.It turns out that as you add more models to the voting process, the probability of error decreases. This is known as [Condorcet's Jury Theorem](http://en.wikipedia.org/wiki/Condorcet%27s_jury_theorem), which was developed by a French political scientist in the 18th century.Anyway, we'll see examples of ensembling below.  Bootstrapping**Some preliminary terminology:** In statistics, "bootstrapping" refers to the process of using "bootstrap samples" to quantify the uncertainty of a model. Bootstrap samples are simply random samples with replacement:<jupyter_code># set a seed for reproducibility
np.random.seed(1)

# create an array of 0 to 9, then sample 10 times with replacement
np.random.choice(a=10, size=10, replace=True)<jupyter_output><empty_output><jupyter_text>BaggingOn their own, decision trees are not competitive with the best supervised learning methods in terms of **predictive accuracy**. However, they can be used as the basis for more sophisticated methods that have much higher accuracy!One of the main issues with decision trees is **high variance**, meaning that different splits in the training data can lead to very different trees. **"Bootstrap aggregation" (aka "bagging")** is a general purpose procedure for reducing the variance of a machine learning method, but is particularly useful for decision trees.What is the bagging process (in general)?- Take repeated bootstrap samples (random samples with replacement) from the training data set- Train our method on each bootstrapped training set and make predictions- Average the predictionsThis increases predictive accuracy by **reducing the variance**, similar to how cross-validation reduces the variance associated with the test set approach (for estimating out-of-sample error) by splitting many times an averaging the results.  Applying bagging to decision treesSo how exactly can bagging be used with decision trees? Here's how it applies to **regression trees**:- Grow B regression trees using B bootstrapped training sets- Grow each tree deep so that each one has low bias- Every tree makes a numeric prediction, and the predictions are averaged (to reduce the variance)It is applied in a similar fashion to **classification trees**, except that during the prediction stage, the overall prediction is based upon a majority vote of the trees.**What value should be used for B?** Simply use a large enough value that the error seems to have stabilized. (Choosing a value of B that is "too large" will generally not lead to overfitting.)  Manually implementing bagged decision trees (with B=3)<jupyter_code>import pandas as pd

# read in vehicle data
vehicles = pd.read_csv('https://raw.githubusercontent.com/justmarkham/DAT4/master/data/used_vehicles.csv')

# convert car to 0 and truck to 1
vehicles['type'] = vehicles.type.map({'car':0, 'truck':1})

# print out data
vehicles
# calculate the number of rows in vehicles
n_rows = vehicles.shape[0]

# set a seed for reproducibility
np.random.seed(123)

# create three bootstrap samples (will be used to select rows from the DataFrame)
sample1 = np.random.choice(a=n_rows, size=n_rows, replace=True)
sample2 = np.random.choice(a=n_rows, size=n_rows, replace=True)
sample3 = np.random.choice(a=n_rows, size=n_rows, replace=True)

# print samples
print sample1
print sample2
print sample3
# use sample1 to select rows from DataFrame
print vehicles.iloc[sample1, :]
from sklearn.tree import DecisionTreeRegressor

# grow one regression tree with each bootstrapped training set
treereg1 = DecisionTreeRegressor(random_state=123)
treereg1.fit(vehicles.iloc[sample1, 1:], vehicles.iloc[sample1, 0])

treereg2 = DecisionTreeRegressor(random_state=123)
treereg2.fit(vehicles.iloc[sample2, 1:], vehicles.iloc[sample2, 0])

treereg3 = DecisionTreeRegressor(random_state=123)
treereg3.fit(vehicles.iloc[sample3, 1:], vehicles.iloc[sample3, 0])
# read in out-of-sample data
oos = pd.read_csv('https://raw.githubusercontent.com/justmarkham/DAT4/master/data/used_vehicles_oos.csv')

# convert car to 0 and truck to 1
oos['type'] = oos.type.map({'car':0, 'truck':1})

# print data
oos
# select feature columns (every column except for the 0th column)
feature_cols = vehicles.columns[1:]

# make predictions on out-of-sample data
preds1 = treereg1.predict(oos[feature_cols])
preds2 = treereg2.predict(oos[feature_cols])
preds3 = treereg3.predict(oos[feature_cols])

# print predictions
print preds1
print preds2
print preds3
# average predictions and compare to actual values
print (preds1 + preds2 + preds3)/3
print oos.price.values<jupyter_output>[  1533.33333333   5100.          13333.33333333]
[ 3000  6000 12000]<jupyter_text>Estimating out-of-sample errorBagged models have a very nice property: **out-of-sample error can be estimated without using the test set approach or cross-validation!**Here's how the out-of-sample estimation process works with bagged trees:- On average, each bagged tree uses about two-thirds of the observations. **For each tree, the remaining observations are called "out-of-bag" observations.**- For the first observation in the training data, predict its response using **only** the trees in which that observation was out-of-bag. Average those predictions (for regression) or take a majority vote (for classification).- Repeat this process for every observation in the training data.- Compare all predictions to the actual responses in order to compute a mean squared error or classification error. This is known as the **out-of-bag error**.**When B is sufficiently large, the out-of-bag error is an accurate estimate of out-of-sample error.**<jupyter_code># set is a data structure used to identify unique elements
print set(range(14))

# only show the unique elements in sample1
print set(sample1)
# use the "set difference" to identify the out-of-bag observations for each tree
print sorted(set(range(14)) - set(sample1))
print sorted(set(range(14)) - set(sample2))
print sorted(set(range(14)) - set(sample3))<jupyter_output>[4, 5, 7, 8]
[5, 6, 8, 10, 11, 12]
[1, 5, 10, 11]<jupyter_text>Thus, we would predict the response for **observation 4** by using tree 1 (because it is only out-of-bag for tree 1). We would predict the response for **observation 5** by averaging the predictions from trees 1, 2, and 3 (since it is out-of-bag for all three trees). We would repeat this process for all observations, and then calculate the MSE using those predictions.  Estimating variable importanceAlthough bagging **increases predictive accuracy**, it **decreases model interpretability** because it's no longer possible to visualize the tree to understand the importance of each variable.However, we can still obtain an overall summary of "variable importance" from bagged models:- To compute variable importance for bagged regression trees, we can calculate the **total amount that the mean squared error is decreased due to splits over a given predictor, averaged over all trees**.- A similar process is used for bagged classification trees, except we use the Gini index instead of the mean squared error.(We'll see an example of this below.)  Random ForestsRandom Forests is a **slight variation of bagged trees** that has even better performance! Here's how it works:- Exactly like bagging, we create an ensemble of decision trees using bootstrapped samples of the training set.- However, when building each tree, **each time a split is considered**, a random sample of m predictors is chosen as split candidates from the full set of p predictors. **The split is only allowed to use one of those m predictors.**Notes:- A new random sample of predictors is chosen for **every single tree at every single split**.- For **classification**, m is typically chosen to be the square root of p. For **regression**, m is typically chosen to be somewhere between p/3 and p.What's the point?- Suppose there is one very strong predictor in the data set. When using bagged trees, most of the trees will use that predictor as the top split, resulting in an ensemble of similar trees that are "highly correlated".- Averaging highly correlated quantities does not significantly reduce variance (which is the entire goal of bagging).- **By randomly leaving out candidate predictors from each split, Random Forests "decorrelates" the trees**, such that the averaging process can reduce the variance of the resulting model.<jupyter_code># read in the Titanic data
titanic = pd.read_csv('https://raw.githubusercontent.com/justmarkham/DAT4/master/data/titanic.csv')

# encode sex feature
titanic['sex'] = titanic.sex.map({'female':0, 'male':1})

# fill in missing values for age
titanic.age.fillna(titanic.age.mean(), inplace=True)

# create three dummy variables, drop the first dummy variable, and store this as a DataFrame
embarked_dummies = pd.get_dummies(titanic.embarked, prefix='embarked').iloc[:, 1:]

# concatenate the two dummy variable columns onto the original DataFrame
# note: axis=0 means rows, axis=1 means columns
titanic = pd.concat([titanic, embarked_dummies], axis=1)

# create a list of feature columns
feature_cols = ['pclass', 'sex', 'age', 'embarked_Q', 'embarked_S']

# print the updated DataFrame
titanic.head(10)
# import class, instantiate estimator, fit with all data
from sklearn.ensemble import RandomForestClassifier
rfclf = RandomForestClassifier(n_estimators=100, max_features='auto', oob_score=True, random_state=1)
rfclf.fit(titanic[feature_cols], titanic.survived)<jupyter_output><empty_output><jupyter_text>These are the most important tuning parameters for Random Forests:- **n_estimators:** more estimators (trees) increases performance but decreases speed- **max_features:** cross-validate to choose an ideal value<jupyter_code># compute the feature importances
pd.DataFrame({'feature':feature_cols, 'importance':rfclf.feature_importances_})
# compute the out-of-bag classification accuracy
rfclf.oob_score_<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08081213385763582, 'fraction_numerical': 0.02247010673300698, 'mean_word_length': 5.4536509580528225, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 29, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '318194', 'license': 'MIT', 'n_tokens_mistral': 3637, 'n_tokens_neox': 3355, 'n_words': 1570, 'path': 'notebooks/16_ensembling.ipynb', 'repo_name': 'larsonma/DataScienceIntro', 'size': '12461'}