<jupyter_start><jupyter_text>XGBoost Lab ReflectionsLet's go back to thinking about a few algorithms we worked on. Decisions treesWe began our exploration of decision trees with a mountain bike example:![](https://raw.githubusercontent.com/zacharski/ml-class/master/labs/pics/dtree77.png)Here's is roughly what we did by hand.1. We determined that if we couldn't ask any questions, we would say the person mountain biked since they mountained bike 9 times and didn't 5 times. So our error rate was 5 out of 14 or roughly 36%2. Next, if we could ask one question we determined that the question should be about Outlook. Now our error rate was 4 out of 14 or 29%3. Then we determined the next question to ask and reduced the error rate more. And then the next question ...In some sense, the algorithm is additive. We start with zero questions with whatever error rate. Add a question and reduce the error rate. Add another question and reduce the rate. And so on.**Additive** is the key word. Let's look at an example, from [Gradient boosting: Distance to target](https://explained.ai/gradient-boosting/L2-loss.html)  by <NAME> and <NAME>. They ask us to imagine writing the formula for *y* that matches this plot:![](https://explained.ai/gradient-boosting/images/L2-loss/L2-loss_additive_2.svg)Like the decision tree example above, our first approximation might be simple, perhaps just the y-intercept:$$y = 30$$as shown in the leftmost picture below. ![](https://explained.ai/gradient-boosting/images/L2-loss/L2-loss_additive_3.svg)Next, we may want to add in the slope of the line and get$$y = 30 + x$$and get the middle graph above.  Finally, we add in the squiggle:$$y = 30 + x + sin(x)$$We have decomposed a complex task into subtasks, each refining the previous approximation. So, again, we have an additive algorithm.This approach shouldn't be surprising to us since this is how we typically develop programs. We get some skeleton code working and then incrementally add to it. BoostingBoosting algorithms work in a similar additive fashion. We first develop a simple model that roughly classifies the data. Next, we add another simple model that is focused on ameliorating the errors of the first. And then we add another and another.$$boosting=model_1 + model_2 + model_3 + ... + model_n$$ How boosting differs from bagging and pastingWith bagging and pasting we created a number of decision trees each of which was trained on different data. **One tree did not influence the construction of another.** Thus, each classifier was independent of the others.  BoostingBoosting is different. Imagine that we create one decision tree classifier. Let's call it Classifier 1. Classifier 1 doesn't perform with 100% accuracy. Next we create a second decision tree classifier and as part of its training data we will use the instances that Classifier got wrong. Now Classifier 2 isn't perfect either and there will be some instances that both Classifier 1 and Classifier 2 got wrong, and, you guessed it, we will use those instances as part of the training data for Classifier 3. 400 ClassifiersSuppose we created 400 classifiers using the bagging algorithm. Since each classifier is independent of the others, we can run those 400 in parallel. Now think about boosting for a moment. Can we run those in parallel? Think about it for 1. second2. seconds3. seconds4. seconds5. secondsSince one classifier is dependent on the errors of the others it seems like we couldn't run them in parallel and training 400 classifiers  sequentially seems impractical. This is true in general with boosting algorithms but as we will see XGBoost is different. Gradient BoostingSuppose I am interested in taking my camper van ![](https://raw.githubusercontent.com/zacharski/ml-class/master/labs/pics/travato2.png)to White Horse Road Dispersed Camping in Utah.![](https://raw.githubusercontent.com/zacharski/ml-class/master/labs/pics/wildHorse.png)And to get there from my home in Santa Fe, I am using an old school paper map.![](https://raw.githubusercontent.com/zacharski/ml-class/master/labs/pics/map.png)A route will be something like.$$route = road_0 + road_1 + road_2 + ... + road_n$$To get to White Horse Road, it looks like my best bet is to start by taking I25 to Albuquerque.$$route = i25 $$Now the difference between where I am and where I want to go is Albuquerque to White Horse. So I performed an action and now my new problem is dealing with this new problem of getting from Albuquerque to White HorseFrom Albuquerque I can take 550 to Farmington$$route = i25  + US550$$and from there take 491 to Monticello Utah$$route = i25  + US550+ US491$$and so on.There are some similarities between this old school mapping and gradient boosting. In gradient boosting we start with a poor model (in our case, we decided to go to Albuquerque). Then we are going to look at the difference between what we want and where we are-- and then take the next step, the delta $\Delta$. Let's look at a simple example of classification of one feature *x* to predict a label *y*. We will label our prediction $\hat{y}$. For gradient boosting our formula is$$\hat{y}=f_0(x) + \Delta_1(x) + \Delta_2(x) + ... + \Delta_m(x)$$Where $\Delta_1$ is the first improvement, $\Delta_2$ the second and so on.Gradient Boosting is an ensemble method, meaning that it is built with a number of sub-classifiers. So perhaps a better Utah analogy is that I hitchhike from here to Albuquerque with one person (one 'classifier'), then go to Framington with another and so on.This is the rough intuition of gradient boosting. In any gradient algorithm there is a parameter called *learning rate* and in a sense it is how big of steps we can take. Suppose we are hiking on a mountain in Utah and suddenly we are fogged in and can't see a thing. We want to get back to our van in the valley.In my 2D Utah it looks like this:![](https://raw.githubusercontent.com/zacharski/ml-class/master/labs/pics/gradient1.png)The purple dot is us near the top of the mountain and the burnt orange dot is our van. So our algorithm is```WHILE NOT AT VAN OR NOT MOVING:  take one step to the left.  IF we are lower than when we started:     stay here at the new location  ELSE     go back to starting point and go one step to the right     IF we are lower than when we started:        stay here at the new location     ELSE        go back to starting point```We repeat the above procedure and get to the state shown on the right above. If we take a step to the right or left we go uphill so we are stuck. We hit what is called a local minima and local minima are a problem with all gradient descent algorithms.Perhaps the one step was too small an increment. So let's say we have a rope. You stay where we are and hold one end of the rope and I walk until I reach the end of the rope. Based on the angle of the rope, we see if I am lower or not and we move accordingly. Now we jump over that local minima and reach a state that looks like the following image on the left. We don't know it, but we are almost to the van!![](https://raw.githubusercontent.com/zacharski/ml-class/master/labs/pics/gradient2.png)We use the rope technique again but this time I jump over the location of our van since I am not at the end of the rope yet and am in the position shown on the right. The learning rate was too large. (Now I am sounding like the three bears tale!)The one step was our learning rate as was our rope technique and you can see that selecting a good one is crucial.  Loss FunctionFor both these examples, one thing we needed was a measure for how far away are we from our goal. Are we better or worse? For the fog on a mountain example, the loss function was our altitude and we are trying to reduce the loss -- the altitude.  Two more examples One Dimensional Team Frisbee GolfHere is my representation of our 1D golf game. The hole is the green circle on the right and our frisbee's location is shown with the lovely pink circle on the left. Let $y$ be the actual distance between the two and $x$ what I see standing by the frisbee--off in the one dimensional distance I see the hole. ![](https://raw.githubusercontent.com/zacharski/ml-class/master/labs/pics/golf1.png)It is player zero's turn and she estimates the distance to be 70 yards.$$f_0(x) = 70$$ She flings the frisbee and ...![](https://raw.githubusercontent.com/zacharski/ml-class/master/labs/pics/golf2.png)Now it is player two's turn. He is only concerned with the  difference, the $\Delta_1$ --the current position of the frisbee and the location of the hole. He estimates it to be 20 yards$$\Delta_1(x) = 20$$So far we have flung the frisbee$$\hat{y}= f+0(x) + \Delta_1(x) = 70 + 20 = 90$$![](https://raw.githubusercontent.com/zacharski/ml-class/master/labs/pics/golf3.png)Now it is player two's turn. She estimates the distance remaining ($\Delta_2$) to be 15 yards...![](https://raw.githubusercontent.com/zacharski/ml-class/master/labs/pics/golf4.png)And she overshot. Player three estimates the remaining distance to be -5 yards and ...![](https://raw.githubusercontent.com/zacharski/ml-class/master/labs/pics/golf5.png)Notice that each player is not concerned with the original problem. She is just concerned with the **residual** --- meaning what is remaining based on the previous players' results.The formula is $$\hat{y} = f_0(x) + \Delta_1(x) + \Delta_2(x) + ... + \Delta_m(x)$$$$=f_0(x) + \sum_{m=1}^M{\Delta_m(x)}$$So the first classifier works on the original problem but all the rest work on the residual. Expenditures on Makeup and ClothesOk, I have exhausted my creativity, so even though I am not keen on this example, let's go back to predicting a young lady's expenditure on makeup based on what she spends on clothes.  And just for readability I am going to make the feature clothes to be represented by *x* and what we want to predict, the makeup, *y*<jupyter_code>import pandas as pd
from pandas import DataFrame
makeup =  [3000, 5000, 12000, 2000, 7000,  15000,  5000,  6000, 8000,  10000]
clothes = [7000, 8000, 25000, 5000, 12000, 30000, 10000, 15000, 20000, 18000]
ladies = ['Ms A','Ms B','Ms C','Ms D','Ms E','Ms F','Ms G','Ms H','Ms I','Ms J',]
monthly = DataFrame({'x': clothes, 'y': makeup}, index= ladies)
monthly<jupyter_output><empty_output><jupyter_text>And for our first prediction $f_0$ let's predict just the average value:<jupyter_code>monthly['f0'] = monthly.y.mean()
monthly<jupyter_output><empty_output><jupyter_text>and the differences between our predictions and the actual values<jupyter_code>monthly['y-f0'] = monthly.y - monthly.f0
monthly<jupyter_output><empty_output><jupyter_text>That $y-f_0$ is the residual. What is left, or how far the first classifier was off. The residual is what the second classifier is trying to predict.Next, we are going to create a classifier Δ1 that predicts $y - f_0$ from the x. Let's say my next classifier has the wacky$$(x - 10000)$$<jupyter_code>monthly['Δ1'] = (monthly['x'] - 10000)
monthly['y-f1'] = monthly['y-f0'] - monthly['Δ1']
monthly<jupyter_output><empty_output><jupyter_text>And the next classifier will try to predict $y-f_1$ based on x.If you understand all these examples, from Utah to Makeup, you have a pretty good intuition on how Gradient Boosting works. XGBoostYou may recall that in the first few videos, we mentioned that XGBoost was one of the state-of-the-art algorithms. The Kaggle competition winners are dominated by deep learning and XGBoost solutions.>I only use XGBoost (Liberty Mutual Property Inspection, Winner's Interview: <NAME>)> As the winner of an increasing amount of Kaggle competitions XGBoost showed us again to be a great all-around algorithm worith having in your toolbox (Dato Winner's Interview, 1st Place, Mad Professors)> The only supervised learning method I used was gradient boosting as implemented in the excellent xgboost package (Recruit Coupon Purchase Winner's Interview, 2nd place, Halla Yang)We are going to start our exploration of XGBoost using the Iris dataset, which we have used before.<jupyter_code>from IPython.display import YouTubeVideo
YouTubeVideo('1jLIRJwfZhg')<jupyter_output><empty_output><jupyter_text>This reminds me of a section of the *Hitchhiker's Guide to the Galaxy* by <NAME>, where Marvin, the robot, is asked to bring two hitchhikers to the bridge and he says:> Here I am, brain the size of a planet, and they ask me to take you to the bridge. Call that job satisfaction, 'cause I don'tXGBoost is an extremely powerful state-of-the-art algorithm and we are using it on a toy example. Oh well. GPU!We are going to be running this code on a Graphics Processing Unit, GPU, a graphics card.To do so, under the runtime menu above, select **Change Runtime Type** and select **GPU**That's it! Now let's check out what GPU we are using:<jupyter_code>!nvidia-smi<jupyter_output>Thu Mar 25 16:33:36 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.56       Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |
| N/A   57C    P8    10W /  70W |      0MiB / 15109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-------[...]<jupyter_text>It is a Tesla T4, which has 320 tensor cores.Now let's load the database The Iris Data Set Load the dataset:<jupyter_code>import pandas as pd
from sklearn.model_selection import train_test_split

iris = pd.read_csv('https://raw.githubusercontent.com/zacharski/ml-class/master/data/iris.csv')

iris_train, iris_test = train_test_split(iris, test_size = 0.2)
train_X = iris_train[['Sepal Length', 'Sepal Width', 'Petal Length', 'Petal Width']]
train_y = iris_train['Class']
test_X = iris_test[['Sepal Length', 'Sepal Width', 'Petal Length', 'Petal Width']]
test_y = iris_test['Class']<jupyter_output><empty_output><jupyter_text>Create an instance of the XGBoost classifierWe are going to create an XGBoost classifier with gpu support.<jupyter_code>from xgboost import XGBClassifier
params = { "n_estimators": 400, 'tree_method':'gpu_hist', 'predictor':'gpu_predictor' }

model = XGBClassifier(**params)
model<jupyter_output><empty_output><jupyter_text>Let's take a look at those parameters.* **n_estimators** the number of classifiers in the boost ensemble. The default is 100.* **tree_method** the tree construction algorithm that is used. `gpu_hist` is a distributed histogram approach (see the [original paper](https://arxiv.org/pdf/1603.02754.pdf))* **predictor** the prediction algorithm to use. `gpu_predictor` means use the gpu!* **max_depth** the depth of the decision trees. The default of 3 is used here. The trees for any ensemble method are typically very shallow.  Fitting model to the data<jupyter_code>model.fit(train_X, train_y)<jupyter_output><empty_output><jupyter_text>evaluate modelFinally let's evaluate the model<jupyter_code>from sklearn.metrics import accuracy_score

iris_predictions = model.predict(test_X)
accuracy_score(test_y, iris_predictions)<jupyter_output><empty_output><jupyter_text>We ran a state-of-the-art algorithm on a GPU. Yay us!Now we are going to back up quite a bit. Bagging and PastingWith bagging and pasting we created a number of decision trees each of which was trained on different data. One tree did not influence the construction of another. Each classifier was independent of the others.  BoostingBoosting is different. Imagine that we create one decision tree classifier. Let's call it Classifier 1. Classifier 1 doesn't perform with 100% accuracy. Next we create a second decision tree classifier and as part of its training data we will use the instances that Classifier got wrong. Now Classifier 2 isn't perfect either and there will be some instances that both Classifier 1 and Classifier 2 got wrong, and, you guessed it, we will use those instances as part of the training data for Classifier 3. 400 ClassifiersSuppose we created 400 classifiers using the bagging algorithm. Since each classifier is independent of the others, we can run those 400 in parallel. Now think about boosting for a moment. Can we run those in parallel?Since one classifier is dependent on the errors of the others it seems like we couldn't run them in parallel and doing 400 classifiers in series seems impractical. Fortunately for us, XGBoost has parallelized training! The task - The Adult DatasetLet's try a bit larger dataset, the [Adult Dataset](http://archive.ics.uci.edu/ml/datasets/Adult). The webpage describes the problem. We are trying to predict whether someone makes more that $50,000 year based on a number of features. The data folder contains both training data `adult.data` and test data `adult.test`.  Prepare the data.<jupyter_code>colNames = ['age', 'workclass', 'fnlwgt', 'education', 'education-num', 'marital-status', 'occupation', 
            'relationship', 'race', 'sex', 'capital-gain', 'capital-loss', 'hours-per-week', 'native-country', 'wage']
adult = pd.read_csv('http://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data', names=colNames)
adult<jupyter_output><empty_output><jupyter_text>divide features and labelslet's create 2 DataFrames, one for the features and one for the labels<jupyter_code>adult_features = adult.drop('wage', axis=1)
adult_labels = adult['wage']
adult_features<jupyter_output><empty_output><jupyter_text>Now let's one hot encode the features using sklearn's OneHotEncoder.<jupyter_code>#TODO  
adultSparse = pd.get_dummies(adult, columns=['age', 'workclass', 'fnlwgt', 'education', 'education-num',
       'marital-status', 'occupation', 'relationship', 'race', 'sex',
       'capital-gain', 'capital-loss', 'hours-per-week', 'native-country'])

adultSparse<jupyter_output><empty_output><jupyter_text>Fantastic!  Let's go ahead divide this up into training and test sets (Notice that this is a bit different than we have been doing it.<jupyter_code>from sklearn.model_selection import train_test_split
adult_train_features, adult_test_features, adult_train_labels, adult_test_labels = train_test_split(adultSparse, adult_labels, test_size = 0.7)
adult_train_features = adult_train_features.drop(['wage'], axis=1)<jupyter_output><empty_output><jupyter_text>You may have noticed that we put a whopping 70% of the data in the test set. We did this because when we are just playing with things to gain an understanding we don't want to wait hours for a result.Create an XGBoost classifier called model with the parameters:* `tree_method: gpu_hist`* `predictor: gpu_predictor`<jupyter_code>## TO DO call it model

from xgboost import XGBClassifier
params = { 'tree_method':'gpu_hist', 'predictor':'gpu_predictor' }

model = XGBClassifier(**params)
model<jupyter_output><empty_output><jupyter_text>Now let's say we want to find the best hyperparameter values for * n_estimators -- let's try 50, 100, 150, 200* max_depth -- let's try 2, 4, 6, 8Go ahead and create the `param_grid`<jupyter_code>model.fit(adult_train_features, adult_train_labels)


from sklearn.model_selection import GridSearchCV, KFold
from keras.models import Sequential
from keras.layers import Dense, Dropout
from keras.wrappers.scikit_learn import KerasClassifier
from keras.optimizers import Adam
import sys
import pandas as pd
seed=42
import numpy as np
n_estimators=[50,100,150,200]
max_depth=[2,4,6,8]

param_grid=dict(n_estimators=n_estimators, max_depth=max_depth)

grid = GridSearchCV(estimator=model, param_grid=param_grid, cv=KFold(random_state=seed), verbose=10)
grid_results = grid.fit(adult_train_features, adult_train_labels)

print("Best: {0}, using {1}".format(grid_results.best_score_, grid_results.best_params_))

means = grid_results.cv_results_['mean_test_score']
stds = grid_results.cv_results_['std_test_score']
params = grid_results.cv_results_['params']

for mean, stdev, param in zip(means, stds, params):
    print('{0} ({1}) with: {2}'.format(mean, stdev, param))
print("it is done")<jupyter_output>/usr/local/lib/python3.7/dist-packages/sklearn/model_selection/_split.py:296: FutureWarning: Setting a random_state has no effect since shuffle is False. This will raise an error in 0.24. You should leave random_state to its default (None), or set shuffle=True.
  FutureWarning
[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.<jupyter_text>Oh my goodness, that took forever.Here are the results: Best: 0.867629204420567, using {'max_depth': 6, 'n_estimators': 200}  Time ConstraintEven with a GPU it is going to take a long time to do an exhaustive search of which parameters are best. There are 16 possible combinations. We may want 5 fold cross validation. That is 80 fits, each of which is creating on average 100 classifiers. And we have around 20,000 instances in our training data. Let's pick a random smaller set of combinations to test.  Let's say we want the search algorithm to select 5 combinations of hyperparameters `param_comb` at random.<jupyter_code># TO DO
from sklearn.model_selection import StratifiedKFold
from sklearn.model_selection import RandomizedSearchCV
param_comb = 5
folds=5
skf = StratifiedKFold(n_splits=folds, shuffle = True, random_state = 1001)
random_search = RandomizedSearchCV(model, param_distributions=param_grid, n_iter=param_comb,  n_jobs=-1, 
                                   cv=skf.split(adult_train_features,adult_train_labels), verbose=3)<jupyter_output><empty_output><jupyter_text>Let's fit the model (this will take awhile)<jupyter_code>%%time 
#grid_result = random_search.fit(adult_train_features, adult_train_labels)
#This is what i did above and it took 40 minutes<jupyter_output>CPU times: user 3 µs, sys: 0 ns, total: 3 µs
Wall time: 6.91 µs<jupyter_text>Now let's see what the best parameters are, make predictions on our test data, and check accuracy...<jupyter_code>grid_results.best_params_

predictions = random_search.best_estimator_.predict(adult_test_features)

accuracy_score(adult_test_labels, predictions)<jupyter_output><empty_output><jupyter_text>This ends our first look at XGBoostT<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09267668974031128, 'fraction_numerical': 0.02001675411137075, 'mean_word_length': 4.372335386072951, 'pattern_counts': {'":': 1, '<': 82, '<?xml version=': 0, '>': 86, 'https://': 16, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '16132', 'license': 'Apache-2.0', 'n_tokens_mistral': 6947, 'n_tokens_neox': 6342, 'n_words': 2876, 'path': 'xgboostLab.ipynb', 'repo_name': 'AbigailBuchholz/test', 'size': '22708'}