<jupyter_start><jupyter_text>Copyright 2020 Google LLC.<jupyter_code># Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.<jupyter_output><empty_output><jupyter_text>XGBoost [XGBoost](https://github.com/dmlc/xgboost) is a powerful toolkit for performing machine learning tasks. Though much of the current excitement in the machine learning sphere comes from deep learning, XGBoost is a new non-deep-learning algorithm that has won data science competitions and a lot of attention. Instead of a deep neural network, XGBoost uses a collection of decision trees (random forest) arranged and optimized by a boosting scheme and unique penalization of trees in the forest.XGBoost started becoming popular as it started to perform as well as, if not better than, modern deep learning approaches, while also using less computing resources to train.  The Dataset To demonstrate XGBoost, we will use a [subset of data](https://www.kaggle.com/joshmcadams/sdss-16) from the [SDSS data release 16](https://www.sdss.org/dr16/).This dataset contains data observed about objects in space, as well as the type of object that was observed. We'll see if we can train a model that can classify objects based on these observations.To get started, upload your `kaggle.json` file and run the code below.<jupyter_code>! chmod 600 kaggle.json && (ls ~/.kaggle 2>/dev/null || mkdir ~/.kaggle) && mv kaggle.json ~/.kaggle/ && echo 'Done'<jupyter_output><empty_output><jupyter_text>Next we'll download the `sdss-16` dataset.<jupyter_code>! kaggle datasets download joshmcadams/sdss-16
! ls<jupyter_output><empty_output><jupyter_text>And then load it up and get some information about it. Notice that we use `.info()` to examine the `DataFrame`. The `.info()` method returns information about the data, including column counts and even memory usage.<jupyter_code>import pandas as pd

df = pd.read_csv('sdss-16.zip')
df.info()<jupyter_output><empty_output><jupyter_text>Exploratory Data Analysis We can tell from our `.info()` call that we have `732,977` non-null columns. First we should double check that every column is non-null.<jupyter_code>df.isnull().any()<jupyter_output><empty_output><jupyter_text>Good, no null data. Let's examine the columns that we are working with.Column | Description-------|-------------`object_id`            | Unique SDSS identifier composed from [skyVersion,rerun,run,camcol,field,obj].`right_ascension`      | J2000 right ascension (r')`declination`          | J2000 declination (r')`u_magnitude`          | Model magnitude u-band`g_magnitude`          | Model magnitude g-band`r_magnitude`          | Model magnitude r-band`i_magnitude`          | Model magnitude i-band`z_magnitude`          | Model magnitude z-band`obs_run_number`       | Run number`rerun_number`         | Re-run number`camera_column`        | Camera column`field_number`         | Field number`spectro_object_id`    | Unique ID`class`                | Type of object (target)`redshift`             | Final redshift`plate_id`             | Plate ID`observation_date`     | Date of observation`fiber_id`             | Fiber IDThe column that we'll be trying to predict is `class`.There are quite a few columns that seem to be more metadata than observation data, so we'll not include them in our features. Good candidates for this category include:* `object_id`* `obs_run_number`* `rerun_number`* `observation_date`* `spectro_object_id`Then there are columns that we need to understand better before we can decide whether we should include them or not. These include:* `camera_column`* `field_number`* `plate_id`* `fiber_id`Looking at the [SDSS DR16 Glossary](https://www.sdss.org/dr16/help/glossary/), these fields should likely not be used for classification either.That leaves us with the following potential feature columns:* `right_ascension`* `declination`* `u_magnitude`* `g_magnitude`* `r_magnitude`* `i_magnitude`* `z_magnitude`* `redshift`[Researching more](https://skyserver.sdss.org/dr16/en/proj/advanced/hubble/simple.aspx), it seems that all of these columns are real observations that might be useful to our model. Now that we know our target and features, let's take a look at them.First, we'll see how balanced our dataset is.<jupyter_code>df['class'].hist()<jupyter_output><empty_output><jupyter_text>Not really balanced. We have less than `100,000` quasars and over `350,000` galaxies. We'll need to stratify our testing split for sure. Let's see how our features are distributed.<jupyter_code>import matplotlib.pyplot as plt


features = [
  'right_ascension',
  'declination',
  'u_magnitude',
  'g_magnitude',
  'r_magnitude',
  'i_magnitude',
  'z_magnitude',
  'redshift',
]

fig, (axs) = plt.subplots(1, 8, figsize=(20, 10))
for i in range(len(features)):
  axs[i].title.set_text(features[i])
  axs[i].boxplot(df[features[i]])<jupyter_output><empty_output><jupyter_text>Wow, that is some spread. Ranges vary from single digits (ex: `redshift`) to large negatives (ex: `i_magnitude` and `z_magnitude`). Some values are relatively normally distributed like `declination`, while others are extremely focused with a wide range of outliers like in `redshift` and `u_magnitude`.Since we'll be using an XGBoost model backed by decision trees, we won't need to normalize this data.  The Model  Data Preprocessing First off, we'll want to convert our target `class` to a numeric value instead of the string values that we currently have. To do that we'll use the [sklearn `LabelEncoder`](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html). This encodes our three classes as the values `0`, `1`, and `2`.<jupyter_code>from sklearn.preprocessing import LabelEncoder


le = LabelEncoder()
df['encoded_class'] = df[['class']].apply(le.fit_transform)['class']
_ = df['encoded_class'].hist()<jupyter_output><empty_output><jupyter_text>We should then split the data, being sure to stratify.<jupyter_code>from sklearn.model_selection import train_test_split


X_train, X_test, y_train, y_test = train_test_split(
    df[features],
    df['encoded_class'],
    test_size=0.2,
    random_state=8675309,
    shuffle=True,
    stratify=df['encoded_class'])

_ = y_train.hist()<jupyter_output><empty_output><jupyter_text>Building the Model Now it is time to actually build the model. It is possible to build an XGBoost model in a similar manner as you would a standard scikit-learn model. For example, to create a classifier we can use `XGBClassifer`:```pythonfrom xgboost import XGBClassifiermodel = XGBClassifier()model.fit(X_train, y_train)```*But you will likely regret that decision!*One of the strengths of XGBoost is its ability to quickly converge on a quality model. In order to do this, we need to set up an input data structure that is optimized for XGBoost processing. This structure is known as the `DMatrix`.To create a `DMatrix`, we load `xgboost` and pass our testing and training data to the `DMatrix` constructor.<jupyter_code>import xgboost as xgb


d_train = xgb.DMatrix(X_train, y_train)
d_test = xgb.DMatrix(X_test, y_test)<jupyter_output><empty_output><jupyter_text>We can now train the model. But what type of model are we training?In the example above, we asked for an `XGBClassifer`, but with XGBoost you can also pass parameters that determine the type of model being trained. There are numerous [XGBoost parameters](https://xgboost.readthedocs.io/en/latest/parameter.html). One of the most important is the [`objective`](https://xgboost.readthedocs.io/en/latest/parameter.htmllearning-task-parameters). This tells XGBoost what type of model it should build.By default, the model is a regression model trained using squared error as a loss. In our case we want a multiclass classification model. Options for building this type of model include `multi:softmax` and `multi:softprob`. We'll go with softmax since it has been used a few times in this course.There are also model-specific parameters. In our case we chose `multi:softmax`, which requires that we set the number of classes using the `num_class` parameter.There are general parameters that apply to all models. One of those, `booster`, is set to `gbtree` by default. With a tree booster, there are [specific parameters](https://xgboost.readthedocs.io/en/latest/parameter.htmlparameters-for-tree-booster) that you can set including `max_depth` for the max depth of the trees and `eta` for the learning rate.Note that we didn't set the number of trees. This is because XGBoost determines the size of the forest using a gradient that eventually converges.You can probably see that [XGBoost parameters](https://xgboost.readthedocs.io/en/latest/parameter.html) can be pretty complex. One parameter opens up the option for more and more. This is because the `xgboost` class is a wrapper over many different types of models, from classification to regression to ranking. And each of those models has many potential backing algorithms, each with their own parameters.> *Check your parameters carefully. XGBoost doesn't throw errors when you add useless parameters. Be sure you are using the correct parameters for your current model configuration.*<jupyter_code>import time

params = {
    # What type of model are we building?
    'objective': 'multi:softmax',

    # multi:softmax parameters
    'num_class': 3,

    # General parameters
    'booster': 'gbtree',

    # Tree-specific parameters
    'max_depth': 10, 
    'eta': 0.1,
}

start = time.time()

model = xgb.train(params, d_train)

end = time.time()
print(end - start)<jupyter_output><empty_output><jupyter_text>We went ahead and timed this training operation just to get a feel for how fast XGBoost is. In our trial run, we trained in just under one minute.We can now see how well our trained model generalizes.<jupyter_code>from sklearn.metrics import f1_score
import numpy as np

preds = model.predict(d_test)
 
print(np.round(f1_score(y_test, preds, average='micro')*100, 2), '%')<jupyter_output><empty_output><jupyter_text>`99.31%`, not bad!Let's not celebrate too soon. How well does this compare to a vanilla random forest?<jupyter_code>from sklearn.ensemble import RandomForestClassifier

model = RandomForestClassifier(random_state=8675309)

start = time.time()

model.fit(X_train, y_train)

end = time.time()
print(end - start)

preds = model.predict(X_test)

print(np.round(f1_score(y_test, preds, average='micro')*100, 2), '%')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09068425391591096, 'fraction_numerical': 0.009251625904552533, 'mean_word_length': 4.946623093681917, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 55, 'https://': 11, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '92517', 'license': 'Apache-2.0', 'n_tokens_mistral': 3317, 'n_tokens_neox': 3161, 'n_words': 1246, 'path': 'content/06_other_models/07_xgboost/colab.ipynb', 'repo_name': 'skykykykykykykykykykykys/applied-machine-learning-intensive', 'size': '10917'}