<jupyter_start><jupyter_text>Introduction to Dask XGBoost By <NAME>-------In this notebook, we will show how to work with Dask XGBoost in RAPIDS.**Table of Contents*** [Introduction to Dask XGBoost](introduction)* [Setup](setup)* [Load Libraries](libraries)* [Create a Cluster and Client](cluster)* [Generate Data](generate)  * [Load Data](load)  * [Simulate Data](simulate)  * [Split Data](split)  * [Check Dimensions](check)* [Distribute Data using Dask cuDF](distribute)* [Set Parameters](parameters)* [Train Model](train)* [Generate Predictions](predict)* [Evaluate Model](evaluate)* [Conclusion](conclusion)  SetupThis notebook was tested using the following Docker containers:* `rapidsai/rapidsai-nightly:0.8-cuda10.0-devel-ubuntu18.04-gcc7-py3.7` from [DockerHub - rapidsai/rapidsai-nightly](https://hub.docker.com/r/rapidsai/rapidsai-nightly)This notebook was run on the NVIDIA Tesla V100 GPU. Please be aware that your system may be different and you may need to modify the code or install packages to run the below examples. If you think you have found a bug or an error, please file an issue here: https://github.com/rapidsai/notebooks/issuesBefore we begin, let's check out our hardware setup by running the `nvidia-smi` command.<jupyter_code>!nvidia-smi<jupyter_output>Thu Dec  2 21:14:06 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.00       Driver Version: 510.06       CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  On   | 00000000:01:00.0 Off |                  N/A |
| N/A    0C    P8    N/A /  N/A |     77MiB /  4096MiB |     N/A      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-------[...]<jupyter_text>Next, let's see what CUDA version we have.<jupyter_code>!nvcc --version<jupyter_output>/bin/bash: nvcc: command not found<jupyter_text>Load LibrariesLet's load some of the libraries within the RAPIDs ecosystem and see which versions we have.<jupyter_code>import cudf; print('cuDF Version:', cudf.__version__)
import dask; print('Dask Version:', dask.__version__)
import dask_cudf; print('Dask cuDF Version:', dask_cudf.__version__)
import dask_xgboost; print('Dask XGBoost Version:', dask_xgboost.__version__)
import numpy as np; print('numpy Version:', np.__version__)
import pandas as pd; print('pandas Version:', pd.__version__)
import sklearn; print('Scikit-Learn Version:', sklearn.__version__)
# import xgboost as xgb; print('XGBoost Version:', xgb.__version__)<jupyter_output>cuDF Version: 21.08.03
Dask Version: 2021.07.1
Dask cuDF Version: 21.08.03
Dask XGBoost Version: 0.1.11
numpy Version: 1.21.4
pandas Version: 1.2.5
Scikit-Learn Version: 1.0.1<jupyter_text>Create a Cluster and ClientLet's start by creating a local cluster of workers and a client to interact with that cluster.<jupyter_code>from dask.distributed import Client
from dask_cuda import LocalCUDACluster


# create a local CUDA cluster
cluster = LocalCUDACluster()
client = Client(cluster)
client<jupyter_output><empty_output><jupyter_text>Generate Data Load DataWe can load the data using `pandas.read_csv`. We've provided a helper function `load_data` that will load data from a CSV file (and will only read the first 1 billion rows if that file is unreasonably big).<jupyter_code># helper function for loading data
def load_data(filename, n_rows):
    if n_rows >= 1e9:
        df = pd.read_csv(filename)
    else:
        df = pd.read_csv(filename, nrows=n_rows)
    return df.values.astype(np.float32)<jupyter_output><empty_output><jupyter_text>Simulate DataAlternatively, we can simulate data for our train and validation datasets. The features will be tabular with `n_rows` and `n_columns` in the training dataset, where each value is either of type `np.float32`. We can simulate data for both classification and regression using the `make_classification` or `make_regression` functions from the Scikit-Learn package.<jupyter_code>from sklearn.datasets import make_classification, make_regression


# helper function for simulating data
def simulate_data(m, n, k=2, random_state=None, classification=True):
    if classification:
        features, labels = make_classification(n_samples=m, n_features=n, 
                                               n_informative=int(n/5), n_classes=k, 
                                               random_state=random_state)
    else:
        features, labels = make_regression(n_samples=m, n_features=n, 
                                           n_informative=int(n/5), n_targets=1, 
                                           random_state=random_state)
    return np.c_[labels, features].astype(np.float32)
# settings
simulate = True
classification = True  # change this to false to use regression
n_rows = int(1e6)  # we'll use 1 millions rows
n_columns = int(100)
n_categories = 2
random_state = np.random.RandomState(43210)
%%time

if simulate:
    dataset = simulate_data(n_rows, n_columns, n_categories, 
                            random_state=random_state, 
                            classification=classification)
else:
    dataset = load_data('/tmp', n_rows)
print(dataset.shape)<jupyter_output>(1000000, 101)
CPU times: user 17.1 s, sys: 6.3 s, total: 23.4 s
Wall time: 15.1 s<jupyter_text>Split DataWe'll split our dataset into a 80% training dataset and a 20% validation dataset.<jupyter_code># identify shape and indices
n_rows, n_columns = dataset.shape
train_size = 0.80
train_index = int(n_rows * train_size)

# split X, y
# these are still arrays, not dfs
X, y = dataset[:, 1:], dataset[:, 0]
del dataset

# split train data
X_train, y_train = X[:train_index, :], y[:train_index]

# split validation data
X_validation, y_validation = X[train_index:, :], y[train_index:]<jupyter_output><empty_output><jupyter_text>Check DimensionsWe can check the dimensions and proportions of our training and validation dataets.<jupyter_code># check dimensions
print('X_train: ', X_train.shape, X_train.dtype, 'y_train: ', y_train.shape, y_train.dtype)
print('X_validation', X_validation.shape, X_validation.dtype, 'y_validation: ', y_validation.shape, y_validation.dtype)

# check the proportions
total = X_train.shape[0] + X_validation.shape[0]
print('X_train proportion:', X_train.shape[0] / total)
print('X_validation proportion:', X_validation.shape[0] / total)<jupyter_output>X_train:  (800000, 100) float32 y_train:  (800000,) float32
X_validation (200000, 100) float32 y_validation:  (200000,) float32
X_train proportion: 0.8
X_validation proportion: 0.2<jupyter_text>Distribute Data using Dask cuDFNext, let's distribute our data across multiple GPUs using Dask cuDF.<jupyter_code># create Pandas DataFrames for X_train and X_validation
n_columns = X_train.shape[1] # shape = (n_row, n_col)
X_train_pdf = pd.DataFrame(X_train)
# rename columns
X_train_pdf.columns = ['feature_' + str(i) for i in range(n_columns)]

X_validation_pdf = pd.DataFrame(X_validation)
# rename columns
X_validation_pdf.columns = ['feature_' + str(i) for i in range(n_columns)]


# create Pandas DataFrames for y_train and y_validation
y_train_pdf = pd.DataFrame(y_train)
# name the only column as y
y_train_pdf.columns = ['y']

y_validation_pdf = pd.DataFrame(y_validation)
# name the only column as y
y_validation_pdf.columns = ['y']
# Dask settings
npartitions = 8

# X
# create Dask DataFrames for X_train and X_validation
X_train_dask_pdf = dask.dataframe.from_pandas(X_train_pdf, npartitions=npartitions)
X_validation_dask_pdf = dask.dataframe.from_pandas(X_validation_pdf, npartitions=npartitions)

# create Dask cuDF DataFrames for X_train and X_validation
X_train_dask_cudf = dask_cudf.from_dask_dataframe(X_train_dask_pdf)
X_validation_dask_cudf = dask_cudf.from_dask_dataframe(X_validation_dask_pdf)


# y
# create Dask DataFrames for y_train and y_validation
y_train_dask_pdf = dask.dataframe.from_pandas(y_train_pdf, npartitions=npartitions)
y_validation_dask_pdf = dask.dataframe.from_pandas(y_validation_pdf, npartitions=npartitions)

# create Dask cuDF DataFrames for y_train and y_validation
y_train_dask_cudf = dask_cudf.from_dask_dataframe(y_train_dask_pdf)
y_validation_dask_cudf = dask_cudf.from_dask_dataframe(y_validation_dask_pdf)
# Optional: persist training and validation data into memory
X_train_dask_cudf = X_train_dask_cudf.persist()
X_validation_dask_cudf = X_validation_dask_cudf.persist()

y_train_dask_cudf = y_train_dask_cudf.persist()
y_validation_dask_cudf = y_validation_dask_cudf.persist()<jupyter_output><empty_output><jupyter_text>Set ParametersThere are a number of parameters that can be set before XGBoost can be run. * General parameters relate to which booster we are using to do boosting, commonly tree or linear model* Booster parameters depend on which booster you have chosen* Learning task parameters decide on the learning scenario. For example, regression tasks may use different parameters with ranking tasks.<jupyter_code># instantiate params
params = {}

# general params
general_params = {'silent': 1}
params.update(general_params)

# booster params
n_gpus = 1  
booster_params = {}
booster_params['max_depth'] = 8
booster_params['grow_policy'] = 'lossguide'
booster_params['max_leaves'] = 2**8
booster_params['tree_method'] = 'gpu_hist'
booster_params['n_gpus'] = 1  # keep this at 1, even if using more than 1 GPU - Dask XGBoost uses 1 GPU per worker
params.update(booster_params)

# learning task params
learning_task_params = {}
if classification:
    learning_task_params['eval_metric'] = 'auc'
    learning_task_params['objective'] = 'binary:logistic'
else:
    learning_task_params['eval_metric'] = 'rmse'
    learning_task_params['objective'] = 'reg:squarederror'
params.update(learning_task_params)
print(params)<jupyter_output>{'silent': 1, 'max_depth': 8, 'grow_policy': 'lossguide', 'max_leaves': 256, 'tree_method': 'gpu_hist', 'n_gpus': 1, 'eval_metric': 'auc', 'objective': 'binary:logistic'}<jupyter_text>Train ModelNow it's time to train our model! We can use the `dask_xgboost.train` function and pass in the parameters, training dataset, the number of boosting iterations, and the list of items to be evaluated during training.<jupyter_code># model training settings
num_round = 100
%%time


bst = dask_xgboost.train(client, params, X_train_dask_cudf, y_train_dask_cudf, num_boost_round=num_round)<jupyter_output><empty_output><jupyter_text>Generate PredictionsWe can generated predictions using the `dask_xgboost.predict` method and then using `dask.dataframe.multi.concat` to concatenate the multiple resulting dataframes together.<jupyter_code>y_predictions = dask_xgboost.predict(client, bst, X_validation_dask_cudf)
y_predictions = dask.dataframe.multi.concat([y_predictions], axis=1)<jupyter_output><empty_output><jupyter_text>Evaluate ModelLastly, we can evaluate our model (depending on classification or regression) and calculate accuracy or rmse, respectively.<jupyter_code>from sklearn.metrics import accuracy_score


if classification:
    thresholded_predictions = (y_predictions[0] > 0.5).compute().to_array() * 1.0
    accuracy = accuracy_score(y_validation, thresholded_predictions)
    print('Accuracy:', accuracy)
else:
    test['squared_error'] = (y_predictions[0] - y_validation_dask_cudf['y'])**2
    rmse = np.sqrt(test.squared_error.mean().compute())
    print('Root Mean Squared Error:', rmse)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11951423785594639, 'fraction_numerical': 0.01867671691792295, 'mean_word_length': 4.164792387543253, 'pattern_counts': {'":': 0, '<': 48, '<?xml version=': 0, '>': 50, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 17, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '179704', 'license': 'MIT', 'n_tokens_mistral': 3997, 'n_tokens_neox': 3739, 'n_words': 1027, 'path': 'Intro Tutorials/08_Introduction_to_Dask_XGBoost.ipynb', 'repo_name': 'Gamers-Blended/NVAITC', 'size': '11948'}