<jupyter_start><jupyter_text>Install Rapids First - Note use only T4 or P100 Or P4 GPU which is compatible for RAPIDS.<jupyter_code>!nvidia-smi
! cat /proc/cpuinfo
# Install RAPIDS
!git clone https://github.com/rapidsai/rapidsai-csp-utils.git
!bash rapidsai-csp-utils/colab/rapids-colab.sh

import sys, os

dist_package_index = sys.path.index('/usr/local/lib/python3.6/dist-packages')
sys.path = sys.path[:dist_package_index] + ['/usr/local/lib/python3.6/site-packages'] + sys.path[dist_package_index:]
sys.path
exec(open('rapidsai-csp-utils/colab/update_modules.py').read(), globals())<jupyter_output>Cloning into 'rapidsai-csp-utils'...
remote: Enumerating objects: 44, done.[K
remote: Counting objects: 100% (44/44), done.[K
remote: Compressing objects: 100% (43/43), done.[K
remote: Total 44 (delta 9), reused 5 (delta 0), pack-reused 0[K
Unpacking objects: 100% (44/44), done.
PLEASE READ
********************************************************************************************************
Colab Notebooks Migration Notice:
 
We have changed the location of the Colab script to our new CSP utilties repo.
We are also dropping support for versions below 0.11, as we are on 0.13 nightlies
 
Not all Colab notebooks are updated (like personal Colabs) and/or are using outdated versions of RAPIDS,
so we HIGHLY encourage users to update their scripts and Colab notebooks both with the new link AND with 
to the new API.  Otherwise, your code MAY BREAK
 
This notice will disappear/be updated on our next release.
 
Please enter in the box your desired RAPIDS version (ex: '0.11' or '0.12', bet[...]<jupyter_text>XGBoost 1.0 Features and Intro New features in XGBoost 1.0- XGBoost supports distributed Computing on Kubernetes- We can use Kubernetes now.- All cloud such as AWS, can use xGBoost- Native Dask interface on XGBoost.- Supports Cudf on RAPIDS.- Already it supports numpy, pandas, libsvm format.- Now it supports CUDA Dataframe.- XGBoost works on GPU already. We can use Cuda Dataframe with end to end on GPU.- Lot of GPU additions.- Learning to rank is GPU accelerated.Internally uses lambda mart ranking algorithm. Uses pairwise ranking approach by sampling many pairs.- Scaling for multi-core CPUs is enhanced.  Loading Data<jupyter_code>import cudf
! pip install xgboost
import xgboost as xgb
print(xgb.__version__)
import sys,tempfile, urllib, os
import pandas as pd
from sklearn.model_selection import train_test_split
import numpy as np
from sklearn.datasets import fetch_openml
covtyp = fetch_openml(name='covertype', version=4)
# Predicting the forest cover type using 53 variables.
# Predict one categorical class. 
covtyp.data.shape
np.unique(covtyp.target)
cov_df = pd.DataFrame(data=np.c_[covtyp['data'], covtyp['target']], columns=covtyp['feature_names'] + ['target'])
cov_df.memory_usage().sum()
cov_df.head()
cov_df.target.value_counts()
cov_df.dtypes<jupyter_output><empty_output><jupyter_text>- Convert all objects into float32 format.- Keep the categorical target as int32<jupyter_code>for cols in cov_df.columns:
    cov_df[cols] = cov_df[cols].astype(np.float32)
cov_df.dtypes
cov_df['target'] = cov_df['target'].astype(np.int32)
cov_df.dtypes<jupyter_output><empty_output><jupyter_text>- Keep target variable from 0 - 7 instead of 1 - 8<jupyter_code>cov_df['target'] = cov_df['target'] - 1
cov_df_x = cov_df.drop(['target'], axis=1)
cov_df_y = pd.DataFrame(cov_df['target'])
X_train, X_test, y_train, y_test = train_test_split(cov_df_x, cov_df_y, train_size=0.75, stratify=cov_df_y, random_state=31)<jupyter_output><empty_output><jupyter_text>Data GPU + XGBoost GPU<jupyter_code>X_train_gdf = cudf.DataFrame.from_pandas(X_train)
X_test_gdf = cudf.DataFrame.from_pandas(X_test)
y_train_gdf = cudf.DataFrame.from_pandas(y_train)
y_test_gdf = cudf.DataFrame.from_pandas(y_test)
!nvidia-smi
# New in 1.0 we can directly use the cuda dataframe
dtrain = xgb.DMatrix(X_train_gdf, label=y_train_gdf)
dtest = xgb.DMatrix(X_test_gdf, label=y_test_gdf)
!nvidia-smi
import time
num_rounds = 200
max_depth = 10
params = {
    'colsample_bylevel' : 1,
    'colsample_bytree' : 1,
    'learning_rate' : 0.1,
    'random_state': 31,
    'objective': 'multi:softmax',
    'num_class' : 7
}
params['tree_method'] = 'gpu_hist'
params['grow_policy'] = 'depthwise'
params['max_depth'] = max_depth
params['max_leaves'] = 0
params['verbosity'] = 0
params['gpu_id'] = 0
params['updater'] = 'grow_gpu_hist'
params['predictor'] = 'gpu_predictor'
gpu_result = {}
start_time = time.time()
# Training it
xgb.train(params, dtrain, num_rounds, evals = {(dtest, 'test')}, evals_result=gpu_result, verbose_eval=20)
end_time = time.time()  
print("GPU Time taken to train = %s"%(end_time - start_time))<jupyter_output>[0]	test-merror:0.17838
[20]	test-merror:0.13971
[40]	test-merror:0.12770
[60]	test-merror:0.11508
[80]	test-merror:0.10647
[100]	test-merror:0.10054
[120]	test-merror:0.09466
[140]	test-merror:0.08770
[160]	test-merror:0.08068
[180]	test-merror:0.07638
[199]	test-merror:0.07186
Time taken to train = 41.82584619522095<jupyter_text>Data RAM + XGBoost GPU - NOTE: Not much time taken to move data from RAM to GPU<jupyter_code># Training on GPU but with data on RAM

dtrain_cpu = xgb.DMatrix(X_train, label=y_train)
dtest_cpu = xgb.DMatrix(X_test, label=y_test)
num_rounds = 200
max_depth = 10
params = {
    'colsample_bylevel' : 1,
    'colsample_bytree' : 1,
    'learning_rate' : 0.1,
    'random_state': 31,
    'objective': 'multi:softmax',
    'num_class' : 7
}
params['tree_method'] = 'gpu_hist'
params['grow_policy'] = 'depthwise'
params['max_depth'] = max_depth
params['max_leaves'] = 0
params['verbosity'] = 0
params['gpu_id'] = 0
params['updater'] = 'grow_gpu_hist'
params['predictor'] = 'gpu_predictor'
gpu_result = {}
start_time = time.time()
# Training it
xgb.train(params, dtrain, num_rounds, evals = {(dtest, 'test')}, evals_result=gpu_result, verbose_eval=20)
end_time = time.time()  
print("RAM Time taken to train = %s" %(end_time - start_time))
! nvidia-smi<jupyter_output>Sat Mar 21 10:46:38 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.64.00    Driver Version: 418.67       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla P100-PCIE...  Off  | 00000000:00:04.0 Off |                    0 |
| N/A   40C    P0    40W / 250W |   2433MiB / 16280MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU  [...]<jupyter_text>Data RAM + XGBoost CPU<jupyter_code>dtrain_cpu = xgb.DMatrix(X_train, label=y_train)
dtest_cpu = xgb.DMatrix(X_test, label=y_test)
params = {}
num_rounds = 200
max_depth = 10

params = {
    'colsample_bylevel' : 1,
    'colsample_bytree' : 1,
    'learning_rate' : 0.1,
    'random_state': 31,
    'objective': 'multi:softmax',
    'num_class' : 7
}
params['tree_method'] = 'hist'
params['grow_policy'] = 'depthwise'
params['max_depth'] = max_depth
params['max_leaves'] = 0
params['verbosity'] = 0
# params['gpu_id'] = 0
# params['updater'] = 'hist'
params['predictor'] = 'cpu_predictor'
cpu_result = {}
start_time = time.time()
# Training it
xgb.train(params, dtrain, num_rounds, evals = {(dtest, 'test')}, evals_result=cpu_result, verbose_eval=20)
end_time = time.time()  
print("RAM Time taken to train = %s" %(end_time - start_time))<jupyter_output>[0]	test-merror:0.17823
[20]	test-merror:0.13898
[40]	test-merror:0.12763
[60]	test-merror:0.11415
[80]	test-merror:0.10697
[100]	test-merror:0.10146
[120]	test-merror:0.09487
[140]	test-merror:0.08651
[160]	test-merror:0.08054
[180]	test-merror:0.07401
[199]	test-merror:0.06994
RAM Time taken to train = 314.41554594039917
{'fraction_non_alphanumeric': 0.18066715364012662, 'fraction_numerical': 0.05003652300949598, 'mean_word_length': 3.7733875653689717, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '66601', 'license': 'MIT', 'n_tokens_mistral': 3216, 'n_tokens_neox': 2888, 'n_words': 712, 'path': 'Learning_XGBoost/XGBoost1_0_CuDF.ipynb', 'repo_name': 'oke-aditya/Machine_Learning', 'size': '8214'}