<jupyter_start><jupyter_text>Distributed XGBoost on Dask in the cloudThis is the accompanying notebook to the blog post [XGBoost – frictionless training on datasets too big for the memory](https://coiled.io/blog/xgboost-frictionless-training/).Swap in your dataset, spin up a cluster in 2 minutes and train at any scale!  Cluster setup<jupyter_code>from dask import dataframe as dd
import coiled<jupyter_output><empty_output><jupyter_text>Order the cluster and look at it coming up in your [Coiled dashboard](https://cloud.coiled.io/):<jupyter_code>%%time
cluster = coiled.Cluster(n_workers=12, software="xgboost-on-coiled")<jupyter_output><empty_output><jupyter_text>Connect to the cluster:<jupyter_code>from dask.distributed import Client, progress
client = Client(cluster)
client<jupyter_output><empty_output><jupyter_text>Load the dataset sample<jupyter_code>mortgage_data = dd.read_parquet(
    "s3://coiled-data/mortgage-2000.parq/*", 
    compression="gzip", 
    columns=columns, 
    storage_options={"anon":True}
)

mortgage_data<jupyter_output><empty_output><jupyter_text>Pin the downloaded dataset to memory:_This step reduces waiting times in subsequent steps that trigger computation._<jupyter_code>mortgage_data = mortgage_data.persist()<jupyter_output><empty_output><jupyter_text>Data preprocessing The dataset needs a little work - we need to prepare categorical columns to a format that is supported by XGBoost.The columns we'll be working with:<jupyter_code>columns = [
    "delinquency_12",
    "interest_rate",
    "loan_age",
    "adj_remaining_months_to_maturity",
    "longest_ever_deliquent",
    "orig_channel",
    "num_borrowers",
    "borrower_credit_score",
    "first_home_buyer",
    "loan_purpose",
    "property_type",
    "num_units",
    "occupancy_status",
    "property_state",
    "zip",
    "mortgage_insurance_percent",
    "coborrow_credit_score",
    "relocation_mortgage_indicator",
]
categorical = [
    "orig_channel",
    "occupancy_status",
    "property_state",
    "first_home_buyer",
    "loan_purpose",
    "property_type",
    "zip",
    "relocation_mortgage_indicator",
    "delinquency_12",
]<jupyter_output><empty_output><jupyter_text>Create a column categorizer:<jupyter_code>from dask_ml.preprocessing import Categorizer

ce = Categorizer(columns=categorical)<jupyter_output><empty_output><jupyter_text>Apply column categorizer:<jupyter_code>mortgage_data = ce.fit_transform(mortgage_data)

mortgage_data.dtypes
# https://github.com/dmlc/xgboost/blame/9a0399e8981b2279d921fe2312f7ab1b880fd3c3/python-package/xgboost/dask.py#L227
# Dask categorical columns are not yet available

# the commit is already in master, can be expected in release 1.4.0

# Because this is not possible yet, I will cast to ints
for col in categorical:
    mortgage_data[col] = mortgage_data[col].cat.codes<jupyter_output><empty_output><jupyter_text>Split the dataset before training<jupyter_code>dependent_vars = mortgage_data.columns.difference(["delinquency_12"])
X, y = mortgage_data.iloc[:, dependent_vars], mortgage_data["delinquency_12"]
from dask_ml.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, shuffle=True, random_state=2
)<jupyter_output><empty_output><jupyter_text>Train XGBoost<jupyter_code>import xgboost as xgb<jupyter_output><empty_output><jupyter_text>Prepare distributed DMatrix structures:<jupyter_code>dtrain = xgb.dask.DaskDMatrix(client, X_train, y_train)    
dtest = xgb.dask.DaskDMatrix(client, X_test, y_test)<jupyter_output><empty_output><jupyter_text>Training params:<jupyter_code>params = {
    "max_depth": 8,
    "max_leaves": 2 ** 8,
    "alpha": 0.9,
    "eta": 0.1,
    "gamma": 0.1,
    "learning_rate": 0.1,
    "subsample": 1,
    "reg_lambda": 1,
    "scale_pos_weight": 2,
    "min_child_weight": 30,
    "objective": "binary:logistic",
    "grow_policy": "lossguide",
}<jupyter_output><empty_output><jupyter_text>Run training<jupyter_code>%%time
output = xgb.dask.train(
    client,
    params,
    dtrain,
    num_boost_round=20,
    evals=[
        (dtrain, 'train'), 
        (dtest, 'test')
    ]
)<jupyter_output><empty_output><jupyter_text>Access results<jupyter_code>booster = output['booster']  # booster is the trained model
history = output['history']  # A dictionary containing evaluation 
booster
history<jupyter_output><empty_output><jupyter_text>Close session and set down the cluster<jupyter_code>cluster.close()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10903841883188985, 'fraction_numerical': 0.015545192094159449, 'mean_word_length': 6.048513302034428, 'pattern_counts': {'":': 13, '<': 61, '<?xml version=': 0, '>': 61, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '88699', 'license': 'MIT', 'n_tokens_mistral': 1663, 'n_tokens_neox': 1581, 'n_words': 304, 'path': 'xgboost-mortgage/xgboost-mortgage.ipynb', 'repo_name': 'coiled/blog-notebooks', 'size': '4503'}