<jupyter_start><jupyter_text>StructureBoost Quick Config - Regression Predicting temperature in California counties This is intended to show the quickest way to configure and run StructureBoost for a regression problem  Imports<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline

from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split
import copy

import structureboost as stb

pd.set_option("display.max_rows",999)
pd.set_option("display.max_columns",999)<jupyter_output><empty_output><jupyter_text>Load and Process Data<jupyter_code>df_ca_TEMP = pd.read_csv('data/CA_County_TEMP.csv')
df_ca_TEMP.sample(5)
## We'll use just two features, county and month
## Target is maximum temperature
X = df_ca_TEMP.loc[:,['county','month']]
y = df_ca_TEMP.TMAX.values
X_trva, X_test, y_trva, y_test = train_test_split(X,y, test_size = 70000)
X_train_big, X_valid, y_train_big, y_valid = train_test_split(X_trva, y_trva, test_size = 10000)
num_train_pts = 1000
X_train = X_train_big.iloc[:num_train_pts,:]
y_train = y_train_big[:num_train_pts]
default_configs = stb.default_config_dict()
default_configs
feature_configs_default = stb.get_basic_config(X_train, default_configs)
feature_configs_default
stboost_CA_def = stb.StructureBoost(num_trees = 1500,
                                    learning_rate=.02,
                                    feature_configs=feature_configs_default, 
                                    max_depth=2,
                                    mode='regression')
stboost_CA_def.fit(X_train, y_train, 
                   eval_set = ((X_valid, y_valid)), 
                   early_stop_past_steps=1, 
                   eval_freq=20)
pred_temps_stb_def = stboost_CA_def.predict(X_test)
stb_default_loss = mean_squared_error(y_test, pred_temps_stb_def)
stb_default_loss<jupyter_output><empty_output><jupyter_text>Incorporate structure of "County" featureRight now, the county feature is using a *complete* graph, where every vertex is adjacent to every other vertex.  This represents the situation where there is essentially no structure in the different values.<jupyter_code># 1653 edges means that every vertex is adjacent to every other one
default_graph = feature_configs_default['county']['graph']
len(default_graph.vertices), len(default_graph.edges), 58*57/2
# If you look at the edges, you will see there is an edge for every pair of counties
# whether they are adjacent or not
default_graph.edges<jupyter_output><empty_output><jupyter_text>Let's make a new config that uses the actual CA County adjacency graph. This graph is included in the package, so we don't have to build it from scratch<jupyter_code>county_graph = stb.graphs.CA_county_graph()
len(county_graph.vertices), len(county_graph.edges)
# This graph only has edges between adjacent counties
county_graph.edges
# Copy the default configuration, and replace 
# the complete graph (created by the 'get_default_config')
# with the actual CA county graph
feature_configs_1 = copy.deepcopy(feature_configs_default)
feature_configs_1['county']['graph'] = county_graph
feature_configs_1<jupyter_output><empty_output><jupyter_text>Run the model with the new config<jupyter_code>stboost_CA_1 = stb.StructureBoost(num_trees = 1500,
                                  learning_rate=.02,
                                  feature_configs=feature_configs_1, 
                                  max_depth=2,
                                  mode='regression')
stboost_CA_1.fit(X_train, y_train, 
                   eval_set = ((X_valid, y_valid)), 
                   early_stop_past_steps=1, 
                   eval_freq=20)
pred_temps_1 = stboost_CA_1.predict(X_test)
stb_1_loss = mean_squared_error(y_test, pred_temps_1)
stb_1_loss<jupyter_output><empty_output><jupyter_text>Incorporating structure in the 'month' featurePreviously we treated the month as a numerical variable.  However, doing so ignoresthe fact that December is "adjacent" to January in the same way that "July" isadjacent to "August".  To more accurately model the structure, we will represent the monthsby a "cycle" graph<jupyter_code>month_graph = stb.graphs.cycle_int_graph(1,12)
month_graph.vertices, month_graph.edges
# Copy the previous config, and change the settings for month
feature_configs_2 = copy.deepcopy(feature_configs_1)
feature_configs_2['month']['feature_type'] = 'categorical_int'
feature_configs_2['month']['graph'] = month_graph
feature_configs_2
default_configs
feature_configs_2 = stb.apply_defaults(feature_configs_2, default_configs)
feature_configs_2
    
stboost_CA_2 = stb.StructureBoost(num_trees = 3000,
                                  learning_rate=.02,
                                  feature_configs=feature_configs_2, 
                                  max_depth=2,
                                  mode='regression')
stboost_CA_2.fit(X_train, y_train, 
                   eval_set = ((X_valid, y_valid)), 
                   early_stop_past_steps=1, 
                   eval_freq=20)
pred_temps_2 = stboost_CA_2.predict(X_test)
stb_2_loss = mean_squared_error(y_test, pred_temps_2)
stb_2_loss
import catboost as cb
cat_features = ['county']
cb1 = cb.CatBoostRegressor(iterations=1500, early_stopping_rounds=20, max_depth=2,
                             learning_rate=.02, cat_features=cat_features)
cb1.fit(X_train, y_train, cat_features, eval_set=(X_valid, y_valid), verbose=25)
pred_temps_cb1 = cb1.predict(X_test)
cb1_loss = mean_squared_error(y_test, pred_temps_cb1)
cb1_loss
cat_features_2 = ['county', 'month']
cb2 = cb.CatBoostRegressor(iterations=1500, early_stopping_rounds=20, max_depth=2,
                             learning_rate=.02, cat_features=cat_features_2)
cb2.fit(X_train, y_train, cat_features_2, eval_set=(X_valid, y_valid), verbose=25)
pred_temps_cb2 = cb2.predict(X_test)
cb2_loss = mean_squared_error(y_test, pred_temps_cb2)
cb2_loss<jupyter_output><empty_output><jupyter_text>Summary of results<jupyter_code>print('StructureBoost (w no Structure):             loss = {}'.format(np.round(stb_default_loss, 5)))
print('StructureBoost w County Structure:           loss = {}'.format(np.round(stb_1_loss,5)))
print('StructureBoost w County+Month Structure:     loss = {}'.format(np.round(stb_2_loss,5)))
print('CatBoost:  (County categorical)              loss = {}'.format(np.round(cb1_loss,5)))
print('CatBoost:  (County, Month categorical)       loss = {}'.format(np.round(cb2_loss,5)))<jupyter_output>StructureBoost (w no Structure):             loss = 117.00744
StructureBoost w County Structure:           loss = 114.44594
StructureBoost w County+Month Structure:     loss = 111.40926
CatBoost:  (County categorical)              loss = 123.57993
CatBoost:  (County, Month categorical)       loss = 125.79488<jupyter_text>Repeat multiple trials to show results aren't noiseIn case you think the above results are just noise, let's run multiple trials and compare.<jupyter_code>num_train_pts = 1000
num_trials = 10
stb_def_loss_vec = np.zeros(num_trials)
stb_1_loss_vec = np.zeros(num_trials)
stb_2_loss_vec = np.zeros(num_trials)
cb1_loss_vec = np.zeros(num_trials)
cb2_loss_vec = np.zeros(num_trials)
for i in range(num_trials):
    X_trva, X_test, y_trva, y_test = train_test_split(X,y, test_size = 70000, random_state=i)
    X_train_big, X_valid, y_train_big, y_valid = train_test_split(X_trva, y_trva, 
                                                                  test_size = 10000, random_state=i)
    X_train = X_train_big.iloc[:num_train_pts,:]
    y_train = y_train_big[:num_train_pts]
    

    stboost_CA_def = stb.StructureBoost(num_trees = 2500,
                                        learning_rate=.02,
                                        feature_configs=feature_configs_default, 
                                        max_depth=2,
                                        mode='regression',
                                        random_seed=i)
    stboost_CA_def.fit(X_train, y_train, 
                       eval_set = ((X_valid, y_valid)), 
                       early_stop_past_steps=1, 
                       eval_freq=20)

    pred_temps_def = stboost_CA_def.predict(X_test)
    stb_def_loss_vec[i] = mean_squared_error(y_test, pred_temps_def)

    stboost_CA_1 = stb.StructureBoost(num_trees = 2500,
                                      learning_rate=.02,
                                      feature_configs=feature_configs_1, 
                                      max_depth=2,
                                      mode='regression',
                                        random_seed=i)
    stboost_CA_1.fit(X_train, y_train, 
                       eval_set = ((X_valid, y_valid)), 
                       early_stop_past_steps=1, 
                       eval_freq=20)
    pred_temps_1 = stboost_CA_1.predict(X_test)
    stb_1_loss_vec[i] = mean_squared_error(y_test, pred_temps_1)
        

    stboost_CA_2 = stb.StructureBoost(num_trees = 2500,
                                      learning_rate=.02,
                                      feature_configs=feature_configs_2, 
                                      max_depth=2,
                                      mode='regression',
                                        random_seed=i)
    stboost_CA_2.fit(X_train, y_train, 
                       eval_set = ((X_valid, y_valid)), 
                       early_stop_past_steps=1, 
                       eval_freq=20)
    pred_temps_2 = stboost_CA_2.predict(X_test)
    stb_2_loss_vec[i] = mean_squared_error(y_test, pred_temps_2)

    
    cat_features_1 = ['county']
    cb1 = cb.CatBoostRegressor(iterations=2500, early_stopping_rounds=20,
                                max_depth=2,learning_rate=.02,
                                cat_features=cat_features_1, random_seed=i)
    cb1.fit(X_train, y_train, cat_features_1, eval_set=(X_valid, y_valid), verbose=25)
    pred_temps_cb1 = cb1.predict(X_test)
    cb1_loss_vec = mean_squared_error(y_test, pred_temps_cb1)
    
    
    cat_features_2 = ['county', 'month']
    cb2 = cb.CatBoostRegressor(iterations=2500, early_stopping_rounds=20,
                                max_depth=2,learning_rate=.02,
                                cat_features=cat_features_2, random_seed=i)

    cb2.fit(X_train, y_train, cat_features_2, eval_set=(X_valid, y_valid), verbose=25)
    pred_temps_cb2 = cb2.predict(X_test)
    cb2_loss_vec[i] = mean_squared_error(y_test, pred_temps_cb2)<jupyter_output>i=0, eval_set_mse = 287.5330526
i=20, eval_set_mse = 221.36355309372655
i=40, eval_set_mse = 187.15124246730008
i=60, eval_set_mse = 168.70139904081816
i=80, eval_set_mse = 158.13729689864496
i=100, eval_set_mse = 151.73155900708042
i=120, eval_set_mse = 146.11520494920347
i=140, eval_set_mse = 142.7596311263404
i=160, eval_set_mse = 139.7020594934463
i=180, eval_set_mse = 137.29698832615915
i=200, eval_set_mse = 134.97572794952603
i=220, eval_set_mse = 132.90427455646326
i=240, eval_set_mse = 131.0483522424845
i=260, eval_set_mse = 129.72632039405877
i=280, eval_set_mse = 128.36498732056234
i=300, eval_set_mse = 127.34410542839761
i=320, eval_set_mse = 126.60119451947244
i=340, eval_set_mse = 125.84256673880445
i=360, eval_set_mse = 125.05767219956273
i=380, eval_set_mse = 124.415584409826
i=400, eval_set_mse = 124.19087978563323
i=420, eval_set_mse = 123.57438634867817
i=440, eval_set_mse = 123.44686373133122
i=460, eval_set_mse = 123.25099983576902
i=480, eval_set_mse = 122.90073776[...]<jupyter_text>Summary of results (multiple trials)<jupyter_code>print('StructureBoost w County+Month Structure:     avg_loss = {}'.format(np.round(np.mean(stb_2_loss_vec),5)))
print('StructureBoost w County Structure:           avg_loss = {}'.format(np.round(np.mean(stb_1_loss_vec),5)))
print('StructureBoost (w no Structure):             avg_loss = {}'.format(np.round(np.mean(stb_def_loss_vec), 5)))
print('CatBoost:  (County categorical)              avg_loss = {}'.format(np.round(np.mean(cb1_loss_vec),5)))
print('CatBoost:  (County, Month categorical)       avg_loss = {}'.format(np.round(np.mean(cb2_loss_vec),5)))
## difference in loss fn for each different split
diff_vec = cb1_loss_vec-stb_2_loss_vec
diff_vec
## mean and std deviation of loss_vec
np.mean(diff_vec), np.std(diff_vec)
# min and max of discrepancy
np.min(diff_vec), np.max(diff_vec)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08821869204096444, 'fraction_numerical': 0.062011128134827835, 'mean_word_length': 3.118897376286948, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 35, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 29, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '453941', 'license': 'MIT', 'n_tokens_mistral': 4751, 'n_tokens_neox': 4127, 'n_words': 780, 'path': 'examples/CA_County_Regression.ipynb', 'repo_name': 'nerubenstein/structureboost', 'size': '12401'}