<jupyter_start><jupyter_text>Try different algorithms for base and meta estimators BackgroundExperiments and testing for the initial model demonstrated best performance with a model based on an `xgboost` regressor. With the change from `XGBRegressor` with bagging to a stacking model, I broke out the Elo model that was embedded in the `xgboost` model, and added another `XGBRegressor` as the meta estimator. As one of the base models, `XGBRegressor` is still likely one of the better options, but I've recently heard of `CatBoost` and `LightGBM` (I'll be using Scikit-learns' `HistGradientBoostingRegressor` instead to avoid adding an extra dependency), and they might perform better. Also, I haven't tested any other algorithms for the meta estimator, so I should try a wide variety for thoroughness. Hypothesis* An alternative flavour of gradient boosting might outperform `XGBRegressor`.* `XGBRegressor` might not be the best algorithm for the meta estimator.  Code Setup<jupyter_code>%load_ext autoreload
%autoreload 2
# %reload_kedro

from functools import partial
from itertools import product

import pandas as pd
import numpy as np
from sklearn.experimental import enable_hist_gradient_boosting
from sklearn.ensemble import GradientBoostingRegressor, HistGradientBoostingRegressor, ExtraTreesRegressor
from sklearn.svm import LinearSVR
from sklearn.linear_model import Ridge, OrthogonalMatchingPursuit, Lasso
from sklearn.neighbors import KNeighborsRegressor
from sklearn.base import clone
from xgboost import XGBRegressor
from catboost import Pool, CatBoostRegressor
import seaborn as sns

from augury import model_tracking
from augury.ml_data import MLData
from augury.ml_estimators import StackingEstimator
from augury.settings import CV_YEAR_RANGE, SEED

from src.model.charts import graph_cv_model_performance
np.random.seed(SEED)
data = MLData(train_year_range=(CV_YEAR_RANGE[1],))
data.data<jupyter_output>2019-12-29 08:35:35,912 - kedro.io.data_catalog - INFO - Loading data from `model_data` (JSONLocalDataSet)...<jupyter_text>Run experiment  Test different algorithms, one level at a timeFor the list of algorithms to try, I'm selecting ones that have performed well while building earlier versions of the model and a few that I haven't tried before. It's not very scientific, but I want to limit how long it takes to run through them all. Also, instead of testing all possible permutations of base and meta estimators, I'll test one level at a time. This allows for testing more algorithms, but risks missing the best overall permutation, which is a tradeoff I'm willing to make to keep from having to run cross-validation for days.<jupyter_code>label = 'algorithm_selection'
default_estimator = StackingEstimator()

estimators = [
    # XGBRegressor is currently the default for both the base & meta estimators,
    # so we don't need to include it in the list of test models.
    # The current default is defined below.
    # XGBRegressor(objective="reg:squarederror", random_state=SEED),
    CatBoostRegressor(random_seed=SEED),
    GradientBoostingRegressor(random_state=SEED),
    HistGradientBoostingRegressor(random_state=SEED),
    ExtraTreesRegressor(random_state=SEED),
    LinearSVR(),
    Ridge(),
    Lasso(),
    OrthogonalMatchingPursuit(),
    KNeighborsRegressor(),
]

models = []

for level in ['pipeline-1', 'meta_regressor']:
    pipeline_steps = 'pipeline__' + level + '__steps'

    for estimator in estimators:
        estimator_name = estimator.__class__.__name__.lower()
        this_estimator = clone(default_estimator)
        this_estimator.get_params().get(pipeline_steps)[-1] = (estimator_name, estimator)
        
        base_estimator_name = estimator_name if level == 'pipeline-1' else 'xgbregressor'
        meta_estimator_name = estimator_name if level == 'meta_regressor' else 'xgbregressor'
        experiment_value = 'base_' + base_estimator_name + '_meta_' + meta_estimator_name
        
        run_info = {'run_tags': {'experiment': label}, 'run_params': {'experiment_value': experiment_value}}

        models.append((this_estimator, data, run_info))
cv_scores = model_tracking.start_run(models, n_jobs=-1)
cv_scores
mean_scores = []

for idx, results in enumerate(cv_scores):
    est_idx = idx % len(estimators)
    est_label = f"{'base' if idx < len(estimators) else 'meta'}_{estimators[est_idx].__class__.__name__}"
    
    mean_scores.append(
        {
            'model': est_label,
            'fit_time': results['fit_time'].mean(),
            'match_accuracy': results['test_match_accuracy'].mean(),
            'mae': abs(results['test_neg_mean_absolute_error'].mean()),
        }
    )

    print('estimator:', est_label)
    print('mean fit time:', results['fit_time'].mean())
    print('mean match accuracy:', results['test_match_accuracy'].mean())
    print('mean MAE:', abs(results['test_neg_mean_absolute_error'].mean()))
    print('')
    
mean_scores_df = pd.DataFrame(mean_scores)
graph_cv_model_performance(mean_scores_df)<jupyter_output>2019-12-29 10:07:50,191 - matplotlib.legend - WARNING - No handles with labels found to put in legend.<jupyter_text>Try full permutations of top models`ExtraTreesRegressor` performed significantly better than other models across both accuracy and MAE, but I want to try a few of the top models in all possible permutations, with a focus on high-accuracy models. This is a change from last season, where I focused more on MAE due to a mis-interpretation of a mathy blog post. Accuracy is where the money is, so I will optimise for that, using MAE is a secondary metric and/or tie-breaker.<jupyter_code>label = 'algorithm_selection'
default_estimator = StackingEstimator()

estimators = [
    XGBRegressor(objective="reg:squarederror", random_state=SEED),
    CatBoostRegressor(random_seed=SEED),
    GradientBoostingRegressor(random_state=SEED),
    ExtraTreesRegressor(random_state=SEED),
]

est_product = list(product(estimators, repeat=2))

models = []

for base_estimator, meta_estimator in est_product:
    this_estimator = clone(default_estimator)

    base_name = base_estimator.__class__.__name__.lower()
    this_estimator.get_params().get('pipeline__pipeline-1__steps')[-1] = (base_name, base_estimator)
    
    meta_name = meta_estimator.__class__.__name__.lower()
    this_estimator.get_params().get('pipeline__meta_regressor__steps')[-1] = (meta_name, meta_estimator)
    
    experiment_value = 'base_' + base_name + '_meta_' + meta_name

    run_info = {'run_tags': {'experiment': label}, 'run_params': {'experiment_value': experiment_value}}

    models.append((this_estimator, data, run_info))
cv_scores = model_tracking.start_run(models, n_jobs=-1)
cv_scores
est_product = list(product(estimators, repeat=2))
mean_scores = []

for idx, results in enumerate(cv_scores):
    est_idx = idx % len(est_product)
    est_label = 'base_' + est_product[est_idx][0].__class__.__name__ + '_meta_' + est_product[est_idx][1].__class__.__name__
    
    mean_scores.append(
        {
            'model': est_label,
            'fit_time': results['fit_time'].mean(),
            'match_accuracy': results['test_match_accuracy'].mean(),
            'mae': abs(results['test_neg_mean_absolute_error'].mean()),
        }
    )

    print('estimator:', est_label)
    print('mean fit time:', results['fit_time'].mean())
    print('mean match accuracy:', results['test_match_accuracy'].mean())
    print('mean MAE:', abs(results['test_neg_mean_absolute_error'].mean()))
    print('')
    
mean_scores_df = pd.DataFrame(mean_scores)
graph_cv_model_performance(mean_scores_df)<jupyter_output>2019-12-29 19:29:07,105 - matplotlib.legend - WARNING - No handles with labels found to put in legend.
{'fraction_non_alphanumeric': 0.09345430978613091, 'fraction_numerical': 0.008425145819831496, 'mean_word_length': 4.908116385911179, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 10, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '99809', 'license': 'MIT', 'n_tokens_mistral': 2482, 'n_tokens_neox': 2335, 'n_words': 675, 'path': 'notebooks/2020_season/2.1-different-algorithms.ipynb', 'repo_name': 'tipresias/augury', 'size': '7715'}