<jupyter_start><jupyter_text>Regression***  Import<jupyter_code>from sklearn.datasets import load_boston
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error
from sklearn.ensemble import ExtraTreesRegressor
from sklearn.ensemble import RandomForestRegressor
from xgboost import XGBRegressor
from vecstack import stacking<jupyter_output><empty_output><jupyter_text>Prepare data<jupyter_code>boston = load_boston()
X, y = boston.data, boston.target

# Make train/test split
# As usual in machine learning task we have X_train, y_train, and X_test
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)<jupyter_output><empty_output><jupyter_text>Initialize 1st level models<jupyter_code># Caution! All models and parameter values are just 
# demonstrational and shouldn't be considered as recommended.
models = [
    ExtraTreesRegressor(random_state=0, n_jobs=-1, 
                        n_estimators=100, max_depth=3),
        
    RandomForestRegressor(random_state=0, n_jobs=-1, 
                          n_estimators=100, max_depth=3),
        
    XGBRegressor(random_state=0, n_jobs=-1, learning_rate=0.1, 
                 n_estimators=100, max_depth=3)
]<jupyter_output><empty_output><jupyter_text>Perform stacking<jupyter_code>S_train, S_test = stacking(models,                     # list of models
                           X_train, y_train, X_test,   # data
                           regression=True,            # regression task (if you need 
                                                       #     classification - set to False)
                           mode='oof_pred_bag',        # mode: oof for train set, predict test 
                                                       #     set in each fold and find mean
                           save_dir=None,              # do not save result and log (to save 
                                                       #     in current dir - set to '.')
                           metric=mean_absolute_error, # metric: callable
                           n_folds=4,                  # number of folds
                           shuffle=True,               # shuffle the data
                           random_state=0,             # ensure reproducibility
                           verbose=2)                  # print all info<jupyter_output>task:       [regression]
metric:     [mean_absolute_error]
mode:       [oof_pred_bag]
n_models:   [3]

model 0:    [ExtraTreesRegressor]
    fold 0: [3.20733439]
    fold 1: [2.87943130]
    fold 2: [2.53026486]
    fold 3: [2.83618694]
    ----
    MEAN:   [2.86330437] + [0.23993093]
    FULL:   [2.86330437]

model 1:    [RandomForestRegressor]
    fold 0: [3.11110485]
    fold 1: [2.78404210]
    fold 2: [2.55707729]
    fold 3: [2.32209992]
    ----
    MEAN:   [2.69358104] + [0.29117900]
    FULL:   [2.69358104]

model 2:    [XGBRegressor]
    fold 0: [2.40318942]
    fold 1: [2.37286943]
    fold 2: [1.89121526]
    fold 3: [1.95382805]
    ----
    MEAN:   [2.15527554] + [0.23404984]
    FULL:   [2.15527554]<jupyter_text>Look at the resultSo now we have OOF from 1st level models and we can build 2nd level model.  But first let's look at the result.  We have three 1st level models, so we expect to get three columns in `S_train` and `S_test`.<jupyter_code>S_train[:5]
S_test[:5]<jupyter_output><empty_output><jupyter_text>Apply 2nd level model<jupyter_code># Initialize 2nd level model
model = XGBRegressor(random_state=0, n_jobs=-1, learning_rate=0.1, 
                     n_estimators=100, max_depth=3)
    
# Fit 2nd level model
model = model.fit(S_train, y_train)

# Predict
y_pred = model.predict(S_test)

# Final prediction score
print('Final prediction score: [%.8f]' % mean_absolute_error(y_test, y_pred))<jupyter_output>Final prediction score: [2.78409081]
{'fraction_non_alphanumeric': 0.08982963345379452, 'fraction_numerical': 0.06659783169850284, 'mean_word_length': 2.2399665551839463, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '620374', 'license': 'MIT', 'n_tokens_mistral': 1424, 'n_tokens_neox': 1248, 'n_words': 303, 'path': 'examples/01_regression.ipynb', 'repo_name': 'stsfk/vecstack', 'size': '3874'}