<jupyter_start><jupyter_text>Blending ensembling - [Predictores base](Predictores-base) - [Separación del 10%](Separación-del-10%) - [Predicción para el Blending](Predicción-para-el-Blending) - [Modelo Blender](Modelo-Blender) - [Predicción del set testing con los modelos base](Predicción-del-set-testing-con-los-modelos-base) - [Predicción del set testing con el blender](Predicción-del-set-testing-con-el-blender)Repito con un super set de entrenamiento: - [Super set de entrenamiento](Super-set-de-entrenamiento) - [Separación del 10% del super-set](Separación-del-10%-del-super-set) - [Predicción para el super Blending](Predicción-para-el-super-Blending) - [Modelo Blender con el super-set](Modelo-Blender-con-el-super-set) - [Predicción del set de testing con los modelos base entrenados con el super-set de entrenamiento](Predicción-del-set-de-testing-con-los-modelos-base-entrenados-con-el-super-set-de-entrenamiento)<jupyter_code>from sklearn.ensemble import RandomForestRegressor, AdaBoostRegressor
from sklearn.metrics import make_scorer, roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import LabelEncoder
from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import KFold
from xgboost import XGBRegressor
import pandas as pd
import os
if '__file__' in locals():
    current_folder = os.path.dirname(os.path.abspath(__file__))
else:
    current_folder = os.getcwd()

set_de_entrenamiento_testing_y_prediccion = '"{}"'.format(os.path.join(
    current_folder,
    '..',
    'Set de entrenamiento, testing y predicción.ipynb'
))
merge_features = '"{}"'.format(os.path.join(current_folder, '..', 'Features', 'Merge features.ipynb'))
predicciones_csv = os.path.join(current_folder, '..', 'predictions.csv')<jupyter_output><empty_output><jupyter_text>Cargo el df con los features.<jupyter_code>pd.options.mode.chained_assignment = None
%run $merge_features
assert(df_features.shape[0] == get_clean_df()['person'].unique().shape[0])<jupyter_output><empty_output><jupyter_text>Cargo el set de entrenamiento.<jupyter_code>%run $set_de_entrenamiento_testing_y_prediccion

labels_with_features = labels.merge(df_features, how='inner', on='person')
train = labels_with_features.drop('label', axis=1)
train_target = labels_with_features['label']<jupyter_output><empty_output><jupyter_text>Predictores baseEn esta sección vamos a preparar los predictores base a utilizar. Estos son los mismos que se encuentran en la carpeta *Algoritmos de ML*.<jupyter_code>base_predictors = []<jupyter_output><empty_output><jupyter_text>AdaBoost<jupyter_code>param = {
    'n_estimators': 128,
    'loss': 'linear',
    'learning_rate': 0.07,
    'base_estimator': DecisionTreeRegressor(criterion='mse', max_depth=4, max_features=None,
            max_leaf_nodes=None, min_impurity_decrease=0.0,
            min_impurity_split=None, min_samples_leaf=1,
            min_samples_split=2, min_weight_fraction_leaf=0.0,
            presort=False, random_state=None, splitter='best')
}

base_predictors.append(AdaBoostRegressor(**param))<jupyter_output><empty_output><jupyter_text>Random Forest<jupyter_code>param = {
    'bootstrap': True,
    'max_depth': 10,
    'max_features': 81,
    'min_samples_leaf': 49,
    'min_samples_split': 8,
    'n_estimators': 56
}

base_predictors.append(RandomForestRegressor(**param))<jupyter_output><empty_output><jupyter_text>DecisionTree<jupyter_code>base_predictors.append(DecisionTreeRegressor(criterion='mse', max_depth=4, max_features=None,
            max_leaf_nodes=None, min_impurity_decrease=0.0,
            min_impurity_split=None, min_samples_leaf=1,
            min_samples_split=2, min_weight_fraction_leaf=0.0,
            presort=False, random_state=None, splitter='best'))<jupyter_output><empty_output><jupyter_text>Xgboost<jupyter_code>param = {
    'objective': 'reg:logistic',
    'colsample_bylevel': 0.6605668347627213,
    'colsample_bytree': 0.5279014819087092,
    'min_child_weight': 4.302125582335056,
    'learning_rate': 0.15803667962605694,
    'max_delta_step': 7.592652591386328,
    'n_estimators': 65,
    'reg_lambda': 1.1181195507921775,
    'max_depth': 9,
    'silent': True,
    'subsample': 0.43744176565530823,
    'reg_alpha': 3.845311207046479,
    'gamma': 6.219264874528072
}

base_predictors.append(XGBRegressor(**param))<jupyter_output><empty_output><jupyter_text>Separación del 10%Separo un 10% del set de entrenamiento para luego entrenar el blending.<jupyter_code>train_90, train_10, train_90_target, train_10_target = train_test_split(train, train_target, test_size=0.1, stratify=train_target)
assert(train_90.shape[0] == train_90_target.shape[0])
assert(train_10.shape[0] == train_10_target.shape[0])<jupyter_output><empty_output><jupyter_text>Predicción para el BlendingAhora vamos a realizar una predicción por cada predictor base del 10% separado, utilizando el 90% del set de entrenamiento (separando este 90% en sets de entrenamiento y validacion para optimizar cada modelo). Estas predicciones las vamos a agregar a una copia del set de entrenamiento (*train_blending_predictions*).**Nota**: falta hacer el cross validation con el 90% para encontrar los parámetros óptimos. Actualmento los parámetros están optimizados para el 100%. Esto implica una filtración.<jupyter_code>train_10_with_base_predictions = train_10.copy()
for predictor in base_predictors:
    train_10_with_base_predictions[predictor.__class__.__name__] = np.nan
%%time
for predictor in base_predictors:
    predictor.fit(train_90, train_90_target) # train
    train_10_with_base_predictions[predictor.__class__.__name__] = predictor.predict(train_10) # predict
train_10_with_base_predictions.head(3)<jupyter_output><empty_output><jupyter_text>Modelo BlenderEntrenamos un modelo blender que use las predicciones de los modelos base (el 10% que separamos) para realizar las predicciones del set de tesing (el que está en kaggle).**Nota**: en este paso podríamos agregar algunos de los features originales.<jupyter_code>blend_train = train_10_with_base_predictions[[predictor.__class__.__name__ for predictor in base_predictors]]
blend_train.head(3)
param = {
    'solver': 'lbfgs'
}
blender = LogisticRegression(**param)
%%time
scores = cross_val_score(blender, blend_train, train_10_target, cv=10, scoring='roc_auc')
print("Accuracy: %0.6f (+/- %0.6f)" % (scores.mean(), scores.std() * 2))
blender.fit(blend_train, train_10_target)<jupyter_output><empty_output><jupyter_text>Predicción del set testing con los modelos base<jupyter_code>testing = labels_to_predict.merge(df_features, how='inner', on='person')
assert(testing.shape[0] == labels_to_predict.shape[0])
testing_with_base_predictions = testing.copy()
for predictor in base_predictors:
    testing_with_base_predictions[predictor.__class__.__name__] = np.nan
%%time
for predictor in base_predictors:
    # como warm_start=False cada vez que llamo fit, el modelo se reinicia
    predictor.fit(train, train_target) # train
    testing_with_base_predictions[predictor.__class__.__name__] = predictor.predict(testing) # predict<jupyter_output>CPU times: user 53.3 s, sys: 133 ms, total: 53.5 s
Wall time: 53.5 s<jupyter_text>Predicción del set testing con el blender<jupyter_code>testing_with_base_predictions_for_blender = testing_with_base_predictions[[predictor.__class__.__name__ for predictor in base_predictors]]
testing_with_base_predictions_for_blender.head(3)<jupyter_output><empty_output><jupyter_text>Predecimos probabilidades.<jupyter_code># esto es muy específico de logistic reggresion
testing_target = pd.DataFrame(data=blender.predict_proba(testing_with_base_predictions_for_blender))[[1]]
testing_target.index = testing_with_base_predictions_for_blender.index
testing_target.index.name = 'person'
testing_target.columns = ['label']<jupyter_output><empty_output><jupyter_text>Vamos a probar prediciendo labels.<jupyter_code>testing_target = pd.DataFrame(data=blender.predict(testing_with_base_predictions_for_blender))
testing_target.index = testing_with_base_predictions_for_blender.index
testing_target.index.name = 'person'
testing_target.columns = ['label']
# si queremos aplicar blending normal:
# testing_target.to_csv(predicciones_csv)<jupyter_output><empty_output><jupyter_text>Super-set de entrenamientoJuntamos el set de testing y el de entrenamiento en un nuevo super-set.<jupyter_code>super_train = train.append(testing)
super_train_target = train_target.append(testing_target['label'])
assert(super_train.shape[0] == super_train_target.shape[0])<jupyter_output><empty_output><jupyter_text>Separación del 10% del super-set<jupyter_code># ya no puedo usar stratify porque en super_train_target tengo 0, 1 y probabilidades
super_train_90, super_train_10, super_train_90_target, super_train_10_target = train_test_split(super_train, super_train_target, test_size=0.1)
assert(super_train_90.shape[0] == super_train_90_target.shape[0])
assert(super_train_10.shape[0] == super_train_10_target.shape[0])<jupyter_output><empty_output><jupyter_text>Predicción para el super BlendingAhora vamos a realizar una predicción por cada predictor base del 10% separado, utilizando el 90% del super-set de entrenamiento (separando este 90% en sets de entrenamiento y validacion para optimizar cada modelo). Estas predicciones las vamos a agregar a una copia del set de entrenamiento (*super_train_blending_predictions*).**Nota**: falta hacer el cross validation con el 90% para encontrar los parámetros óptimos. Actualmento los parámetros están optimizados para el 100% del set de entrenamiento original. Esto implica una filtración.<jupyter_code>super_train_10_with_base_predictions = super_train_10.copy()
for predictor in base_predictors:
    super_train_10_with_base_predictions[predictor.__class__.__name__] = np.nan
%%time
for predictor in base_predictors:
    predictor.fit(super_train_90, super_train_90_target) # train
    super_train_10_with_base_predictions[predictor.__class__.__name__] = predictor.predict(super_train_10) # predict<jupyter_output>CPU times: user 1min 56s, sys: 280 ms, total: 1min 56s
Wall time: 1min 56s<jupyter_text>Modelo Blender con el super-setNecesitamos un modelo en el cual los targets puedan ser floats, por eso no usamos LogisticRegression. En este caso decidimos utilizar xgboost.<jupyter_code>super_blend_train = super_train_10_with_base_predictions[[predictor.__class__.__name__ for predictor in base_predictors]]
super_blend_train.head(3)
param = {
#     'n_estimators': 'lbfgs'
    'solver': 'lbfgs'
}
super_blender = LogisticRegression(**param)
%%time
scores = cross_val_score(super_blender, super_blend_train, super_train_10_target, cv=10, scoring='roc_auc')
print("Accuracy: %0.6f (+/- %0.6f)" % (scores.mean(), scores.std() * 2))
super_blender.fit(super_blend_train, super_train_10_target)<jupyter_output><empty_output><jupyter_text>Predicción del set de testing con los modelos base entrenados con el super-set de entrenamiento<jupyter_code>super_testing_with_base_predictions = testing.copy()
for predictor in base_predictors:
    super_testing_with_base_predictions[predictor.__class__.__name__] = np.nan
%%time
for predictor in base_predictors:
    # como warm_start=False cada vez que llamo fit, el modelo se reinicia
    predictor.fit(super_train, super_train_target) # train
    super_testing_with_base_predictions[predictor.__class__.__name__] = predictor.predict(testing) # predict<jupyter_output>CPU times: user 2min 7s, sys: 365 ms, total: 2min 7s
Wall time: 2min 7s<jupyter_text>Predicción del set testing con el super blender<jupyter_code>testing_with_base_predictions_for_super_blender = super_testing_with_base_predictions[[predictor.__class__.__name__ for predictor in base_predictors]]
testing_with_base_predictions_for_super_blender.head(3)
predictions = super_blender.predict(testing_with_base_predictions_for_super_blender)
predictions = pd.DataFrame(data=predictions, index=testing_with_base_predictions_for_super_blender.index)
predictions.columns = ['label']
predictions.head(3)
# predictions.to_csv(predicciones_csv)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09751700401540604, 'fraction_numerical': 0.03032041301319348, 'mean_word_length': 7.324693042291951, 'pattern_counts': {'":': 0, '<': 82, '<?xml version=': 0, '>': 82, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '172294', 'license': 'MIT', 'n_tokens_mistral': 4443, 'n_tokens_neox': 4148, 'n_words': 832, 'path': 'Ensambles/Blending.ipynb', 'repo_name': 'seblaz/7506-datos-tp2', 'size': '12203'}