<jupyter_start><jupyter_text>Comparison of Scalers * Creating a pipeline for three different scaler options * Robset Scaler has lower variance of metrics values than the two others and also better R2 and squared error values<jupyter_code>scaler_pipelines = {
    "robust_scaler": make_pipeline(
        VarianceThreshold(), RobustScaler(), KNeighborsRegressor()
    ),
    "standard_scaler": make_pipeline(
        VarianceThreshold(), StandardScaler(), KNeighborsRegressor()
    ),
    "min_max_scaler": make_pipeline(
        VarianceThreshold(), MinMaxScaler(), KNeighborsRegressor()
    ),
}


scores = crossvalidate_pipeline_scores(
    X=X_train,
    y=y_train,
    pipelines=scaler_pipelines,
    n_splits=10,
    random_state=random_state,
)

plot_scores(scores=scores)<jupyter_output>Crossvalidating robust_scaler
Crossvalidating standard_scaler
Crossvalidating min_max_scaler<jupyter_text>Optimizing the Hyperparameters for Model Without Feature Selection * Using scoring metric of mean squared error<jupyter_code>grid_search_pipeline = make_pipeline(
    VarianceThreshold(), RobustScaler(), KNeighborsRegressor()
)

parameters = {
    "kneighborsregressor__n_neighbors": list(range(4, 18)),
    "kneighborsregressor__p": list(range(1, 3)),
}
clf = GridSearchCV(
    grid_search_pipeline, parameters, cv=10, n_jobs=3, scoring="neg_mean_squared_error"
)
clf.fit(X_train, y_train)
print("\nBest Parameters:", clf.best_params_)<jupyter_output><empty_output><jupyter_text>Feature Selection * The chosen algorithm for recursive feature elimination and cross-validated selection makes a lot of difference * Extra Trees and Elastic Net hold a very small amount of columns. A bit too small * Random Forest seems better and keeps way more variables<jupyter_code>selector = RFECV(
    ExtraTreeRegressor(random_state=random_state),
    step=1,
    cv=10,
    scoring="neg_mean_squared_error",
    n_jobs=3,
)
selector_extra_trees = selector.fit(X_train, y_train)
columns_to_keep_extra_trees = X_train.columns[selector_extra_trees.support_]
print("Columns to keep with Extra Trees:", columns_to_keep_extra_trees)
print(
    "Proportion of features kept with Extra Trees:",
    round(len(columns_to_keep_extra_trees) * 100 / len(X_train.columns), 1),
    "%",
)

selector = RFECV(
    ElasticNet(random_state=random_state),
    step=1,
    cv=10,
    scoring="neg_mean_squared_error",
    n_jobs=3,
)

selector_glmnet = selector.fit(X_train, y_train)
columns_to_keep_glmnet = X_train.columns[selector_glmnet.support_]
print("Columns to keep with Elastic Net:", columns_to_keep_glmnet)
print(
    "Proportion of features kept with Elastic Net:",
    round(len(columns_to_keep_glmnet) * 100 / len(X_train.columns), 1),
    "%",
)

selector = RFECV(
    RandomForestRegressor(n_estimators=100, random_state=random_state),
    step=1,
    cv=10,
    scoring="neg_mean_squared_error",
    n_jobs=3,
)
selector_rf = selector.fit(X_train, y_train)
columns_to_keep_rf = X_train.columns[selector_rf.support_]
print("Columns to keep with Random Forest:", columns_to_keep_rf)
print(
    "Proportion of features kept with Random Forest:",
    round(len(columns_to_keep_rf) * 100 / len(X_train.columns), 1),
    "%",
)<jupyter_output>Columns to keep with Extra Trees: Index(['iphone', 'samsunggalaxy', 'iphonedispos'], dtype='object')
Proportion of features kept with Extra Trees: 10.7 %
Columns to keep with Elastic Net: Index(['googleperunc'], dtype='object')
Proportion of features kept with Elastic Net: 3.6 %
Columns to keep with Random Forest: Index(['iphone', 'samsunggalaxy', 'ios', 'googleandroid', 'iphonecampos',
       'samsungcampos', 'iphonecamneg', 'iphonecamunc', 'iphonedispos',
       'samsungdispos', 'iphonedisneg', 'samsungdisneg', 'iphonedisunc',
       'samsungdisunc', 'iphoneperpos', 'samsungperpos', 'iphoneperneg',
       'samsungperneg', 'iphoneperunc'],
      dtype='object')
Proportion of features kept with Random Forest: 67.9 %<jupyter_text>Optimizing the Hyperparameters for Model with Feature Selection * Using scoring metric of mean squared error * Hyperparameters almost the same as in the model without feature selection<jupyter_code>grid_search_pipeline = make_pipeline(
    VarianceThreshold(), RobustScaler(), KNeighborsRegressor()
)

parameters = {
    "kneighborsregressor__n_neighbors": list(range(4, 18)),
    "kneighborsregressor__p": list(range(1, 3)),
}
clf = GridSearchCV(
    grid_search_pipeline, parameters, cv=10, n_jobs=3, scoring="neg_mean_squared_error"
)
clf.fit(X_train[columns_to_keep_rf], y_train)
print("\nBest Parameters:", clf.best_params_)<jupyter_output><empty_output><jupyter_text>Performance of the optimized model compared to unoptimized model * Feature selection does not make much difference for crossvalidated results... * but is for the test set we see that the model without feature selection works better<jupyter_code>def drop_preselected_columns(X):
    return X[:, selector_rf.support_]


final_pipelines = {
    "default_model": make_pipeline(
        VarianceThreshold(), RobustScaler(), KNeighborsRegressor()
    ),
    "default_model_feature_selection": make_pipeline(
        VarianceThreshold(),
        RobustScaler(),
        FunctionTransformer(drop_preselected_columns, validate=False),
        KNeighborsRegressor(),
    ),
    "optimized_model": make_pipeline(
        VarianceThreshold(), RobustScaler(), KNeighborsRegressor(n_neighbors=15, p=1)
    ),
    "optimized_model_feature_selection": make_pipeline(
        VarianceThreshold(),
        RobustScaler(),
        FunctionTransformer(drop_preselected_columns, validate=False),
        KNeighborsRegressor(n_neighbors=15, p=1),
    ),
}

scores_optimized = crossvalidate_pipeline_scores(
    X=X_train,
    y=y_train,
    pipelines=final_pipelines,
    n_splits=30,
    random_state=random_state,
)

plot_scores(scores=scores_optimized)

train_and_plot_prediction_metrics(
    X_train=X_train,
    y_train=y_train,
    X_test=X_test,
    y_test=y_test,
    pipelines=final_pipelines,
)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10043017868960953, 'fraction_numerical': 0.010258107213765718, 'mean_word_length': 5.556399132321041, 'pattern_counts': {'":': 11, '<': 19, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '357519', 'license': 'MIT', 'n_tokens_mistral': 2072, 'n_tokens_neox': 1969, 'n_words': 427, 'path': 'notebooks/optimization_knn_iphone.ipynb', 'repo_name': 'TuomoKareoja/phone-sentiment-analysis', 'size': '6044'}