<jupyter_start><jupyter_text>p-value for model selectionThere are some changes we could make to the model selection process, so we will investigate one now.  We do not want to start p-hacking (trying many options until we get good results), so we will stick with our previously selected models.  This investigation is just out of curiosity to see if other methods could be used in future for making better decisions.We will investigate the use of p-values in deciding whether there is a significant difference between the null hypothesis (baseline model) and the alternative hypothesis (either the Average or Ridge model).  If the difference between model predictions are not statistically significant, then it would make sense to stick with the simpler baseline model.  Often a p-value of 0.05 is chosen as a cut-off to identify statitically significant differences.  Anything lower than this is considered statistically significant.References:1. Applying p-value significance testing: https://machinelearningmastery.com/parametric-statistical-significance-tests-in-python/1. Limitations of p-values: https://sphweb.bumc.bu.edu/otlt/MPH-Modules/EP/EP713_RandomError/EP713_RandomError7.html1. p-hacking in the larger scientific community: https://fivethirtyeight.com/features/science-isnt-broken/<jupyter_code># common notebook config
%run notebook-config.ipynb
# we use ttest_rel because we will be comparing predictions made on related samples (cross-validation folds)
#  rather than independent samples (which would require the use of ttest_ind)
from scipy.stats import ttest_rel

import warnings
warnings.filterwarnings('ignore')

models_filepath = "../models"
models_df = pd.read_csv("{}/championship_transformation_selected_models.csv".format(models_filepath))

# Calculate root mean squared error which has the same units as the target variables
models_df["rmse"] = np.sqrt(models_df["mse"])
def calc_p_values(df):
    """ Calculate p-values for each target variable and season
    
    Return pivot table showing test RMSE and p-value per target variable and season,
    and which model was selected as the best by taking the lowest average previous RMSE
    """
    # Make sure we are just using the tuned models
    df = df[(df.tuned_model)]

    # Set order of models (so they appear in this order in the pivot table later)
    ordered_model_names = ["Baseline", "Average", "Ridge"]
    df['model_name'] = pd.Categorical(
            df['model_name'],
            ordered_model_names,
            ordered=True)

    # Preprocessing to gather prior model perforances together for easier access later
    def list_of_previous_values(x):
        return [list(val.values) for val in x.shift(1).expanding()]

    df["previous_rmses"] = df.groupby(["model_name", "target_variable"])["rmse"].transform(list_of_previous_values)

    # Pivot the tuned models so that there is one row per target_variable and season combination
    pivot_df = df.pivot(index=["target_variable", "season"],
                        columns='model_name',
                        values=["best_model", "rmse", "previous_rmses"])

    # Calculate p-value using ttest between the baseline RMSEs and each model RMSEs
    def calc_p_value_with_baseline(x):
        p_values = []
        for model_name in ordered_model_names:
            # TODO: would be interesting to investigate the impact of the "alternative" parameter
            #  e.g. with a value of "less" or "greater" rather than "two-sided"
            #  but this doesn't accept NaN values
            tstat, p_value_with_baseline = ttest_rel(x.Baseline, x[model_name], nan_policy="omit")
            p_values.append(p_value_with_baseline)
        return p_values

    p_values = pivot_df["previous_rmses"].apply(calc_p_value_with_baseline, axis=1, result_type="broadcast")

    # Save p-values as new columns to the pivot table
    pivot_df["p_value_with_baseline", "Baseline"] = p_values.Baseline
    pivot_df["p_value_with_baseline", "Average"] = p_values.Average
    pivot_df["p_value_with_baseline", "Ridge"] = p_values.Ridge


    return pivot_df[["best_model", "rmse", "p_value_with_baseline"]]<jupyter_output><empty_output><jupyter_text>We can start by looking at the p-values for home_corners_against_mean_premier. The previous model selection performed poorly for this value and chose the ridge model for many seasons when the baseline performed better.<jupyter_code>filtered_df = models_df[(models_df.target_variable == "home_corners_against_mean_premier")]

calc_p_values(filtered_df)<jupyter_output><empty_output><jupyter_text>Note that '--' indicates the values used in calculating the p-value were identical.  The baseline column is all '--' because it is being compared with itself.All of the p-values in this case are considerably above 0.05 indicating there is no significant performance difference between the baseline and either the average or ridge models.  Using this method we would have always chosen to stick with the baseline model for this target variable.  In hindsight this would have resulted in better model choices. Now let's look at the p-values for home_shotsOnTarget_for_mean_premier.  In this case the previous model_selection method did well to choose the ridge model over the baseline.<jupyter_code>filtered_df = models_df[(models_df.target_variable == "home_shotsOnTarget_for_mean_premier")]

calc_p_values(filtered_df)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07295504789977893, 'fraction_numerical': 0.003316138540899042, 'mean_word_length': 4.5005065856129685, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '323504', 'license': 'MIT', 'n_tokens_mistral': 1547, 'n_tokens_neox': 1462, 'n_words': 554, 'path': 'notebooks/002b-jah-p-value-experiment.ipynb', 'repo_name': 'JaHicks/premier_league_prediction', 'size': '5428'}