<jupyter_start><jupyter_text>JSMP: Ensemble SelectionThis was done with just 1-2 days to go until the deadline, so it was quite rushed and ad hoc, but I feel I achieved sufficient diversity in the ensemble members to build a robust final submission. Only time will tell!<jupyter_code>import os
import json
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import jsmp
jsmp.env_config("config.json")<jupyter_output>Value for DATA_DIR has been set!<jupyter_text>Setup File paths<jupyter_code>comp_dir = os.path.join(os.environ.get("DATA_DIR"), "jane-street-market-prediction")
model_dir = os.path.join(comp_dir, "models_CV_Kaggle")
esmbl_dir = os.path.join(comp_dir, "final_ensemble")<jupyter_output><empty_output><jupyter_text>K-Fold performance statsConcatenate all the K-fold performance summaries of all the models to compare:<jupyter_code>df_list = []
for f in os.listdir(model_dir):
    if f.endswith(".csv"):
        cv_df = pd.read_csv(os.path.join(model_dir, f))
        cv_df.loc[:, 'model'] = f[:16]
        df_list.append(cv_df)
df = pd.concat(df_list)
df<jupyter_output><empty_output><jupyter_text>Performance summary across all foldsFor a given model and action threshold level, across all 5 folds:* Compute the average utility* Compute the standard deviation of utility* Compute the ratio of the 2<jupyter_code>grp_cols = ['model', 'threshold']

# avg utility and its st. dev. accross folds
avg_u_df = df.groupby(grp_cols)[['utility']].mean().rename(columns={'utility': 'u_mean'}).reset_index()
std_u_df = df.groupby(grp_cols)[['utility']].std().rename(columns={'utility': 'u_vol'}).reset_index()
u_df = pd.merge(avg_u_df, std_u_df, on=grp_cols)

# ratio of avg and st.dev
u_df.loc[:, 'ratio'] = u_df['u_mean'] / u_df['u_vol']

# sort and display
u_df = u_df.sort_values("u_mean", ascending=False)
u_df.head(11)<jupyter_output><empty_output><jupyter_text>Selection Top 6 average utility<jupyter_code>e1 = u_df.sort_values("u_mean", ascending=False).groupby('model').head(1).head(6)
e1<jupyter_output><empty_output><jupyter_text>Top 7 ratios* Subject to more than 1500 average utility<jupyter_code>e2 = u_df.sort_values("ratio", ascending=False).groupby('model').head(1).query("u_mean > 1500").head(7)
e2<jupyter_output><empty_output><jupyter_text>Fold championsIgnoring the threshold, I summarised the performance by each fold and picked the top model in each fold (so long the model's average utility was above 1400):<jupyter_code># fold-level average utility 
grp_cols_2 = ['model', 'kfold']
avg_u_df_2 = df.groupby(grp_cols_2)[['utility']].mean().rename(columns={'utility': 'k_mean'}).reset_index()

# overall utility performance for quality control 
avg_u_df_3 = df.groupby('model')[['utility']].mean().rename(columns={'utility': 'u_mean'}).reset_index()
std_u_df_3 = df.groupby('model')[['utility']].std().rename(columns={'utility': 'u_vol'}).reset_index()
avg_u_df_2 = avg_u_df_2.merge(avg_u_df_3, on='model').merge(std_u_df_3, on='model')

# choose the top performer in each fold so long as it meets the 1400 threshold 
e3 = avg_u_df_2.sort_values(['kfold', 'k_mean'], ascending=False).query("u_mean > 1400").groupby('kfold').head(1)
e3<jupyter_output><empty_output><jupyter_text>Final ensembleI combined all the above into one table and assigned weights to each model based on occurence frequency.* Note I was cheeky and gave my favourite model `D20210220_100512` a bit of extra weighting :-)<jupyter_code>e_hi = pd.DataFrame([{'model': 'D20210220_100512'} for i in range(2)])
e_df = pd.concat([e1[['model']], e2[['model']], e3[['model']], e_hi])
e_df = e_df.model.value_counts(normalize=True).to_frame().reset_index()
e_df.columns = ['model', 'weight']
e_df.to_csv(os.path.join(esmbl_dir, "ensemble_weights.csv"), index=False)
e_df<jupyter_output><empty_output><jupyter_text>Appendix Threshold pickingFrom the above, I couldn't really say which action threshold would be best to use, so I came up with a function of the weight. The threshold would be more conservative for higher weights:<jupyter_code>p_int, p_div, p_max = 0.5, 300, 0.95
x = np.arange(0, 175, 0.5)
pwrs = np.arange(0.5, 2.001, 0.125)
fig, ax = plt.subplots(figsize=(10,5))
for p in pwrs:
    p = np.round(p, 2)
    y = p_int + (x/p_div) ** p
    y = np.where(y > p_max, p_max, y)
    ax.plot(x, y, label=str(p))
plt.legend(loc="lower right")
plt.xlabel("weight")
plt.ylabel("probability")
plt.ylim(p_int, 1)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14151154967481497, 'fraction_numerical': 0.02511773940345369, 'mean_word_length': 6.495798319327731, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 39, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '298967', 'license': 'Apache-2.0', 'n_tokens_mistral': 1716, 'n_tokens_neox': 1642, 'n_words': 396, 'path': 'ensemble_selection.ipynb', 'repo_name': 'robbie-manolache/kaggle-jsmp', 'size': '4459'}