<jupyter_start><jupyter_text>Table of Contents1&nbsp;&nbsp;Feature importance from from linear model1.1&nbsp;&nbsp;Train the model and get reference score1.2&nbsp;&nbsp;Get random permutation scores and compute feature importance2&nbsp;&nbsp;With embeddings2.1&nbsp;&nbsp;Train the model2.2&nbsp;&nbsp;Get feature importance3&nbsp;&nbsp;Neural net with hidden layer and embeddings3.1&nbsp;&nbsp;Train the model3.2&nbsp;&nbsp;Evaluate feature importance4&nbsp;&nbsp;Compare feature importances5&nbsp;&nbsp;Why is t2m_std not important?5.1&nbsp;&nbsp;Is there a good correlation between forecast error and raw ansemble spread?5.2&nbsp;&nbsp;What about the correlation between the error and PP spread?5.3&nbsp;&nbsp;Does the NN care at all about the fc spread?  Feature importanceThe skill of our models certainly increases with auxiliary variables. Of course we would like to know which of the 38 variables we added actually helped the predictions.Random permutation importance gives us some insight. For this we randomly permute one feature at a time from the test set and evaluate the decrease in skill.<jupyter_code>%load_ext autoreload
%autoreload 2
%matplotlib inline
from nn_src.imports import *
from nn_src.utils import get_datasets
#DATA_DIR = '/Users/stephanrasp/data/'
# DATA_DIR = '/scratch/srasp/ppnn_data/'
DATA_DIR = '/Volumes/SanDisk/data/ppnn_data/'
aux_train_set, aux_test_set = get_datasets(DATA_DIR, 'aux_15_16.pkl', ['2015-01-01', '2016-01-01'], aux=True)
n_features = aux_train_set.features.shape[1]; n_features<jupyter_output><empty_output><jupyter_text>Feature importance from from linear model  Train the model and get reference score<jupyter_code>fc_aux = build_fc_model(n_features, 2, compile=True, lr=0.02)
fc_aux.summary()
fc_aux.fit(aux_train_set.features, aux_train_set.targets, 1024, 30, verbose=0)
# Get the reference score from the last model we trained
ref_score = fc_aux.evaluate(aux_test_set.features, aux_test_set.targets, 4096, 0); ref_score<jupyter_output><empty_output><jupyter_text>Get random permutation scores and compute feature importance<jupyter_code>def eval_shuf(m, idx, emb=False):
    x_shuf = aux_test_set.features.copy()
    x_shuf[:, idx] = np.random.permutation(x_shuf[:, idx])
    x = [x_shuf, aux_test_set.cont_ids] if emb else x_shuf
    return m.evaluate(x, aux_test_set.targets, 4096, 0)
def perm_imp(m):
    scores = [eval_shuf(m, i) for i in range(len(aux_test_set.feature_names))]
    fimp = np.array(scores) - ref_score
    df = pd.DataFrame(columns=['Feature', 'Importance'])
    df['Feature'] = aux_test_set.feature_names; df['Importance'] = fimp
    return df
fimp_fc_aux = perm_imp(fc_aux)
fig, ax = plt.subplots(figsize=(12, 5))
sns.barplot(data=fimp_fc_aux, y='Importance', x='Feature', ax=ax)
plt.xticks(rotation=90);<jupyter_output><empty_output><jupyter_text>With embeddings  Train the model<jupyter_code>emb_size = 2
max_id = int(np.max([aux_train_set.cont_ids.max(), aux_test_set.cont_ids.max()]))
max_id
fc_aux_emb = build_emb_model(n_features, 2, [], emb_size, max_id, compile=True, lr=0.02)
fc_aux_emb.fit([aux_train_set.features, aux_train_set.cont_ids], aux_train_set.targets, 
               epochs=30, batch_size=1024, verbose=0);
ref_score = fc_aux_emb.evaluate([aux_test_set.features, aux_test_set.cont_ids], aux_test_set.targets, 4096, 0)
ref_score<jupyter_output><empty_output><jupyter_text>Get feature importance<jupyter_code>def perm_imp_emb(m, ref):
    scores = [eval_shuf(m, i, True) for i in range(len(aux_test_set.feature_names))]
    ids_shuf = np.random.permutation(aux_test_set.cont_ids)
    scores += [m.evaluate([aux_test_set.features, ids_shuf], aux_test_set.targets, 4096, 0)]
    fimp = np.array(scores) - ref
    df = pd.DataFrame(columns=['Feature', 'Importance'])
    df['Feature'] = aux_test_set.feature_names + ['Embedding']; df['Importance'] = fimp
    return df
fimp_fc_aux_emb = perm_imp_emb(fc_aux_emb, ref_score)
fig, ax = plt.subplots(figsize=(12, 5))
sns.barplot(data=fimp_fc_aux_emb, y='Importance', x='Feature', ax=ax)
plt.xticks(rotation=90);<jupyter_output><empty_output><jupyter_text>Neural net with hidden layer and embeddingsThis is our best configuration for one year of data.  Train the model<jupyter_code>nn_aux_emb = build_emb_model(n_features, 2, [50], emb_size, max_id, compile=True, lr=0.01)
nn_aux_emb.fit([aux_train_set.features, aux_train_set.cont_ids], aux_train_set.targets, 
               epochs=30, batch_size=1024, verbose=0);
ref_score = nn_aux_emb.evaluate([aux_test_set.features, aux_test_set.cont_ids], aux_test_set.targets, 4096, 0)
ref_score<jupyter_output><empty_output><jupyter_text>Evaluate feature importance<jupyter_code>fimp_nn_aux_emb = perm_imp_emb(nn_aux_emb, ref_score)
fig, ax = plt.subplots(figsize=(12, 5))
sns.barplot(data=fimp_nn_aux_emb, y='Importance', x='Feature', ax=ax)
plt.xticks(rotation=90);<jupyter_output><empty_output><jupyter_text>Compare feature importances<jupyter_code>fimp_fc_aux = fimp_fc_aux.append({'Feature': 'Embedding'}, ignore_index=True)
fimps = [fimp_fc_aux, fimp_fc_aux_emb, fimp_nn_aux_emb]
comb_df = pd.DataFrame(data=fimp_fc_aux['Feature']); comb_df.head()
comb_df['FCN-aux'] = fimp_fc_aux['Importance']
comb_df['FCN-aux-emb'] = fimp_fc_aux_emb['Importance']
comb_df['NN-aux-emb'] = fimp_nn_aux_emb['Importance']
comb_df['Mean importance'] = comb_df.iloc[:, 1:].mean(axis=1)
comb_df.sort_values('Mean importance', ascending=False, inplace=True)
comb_df.head()
len(fimp_fc_aux)
comb_df.sum()
comb_df.iloc[0, 1:] /= 10
comb_df['Feature'].iloc[0] = 't2m_mean / 10'
melt_df = comb_df.iloc[:15, :4].melt(id_vars='Feature', var_name='Experiment', value_name='Importance')
melt_df['Feature'] = [f.replace('_fc_', '_') for f in melt_df['Feature']]
sns.set_style('white')
sns.set_palette(['#1E90FF', '#B22222', '#228B22'])
fig, ax = plt.subplots(figsize=(8, 4))
sns.barplot(data = melt_df, x='Feature', y='Importance', hue='Experiment', ax=ax)
plt.xticks(rotation=90);
sns.despine()
plt.title('Feature importance');
fig.savefig('./feature-importance.pdf', bbox_inches='tight')<jupyter_output><empty_output><jupyter_text>Why is t2m_std not important?  Is there a good correlation between forecast error and raw ansemble spread?<jupyter_code>fc_err = aux_test_set.features[:, 0] * aux_test_set.scale_factors[0] - aux_test_set.targets
ens_spread = aux_test_set.features[:, 1] * aux_test_set.scale_factors[1]
plt.scatter(np.abs(fc_err[::10]), ens_spread[::10], alpha=0.05, s=10);
np.corrcoef(np.abs(fc_err), ens_spread)[0, 1]
np.abs(fc_err).mean(), ens_spread.mean(), ens_spread.mean()/np.abs(fc_err).mean()<jupyter_output><empty_output><jupyter_text>What about the correlation between the error and PP spread?<jupyter_code>preds = nn_aux_emb.predict([aux_test_set.features, aux_test_set.cont_ids], 4096, 0)
pp_spread = np.abs(preds[:, 1])
plt.scatter(np.abs(fc_err[::10]), pp_spread[::10], alpha=0.05, s=10);
np.corrcoef(np.abs(fc_err), pp_spread)[0, 1]
pp_spread.mean()/np.abs(fc_err).mean()<jupyter_output><empty_output><jupyter_text>Does the NN care at all about the fc spread?<jupyter_code>plt.scatter(ens_spread[::10], pp_spread[::10], alpha=0.05, s=10);
np.corrcoef(ens_spread, pp_spread)[0, 1]
np.mean(pp_spread), np.mean(ens_spread)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1386548819223864, 'fraction_numerical': 0.027344289462781383, 'mean_word_length': 7.530035335689046, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '542623', 'license': 'MIT', 'n_tokens_mistral': 2884, 'n_tokens_neox': 2707, 'n_words': 518, 'path': 'nn_postprocessing/notebooks/feature_importance.ipynb', 'repo_name': 'cd663/ppnn', 'size': '7241'}