<jupyter_start><jupyter_text>Swish-based classifier using cosine-annealed LR with restarts and data augmentation- Swish activation, 4 layers, 100 neurons per layer- LR using cosine-annealing with restarts and cycle multiplicity of 2- Data is augmentaed via phi rotations, and transvers and longitudinal flips- Validation score use ensemble of 10 models weighted by loss  Import modules<jupyter_code>%matplotlib inline
%reload_ext autoreload
%autoreload 2
from __future__ import division
import sys
import os
sys.path.append('../')
from modules.basics import *
from modules.class_basics import *<jupyter_output>/home/giles/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Using TensorFlow backend.<jupyter_text>Options<jupyter_code>with open(DATA_PATH/'feats.pkl', 'rb') as fin:
    train_feats = pickle.load(fin)
    train_feats = [var for x in train_feats for var in train_feats[x]]
n_models = 10
patience = 50
max_epochs = 200

ensemble_size = 10
ensemble_mode = 'loss'

compile_args = {'loss':'binary_crossentropy', 'optimizer':'adam'}
train_params = {'epochs' : 1, 'batch_size' : 256, 'verbose' : 0}
model_params = {'version':'modelSwish', 'n_in':len(train_feats), 'compile_args':compile_args, 'mode':'classifier'}

print ("\nTraining on", len(train_feats), "features:", [var for var in train_feats])<jupyter_output>Training on 30 features: ['DER_mass_MMC', 'DER_mass_transverse_met_lep', 'DER_mass_vis', 'DER_pt_h', 'DER_deltaeta_jet_jet', 'DER_mass_jet_jet', 'DER_prodeta_jet_jet', 'DER_deltar_tau_lep', 'DER_pt_tot', 'DER_sum_pt', 'DER_pt_ratio_lep_tau', 'DER_met_phi_centrality', 'DER_lep_eta_centrality', 'PRI_met_pt', 'PRI_met_sumet', 'PRI_jet_num', 'PRI_jet_all_pt', 'PRI_tau_px', 'PRI_tau_py', 'PRI_tau_pz', 'PRI_lep_px', 'PRI_lep_pz', 'PRI_jet_leading_px', 'PRI_jet_leading_py', 'PRI_jet_leading_pz', 'PRI_jet_subleading_px', 'PRI_jet_subleading_py', 'PRI_jet_subleading_pz', 'PRI_met_px', 'PRI_met_py']<jupyter_text>Import data<jupyter_code>train_fy = FoldYielder(h5py.File(DATA_PATH/'train.hdf5', "r+"))<jupyter_output><empty_output><jupyter_text>Determine LR<jupyter_code>lr_finder = fold_lr_find(train_fy, get_model, model_params, train_params,
                         lr_bounds=[1e-5,1e-1], train_on_weights=True, verbose=0)<jupyter_output>2 classes found, running in binary mode<jupyter_text>Train classifier<jupyter_code>model_gen_params = {**model_params, 'compile_args':{**compile_args, 'lr':2e-3}}
use_callbacks = {'SWA':{'renewal':10, 'sgd_replacement':False, 'start':10},}
ams_args = {'n_total':250000, 'br':10, 'delta_b':0}
plots = ['history', 'lr']
results, histories, cycle_losses = fold_train_model(train_fy, n_models,
                                                    model_gen_params=model_gen_params,
                                                    train_params=train_params, train_on_weights=True,
                                                    use_callbacks=use_callbacks, ams_args=ams_args,
                                                    max_epochs=max_epochs, patience=patience,
                                                    verbose=1, plots=plots)<jupyter_output>Training using weights
Training model 1 / 10
2 classes found, running in binary mode

1 New best found: 3.498952200752683e-05
2 New best found: 3.4219574829330666e-05
3 New best found: 3.299137515714392e-05
5 New best found: 3.293455275270389e-05
6 New best found: 3.270503765234025e-05
8 New best found: 3.2682850883429636e-05
SWA beginning
model is 0 epochs old
10 swa loss 3.25125993025722e-05, default loss 3.25125993025722e-05
10 New best found: 3.25125993025722e-05
model is 1 epochs old
11 swa loss 3.209855660243193e-05, default loss 3.235721848468529e-05
11 New best found: 3.209855660243193e-05
model is 2 epochs old
12 swa loss 3.189059983851621e-05, default loss 3.217259743832983e-05
12 New best found: 3.189059983851621e-05
model is 3 epochs old
13 swa loss 3.175264508608962e-05, default loss 3.1838186692039014e-05
13 New best found: 3.175264508608962e-05
model is 4 epochs old
14 swa loss 3.167532697261777e-05, default loss 3.224740042933263e-05
14 New best found: 3.167532697261777[...]<jupyter_text>Construct ensemble<jupyter_code>with open('train_weights/results_file.pkl', 'rb') as fin:   
    results = pickle.load(fin)
%%capture --no-stdout
ensemble, weights = assemble_ensemble(results, ensemble_size, metric=ensemble_mode)<jupyter_output>Choosing ensemble by loss
Model 0 is 0 with loss = 3.114236718683969e-05
Model 1 is 5 with loss = 3.115701511489559e-05
Model 2 is 3 with loss = 3.1188121470040644e-05
Model 3 is 1 with loss = 3.1221938020921894e-05
Model 4 is 7 with loss = 3.139824844402028e-05
Model 5 is 2 with loss = 3.1718486565078196e-05
Model 6 is 6 with loss = 3.173174906019364e-05
Model 7 is 8 with loss = 3.206437395274399e-05
Model 8 is 4 with loss = 3.222849868218205e-05
Model 9 is 9 with loss = 3.2589466631179676e-05<jupyter_text>Response on validation data without TTA<jupyter_code>val_fy = FoldYielder(h5py.File(DATA_PATH/'val.hdf5', "r+"))
fold_ensemble_predict(ensemble, weights, val_fy, ensemble_size=ensemble_size, verbose=1)
val_df = convert_to_df(val_fy.source)
print('Testing ROC AUC: unweighted {}, weighted {}'.format(roc_auc_score(val_df.gen_target, val_df.pred_class),
                                                           roc_auc_score(val_df.gen_target, val_df.pred_class, sample_weight=val_df.gen_weight)))
%%time
cut_kde = kde_optimise_cut(val_df, top_perc=0.05, br=10, w_factor=250000/50000)
%%time
cut_mean = mean_optimise_cut(val_df, top_perc=0.05, br=10, w_factor=250000/50000)<jupyter_output>Mean cut at 0.95106816 corresponds to AMS of 3.6880655171390773
Maximum AMS for data is 3.9055505909001806 at cut of 0.963705837726593
CPU times: user 87.3 ms, sys: 42.1 ms, total: 129 ms
Wall time: 81.9 ms<jupyter_text>Test scoring - Without TTA<jupyter_code>test_fy = FoldYielder(h5py.File(DATA_PATH/'testing.hdf5', "r+"))
%%time
fold_ensemble_predict(ensemble, weights, test_fy, ensemble_size=ensemble_size, verbose=1)
score_test_data(test_fy.source, cut_kde)
score_test_data(test_fy.source, cut_mean)<jupyter_output>Public:Private AMS: 3.7206105470162196 : 3.7887472000729208<jupyter_text>Save/Load<jupyter_code>name = "weights/Swish_CLR_TTA"
saveEnsemble(name, ensemble, weights, compile_args, overwrite=1)
ensemble, weights, compile_args, _, _ = loadEnsemble(name)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10371819960861056, 'fraction_numerical': 0.12268553364443775, 'mean_word_length': 5.007233273056058, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 30, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '516712', 'license': 'MIT', 'n_tokens_mistral': 2962, 'n_tokens_neox': 2439, 'n_words': 548, 'path': 'notebooks/6_Model_SWA_Data_Fixing-Redux-M.ipynb', 'repo_name': 'GilesStrong/HiggsML_Investigations', 'size': '6643'}