<jupyter_start><jupyter_text>Swish-based classifier using cosine-annealed LR with restarts- Swish activation, 4 layers, 100 neurons per layer- LR using cosine-annealing with restarts and cycle multiplicity of 2- Validation score use ensemble of 10 models weighted by loss  Import modules<jupyter_code>%matplotlib inline
from __future__ import division
import sys
import os
sys.path.append('../')
from Modules.Basics import *
from Modules.Class_Basics import *<jupyter_output>/home/giles/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
/home/giles/anaconda3/lib/python3.6/site-packages/statsmodels/compat/pandas.py:56: FutureWarning: The pandas.core.datetools module is deprecated and will be removed in a future version. Please use the pandas.tseries module instead.
  from pandas.core import datetools
Using TensorFlow backend.<jupyter_text>Options<jupyter_code>with open(dirLoc + 'features.pkl', 'rb') as fin:
    classTrainFeatures = pickle.load(fin)
nSplits = 10
patience = 2
maxEpochs = 200

ensembleSize = 10
ensembleMode = 'loss'

compileArgs = {'loss':'binary_crossentropy', 'optimizer':'adam'}
trainParams = {'epochs' : 1, 'batch_size' : 256, 'verbose' : 0}
modelParams = {'version':'modelSwish', 'nIn':len(classTrainFeatures), 'compileArgs':compileArgs, 'mode':'classifier'}

print ("\nTraining on", len(classTrainFeatures), "features:", [var for var in classTrainFeatures])<jupyter_output>Training on 31 features: ['DER_mass_MMC', 'DER_mass_transverse_met_lep', 'DER_mass_vis', 'DER_pt_h', 'DER_deltaeta_jet_jet', 'DER_mass_jet_jet', 'DER_prodeta_jet_jet', 'DER_deltar_tau_lep', 'DER_pt_tot', 'DER_sum_pt', 'DER_pt_ratio_lep_tau', 'DER_met_phi_centrality', 'DER_lep_eta_centrality', 'PRI_met_pt', 'PRI_met_sumet', 'PRI_jet_num', 'PRI_jet_all_pt', 'PRI_tau_px', 'PRI_tau_py', 'PRI_tau_pz', 'PRI_lep_px', 'PRI_lep_py', 'PRI_lep_pz', 'PRI_jet_leading_px', 'PRI_jet_leading_py', 'PRI_jet_leading_pz', 'PRI_jet_subleading_px', 'PRI_jet_subleading_py', 'PRI_jet_subleading_pz', 'PRI_met_px', 'PRI_met_py']<jupyter_text>Import data<jupyter_code>trainData = BatchYielder(h5py.File(dirLoc + 'train.hdf5', "r+"))<jupyter_output><empty_output><jupyter_text>Determine LR<jupyter_code>lrFinder = batchLRFind(trainData, getModel, modelParams, trainParams,
                       lrBounds=[1e-5,1e-1], trainOnWeights=True, verbose=0)<jupyter_output>2 classes found, running in binary mode<jupyter_text>Train classifier<jupyter_code>results, histories = batchTrainClassifier(trainData, nSplits, getModel, 
                                          {**modelParams, 'compileArgs':{**compileArgs, 'lr':2e-3}},
                                          trainParams, trainOnWeights=True, maxEpochs=maxEpochs,
                                          cosAnnealMult=2, plotLR=1, reduxDecay=1,
                                          patience=patience, verbose=1, amsSize=250000)<jupyter_output>Using cosine annealing
Training using weights
Running fold 1 / 10
2 classes found, running in binary mode

1 New best found: 4.0527690987994985e-05
2 New best found: 3.66067542809836e-05
3 New best found: 3.5452815589883626e-05
5 New best found: 3.455390610186354e-05
6 New best found: 3.420145621707956e-05
7 New best found: 3.41842311220356e-05
10 New best found: 3.4111118152970534e-05
11 New best found: 3.3693034605349074e-05
12 New best found: 3.357997982804192e-05
13 New best found: 3.3474291803674945e-05
14 New best found: 3.328773209968263e-05
15 New best found: 3.325652302482285e-05
23 New best found: 3.325626533009518e-05
24 New best found: 3.313985145475399e-05
27 New best found: 3.296079396240953e-05
28 New best found: 3.2849000832078415e-05
29 New best found: 3.281121221608239e-05
30 New best found: 3.281037384813196e-05
31 New best found: 3.280242065524597e-05
55 New best found: 3.263182843504765e-05
58 New best found: 3.262222260580248e-05
60 New best found: 3.2607069091121[...]<jupyter_text>Comparing to the Swish model trained with LR cycling (Model_3), the cosine annealing and restarting reaches a lower loss (3.23e-5 c.f. 3.26e-5) and a higher AMS (3.71 c.f. 3.57)  Construct ensemble<jupyter_code>with open('train_weights/resultsFile.pkl', 'rb') as fin:   
    results = pickle.load(fin)
ensemble, weights = assembleEnsemble(results, ensembleSize, ensembleMode, compileArgs)<jupyter_output>Choosing ensemble by loss
Model 0 is 6 with loss = 3.113342969518271e-05
Model 1 is 9 with loss = 3.1348538483897614e-05
Model 2 is 3 with loss = 3.172640918201068e-05
Model 3 is 4 with loss = 3.2283093731530244e-05
Model 4 is 1 with loss = 3.252710453896118e-05
Model 5 is 2 with loss = 3.257728842733674e-05
Model 6 is 7 with loss = 3.258183388116807e-05
Model 7 is 0 with loss = 3.259842610052042e-05
Model 8 is 5 with loss = 3.314781957888044e-05
Model 9 is 8 with loss = 3.325023349599883e-05<jupyter_text>Response on validation data<jupyter_code>valData = BatchYielder(h5py.File(dirLoc + 'val.hdf5', "r+"))
batchEnsemblePredict(ensemble, weights, valData, ensembleSize=ensembleSize, verbose=1)
print('Testing ROC AUC: unweighted {}, weighted {}'.format(roc_auc_score(getFeature('targets', valData.source), getFeature('pred', valData.source)),
                                                           roc_auc_score(getFeature('targets', valData.source), getFeature('pred', valData.source), sample_weight=getFeature('weights', valData.source))))
amsScanSlow(convertToDF(valData.source))
%%time
bootstrapMeanAMS(convertToDF(valData.source), N=512)<jupyter_output>50000 candidates loaded

Mean AMS=3.8+-0.2, at mean cut of 0.95+-0.01
Exact mean cut 0.9523912743898109, corresponds to AMS of 3.7242194839980636
CPU times: user 2.26 s, sys: 9.15 s, total: 11.4 s
Wall time: 2min 44s<jupyter_text>The performance differences between cosine annealing on the validation data are more closely tied;  the overall AMS is slightly worse (3.77 c.f. 3.78) but the AMS corresonding to the mean cut is slightly higher (3.724 c.f. 3.721).The performance of the training data showed improvements, as well, so it is likely that the LR schedule works better than a flat LR. It is possbile that the benefits might become more apparent once later methods are included in the training and architecture.  Test scoring<jupyter_code>testData = BatchYielder(h5py.File(dirLoc + 'testing.hdf5', "r+"))
%%time
batchEnsemblePredict(ensemble, weights, testData, ensembleSize=ensembleSize, verbose=1)
scoreTestOD(testData.source, 0.9523912743898109)<jupyter_output>Public:Private AMS: 3.598078076831359 : 3.684632093115734<jupyter_text>Save/Load<jupyter_code>name = "weights/Swish_CLR"
saveEnsemble(name, ensemble, weights, compileArgs, overwrite=True)
ensemble, weights, compileArgs, _, _ = loadEnsemble(name)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10219291959294825, 'fraction_numerical': 0.12082556972910993, 'mean_word_length': 5.197158081705151, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 30, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '181971', 'license': 'MIT', 'n_tokens_mistral': 2978, 'n_tokens_neox': 2441, 'n_words': 631, 'path': 'notebooks/4_Model_Cyclical-LR.ipynb', 'repo_name': 'GilesStrong/HiggsML_Investigations', 'size': '6977'}