<jupyter_start><jupyter_text>Seemed like we could deal with extra capacity in our network, so these are the results of adding extra MLP layers.Loading the pickle======<jupyter_code>import pylearn2.utils
import pylearn2.config
import theano
import neukrill_net.dense_dataset
import neukrill_net.utils
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt
import holoviews as hl
%load_ext holoviews.ipython
import sklearn.metrics
cd ..
settings = neukrill_net.utils.Settings("settings.json")
run_settings = neukrill_net.utils.load_run_settings(
    "run_settings/alexnet_based_norm_global_8aug_pmlp1.json", settings, force=True)
model = pylearn2.utils.serial.load(run_settings['alt_picklepath'])
def plot_monitor(model,c = 'valid_y_nll'):
    channel = model.monitor.channels[c]
    plt.title(c)
    plt.grid(which="both")
    plt.plot(channel.example_record,channel.val_record)
    return None<jupyter_output><empty_output><jupyter_text>Adding one MLP layer======The following are the final logs from the best pickle file saved.<jupyter_code>%run ~/repos/pylearn2/pylearn2/scripts/print_monitor.py /disk/scratch/neuroglycerin/models/alexnet_based_norm_global_8aug_pmlp1.pkl
plot_monitor(model,c="valid_objective")
plot_monitor(model,c="valid_y_nll")
plot_monitor(model,c="train_objective")
plot_monitor(model,c="train_y_nll")<jupyter_output><empty_output><jupyter_text>A bit of overfitting happening here, but is it more than normal. Looking at the network _before_ adding that layer:<jupyter_code>run_settings = neukrill_net.utils.load_run_settings(
    "run_settings/alexnet_based_norm_global_8aug.json", settings, force=True)
old = pylearn2.utils.serial.load(run_settings['pickle abspath'])
plot_monitor(old,c="train_y_nll")<jupyter_output><empty_output><jupyter_text>It drops faster and ends lower than without the MLP layer. Suspect whatever is happening here, the extra MLP layer is not having much of an effect. Two More MLP Layers=======Also ran a model adding two more MLP layers, tracking the results of this model:<jupyter_code>run_settings = neukrill_net.utils.load_run_settings(
    "run_settings/alexnet_based_norm_global_8aug_pmlp2.json", settings, force=True)
twomlp = pylearn2.utils.serial.load(run_settings['alt_picklepath'])
%run ~/repos/pylearn2/pylearn2/scripts/print_monitor.py /disk/scratch/neuroglycerin/models/alexnet_based_norm_global_8aug_pmlp2.pkl
plot_monitor(twomlp, c="valid_objective")
plot_monitor(twomlp, c="train_objective")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09824281150159744, 'fraction_numerical': 0.00718849840255591, 'mean_word_length': 7.7894736842105265, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '557588', 'license': 'MIT', 'n_tokens_mistral': 888, 'n_tokens_neox': 867, 'n_words': 174, 'path': 'notebooks/model_modifications/Adding MLP Results.ipynb', 'repo_name': 'Neuroglycerin/neukrill-net-work', 'size': '2504'}