<jupyter_start><jupyter_text>Some findings from code:Even for the same hyperparameters, sometimes the model gives nan train and/or validation losses after some epochs. We should re-run those cases to see if nan comes everytime or only some times. I have noticed cases where a perfectly running hyperparameter gives nan in some runs. Simply discarding a HP if it gives nan isn't the best thing we can do.Obviously we should know why sometimes calculations go into nan phase.<jupyter_code># Learn the best dropout HP model again
# parameters used: d=0.05, z=0, lr=0.001, hidden=256, epochs=5
# !python Sentiment_haste.py --model_file=gru_sentiment_haste_d0.05_z0_h256.pt

"""
The output is saved in manual_output_files/d0.05 z0 h256 training output.txt
"""
# Visualize the haste sentiment model
# !python Visualize_sentiment_haste.py --model_file=gru_sentiment_haste_d0.05_z0_h256.pt --visualize_file_suffix=haste_d0.05_z0_h256
# Learn the best dropout HP model again
# parameters used: d=0.05, z=0, lr=0.001, hidden=256, epochs=5
# !python Sentiment_haste.py --model_file=gru_sentiment_haste_d0.1_z0_h256.pt
"""
Output is saved in manual_output_files/d0.1 z0 h256 training output.txt
"""
# Produce loss-vs-epochs for best dropconnect model.
import torch
torch.manual_seed(125)

# !python Sentiment_haste.py --model_file=gru_sentiment_haste_again_d0.05_z0_h256.pt --hidden_units=256 --dropout=0.05 --zoneout=0
# Produce loss-vs-epochs for the baseline model.
# !python Sentiment_haste.py --model_file=gru_sentiment_haste_again_d0.0_z0_h128.pt --hidden_units=128 --dropout=0 --zoneout=0
# Produce loss-vs-epochs for the baseline model.
import torch
torch.manual_seed(11)
# !python Sentiment_haste.py --model_file=gru_sentiment_haste_again_d0_z0.1_h128.pt --hidden_units=128 --dropout=0 --zoneout=0.1
# Adversarial example with dropout
import torch
torch.manual_seed(6)
# !python Sentiment_haste.py --model_file=gru_sentiment_haste_again_d0_z0.1_h128.pt --hidden_units=128 --dropout=0 --zoneout=0.1

!python Sentiment_haste.py --model_file=gru_sentiment_haste_again_d0.05_z0_h256.pt --hidden_units=256 --dropout=0.05 --zoneout=0

"""
Zoneout = 0.1
Test data: loss 1.3650542838232858, acc 0.7142857142857143, precision 0.39361702127659576, recall 0.9736842105263158, f1 0.5606060606060607.
Test data: loss 1.3837055819375175, acc 0.7053571428571429, precision 0.3723404255319149, recall 0.9722222222222222, f1 0.5384615384615384.

DC = 0.05
Test data: loss 0.5370331874915532, acc 0.7946428571428571, precision 0.6276595744680851, recall 0.8805970149253731, f1 0.732919254658385.
Test data: loss 0.5311861549104963, acc 0.7857142857142857, precision 0.6276595744680851, recall 0.855072463768116, f1 0.7239263803680982.
"""
# Adv example without dropout
import torch
torch.manual_seed(5)
!python Sentiment_haste.py --model_file=gru_sentiment_haste_again_d0_z0.1_h128.pt --hidden_units=128 --dropout=0 --zoneout=0.1
"""
Zoneout = 0.1
Test data: loss 1.3526632530348641, acc 0.7098214285714286, precision 0.3829787234042553, recall 0.972972972972973, f1 0.5496183206106869.

DC = 0.05
Test data: loss 0.5440274562154498, acc 0.7901785714285714, precision 0.6382978723404256, recall 0.8571428571428571, f1 0.7317073170731707.
"""
# We will compare baseline and dropout models for their performance on adversarial test input.
torch.manual_seed(14)
!python Sentiment_haste.py --model_file=gru_sentiment_haste_again_d0_z0.1_h128.pt --hidden_units=128 --dropout=0 --zoneout=0.1

!python Sentiment_haste.py --model_file=gru_sentiment_haste_again_d0.05_z0_h256.pt --hidden_units=256 --dropout=0.05 --zoneout=0

!python Sentiment_haste.py --model_file=gru_sentiment_haste_again_d0_z0.pt --hidden_units=128 --dropout=0 --zoneout=0<jupyter_output>model file passed = gru_sentiment_haste_again_d0_z0.1_h128.pt
tune dropconnect = False
cuda
Creating tabular data.
Time to create tabular data: 62 sec.
Building vocabulary from glove embeddings.
Model loaded from file sentiment_models/gru_sentiment_haste_again_d0_z0.1_h128.pt.
Checking performance on Test data.
Test data: loss 1.3526632530348641, acc 0.7098214285714286, precision 0.3829787234042553, recall 0.972972972972973, f1 0.5496183206106869.
model file passed = gru_sentiment_haste_again_d0.05_z0_h256.pt
tune dropconnect = False
cuda
Creating tabular data.
Time to create tabular data: 62 sec.
Building vocabulary from glove embeddings.
Model loaded from file sentiment_models/gru_sentiment_haste_again_d0.05_z0_h256.pt.
Checking performance on Test data.
Test data: loss 0.5440274562154498, acc 0.7901785714285714, precision 0.6382978723404256, recall 0.8571428571428571, f1 0.7317073170731707.
model file passed = gru_sentiment_haste_again_d0_z0.pt
tune dropconnect = False
cuda
Creating[...]
{'fraction_non_alphanumeric': 0.08871989860583017, 'fraction_numerical': 0.19180397127165189, 'mean_word_length': 5.81294964028777, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '476741', 'license': 'MIT', 'n_tokens_mistral': 2307, 'n_tokens_neox': 1781, 'n_words': 421, 'path': 'colab_notebooks/Visualize Haste Sentiment Model.ipynb', 'repo_name': 'Amit-GH/loss-landscape', 'size': '4734'}