<jupyter_start><jupyter_text>PyTorch: Cross Validation Using the new initialization method, I'll give CV another shot.  Setup<jupyter_code>import audiomod
import ptmod

import pandas as pd
import numpy as np

import torch
import torch.nn as nn
import torch.optim as optim

from functools import partial

import matplotlib.pyplot as plt
# import seaborn as sns
# plt.style.use('seaborn')

from copy import deepcopy
import pickle

%matplotlib inline
sax1203_dg = audiomod.pull_datagroup_from_db('sax1203')
params = {
    'c1': [5,2,10],
    'p1': [2,2],
    'c2': [5,2,20],
    'p2': [2,2],
    'f1': [700,100],
    'f2': [100,2]
}<jupyter_output><empty_output><jupyter_text>Off to the races... A couple things to try:1. set `nesterov=True` on SGD2. `optim.RMSprop(params, lr=0.01, alpha=0.99, eps=1e-08, weight_decay=0, momentum=0, centered=False)`3. `optim.Adadelta(params, lr=1.0, rho=0.9, eps=1e-06, weight_decay=0)` -- similar to RMSprop, learning rate is automatic4. `optim.Adam(params, lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0)`<jupyter_code>cv_scores, scores_bundle, losses = ptmod.crossval(
    params,
    sax1203_dg,
    0.25,
    partial(optim.Adam, lr=0.0002, betas=(0.9, 0.999), eps=1e-08, weight_decay=0.01),
    nn.CrossEntropyLoss(),
    12,
    16,
    folds=6,
    rs=42
)<jupyter_output>*** Fold 1 ***

Train set length: 765
Test set length: 155
Random seed: 42

Epoch 1
 * Avg loss: 0.682	Time: 8914.838 ms
 * Weights updated: True
Epoch 2
 * Avg loss: 0.632	Time: 10561.019 ms
 * Weights updated: True
Epoch 3
 * Avg loss: 0.582	Time: 9439.656 ms
 * Weights updated: True
Epoch 4
 * Avg loss: 0.539	Time: 8879.065 ms
 * Weights updated: True
Epoch 5
 * Avg loss: 0.503	Time: 8860.816 ms
 * Weights updated: True
Epoch 6
 * Avg loss: 0.481	Time: 9003.121 ms
 * Weights updated: True
Epoch 7
 * Avg loss: 0.458	Time: 8822.594 ms
 * Weights updated: True
Epoch 8
 * Avg loss: 0.434	Time: 8813.453 ms
 * Weights updated: True
Epoch 9
 * Avg loss: 0.420	Time: 8804.404 ms
 * Weights updated: True
Epoch 10
 * Avg loss: 0.418	Time: 8732.613 ms
 * Weights updated: True
Epoch 11
 * Avg loss: 0.405	Time: 8708.197 ms
 * Weights updated: True
Epoch 12
 * Avg loss: 0.394	Time: 9932.940 ms
 * Weights updated: True

Training Complete

Score	Train	Test
------------------------
Acc	0.837	0.845
[...]<jupyter_text>Settings that were alright:* partial(optim.SGD, lr=0.01, weight_decay=0.01, momentum=0.7, nesterov=True), 10 epochs, 8 per batch (also with wd=0.02)* partial(optim.SGD, lr=0.01, weight_decay=0.01, momentum=0.8, nesterov=True), 15 epochs, 16 per batch (overfitting, and still so with wd=0.02)* **partial(optim.SGD, lr=0.01, weight_decay=0.03, momentum=0.9, nesterov=True), 13 epochs, 16 per batch*** **partial(optim.Adam, lr=0.0001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0.01), 13-15 epochs, 16 per batch** Best yet!* **partial(optim.Adam, lr=0.0005, betas=(0.9, 0.999), eps=1e-08, weight_decay=0.01), 10-12 epochs, 16 per batch** A little faster than the former* **partial(optim.Adam, lr=0.0002, betas=(0.9, 0.999), eps=1e-08, weight_decay=0.01), 10-12, 16** Perhaps just right???<jupyter_code># # save scores
# cv_171205 = {
#     "avg_scores": cv_scores,
#     "scores_bundle": scores_bundle,
#     "losses": losses
# }

# with open("../data/final_cv_171205-1.p", 'wb') as pf:
#     pickle.dump(cv_171205, pf)
def avg_2darrays(arrays, unbalanced=True):
    """
    Computes the average array of a list or array of 2D arrays.
    ---
    arrays: list or array containing at least two numpy arrays, all
        must be two dimensions (will add more later)
    unbalanced: if True, will trim axes to match (bool)
    """
    
    assert len(arrays) > 0, "Input list/array is empty"
    
    if len(arrays) == 1:
        return(arrays[0])

    if unbalanced:
        # this works for n dim
        min_dims = []
        for dim, _ in enumerate(arrays[0].shape):
            min_dim = arrays[0].shape[dim]
            for arr in arrays[1:]:
                min_dim = min(min_dim, arr.shape[dim])
            min_dims.append(min_dim)
        # this does not
        summed = 0
        for arr in arrays:
            summed += arr[:min_dims[0], :min_dims[1]]
        
    else:            
        summed = 0
        for arr in arrays:
            summed += arr
    
    return summed / len(arrays)
ptmod.plot_loss(avg_2darrays(losses), "Final CV (Adam)")
ptmod.plot_loss(avg_2darrays(losses), "Final CV (Adam)", plot_values='epoch')
def get_stats_per_epoch(num_epochs, kfolds, params, datagroup):
    """
    Brute force way to do this, but I'm not rewriting fit and/or crossval 
    today...
    """
    
    scores_by_epoch = {}
    for epoch in range(1, num_epochs+1):
        print("\n\n" + "*" * 40)
        print("*" + "Round {}".format(epoch).center(38) + "*")
        print("*" * 40 + "\n")
        cv_scores, _, losses = ptmod.crossval(
            params,
            datagroup,
            0.25,
            partial(optim.Adam, lr=0.0002, betas=(0.9, 0.999), eps=1e-08, weight_decay=0.01),
            nn.CrossEntropyLoss(),
            epoch,
            16,
            folds=kfolds,
            rs=42
        )
        scores_by_epoch[epoch] = {'scores': cv_scores, 'loss': losses}
        
    return scores_by_epoch
test_scores = get_stats_per_epoch(20, 5, params, sax1203_dg)
# test_scores_20 = test_scores

# with open("../data/cv20_scores_by_epoch.p", 'wb') as pf:
#     pickle.dump(test_scores_20, pf)
# with open("../data/cv20_scores_by_epoch.p", 'rb') as pf:
#     test_scores = pickle.load(pf)
test_scores[14] = {'scores': cv_scores, 'loss': losses}
test_scores[3]['scores']
[test_scores[i]['scores']['acc'][0] for i in range(1, len(test_scores)+1)]
x = range(1, len(test_scores)+1)
y_train = [test_scores[i]['scores']['acc'][0] for i in range(1, len(test_scores)+1)]
y_test = [test_scores[i]['scores']['acc'][1] for i in range(1, len(test_scores)+1)]

fig = plt.figure(figsize=(12,8))
plt.plot(x, y_train, "r-", label="In-Sample Accuracy")
plt.plot(x, y_test, "b-", label="Out-of-Sample Accuracy")
plt.grid()
plt.legend(fontsize=12)
plt.xticks([i+1 for i in range(20)])
plt.xlabel("Epochs", fontsize=16)
plt.ylabel("Accuracy", fontsize=16)
# plt.title("In- and Out-of-Sample Accuracy by Epoch Length", fontsize=16)

plt.savefig("../specs/acc_by_epoch.png", dpi=200);<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1529874213836478, 'fraction_numerical': 0.0770440251572327, 'mean_word_length': 3.306702775897089, 'pattern_counts': {'":': 3, '<': 12, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '484662', 'license': 'MIT', 'n_tokens_mistral': 2784, 'n_tokens_neox': 2452, 'n_words': 629, 'path': '5-sax-detector/code/14-pytorch-crossval.ipynb', 'repo_name': 'davidluther/Metis-Projects', 'size': '6360'}