<jupyter_start><jupyter_text>Undertale & Deltarune Soundtrack Generator ---  Table of Contents0. [**Table of Contents**](Table-of-Contents)1. [**Imports**](Imports)2. [**Data Processing**](Data-Processing)    2.1 [Data Loading](Data-Loading)        2.2 [Data Preprocessing](Data-Preprocessing)        2.3 [Dataset & Dataloader Definition](Dataset-&-Dataloader-Definition)    3. [**Model Definition**](Model-Definition)    4. [**Hyperparameters & Instantiation**](Hyperparameters-&-Instantiation)5. [**Training**](Training) ---  Imports[(go to top)](Undertale-&-Deltarune-Soundtrack-Generator)Import required packages<jupyter_code>import os                                         # File handling
import itertools                                  # chain() for merging lists
import random                                     # Shuffling
import collections                                # Useful tools like Counter, OrderedDict
import math                                       # For... math
from decimal import Decimal                       # Scientific notations in string formatting
from time import time                             # For use in progress bar

import tqdm.auto as tqdm                          # Progress bar

from IPython.display import clear_output

import torch                                      # Deep Learning Framework
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.utils.data import Dataset, DataLoader

import matplotlib.pyplot as plt                   # Plotting training progress
from matplotlib.ticker import AutoLocator
%matplotlib inline

fig_bg_color = "lightsteelblue"
plot_bg_color = "slategray"
fontsize = 20<jupyter_output><empty_output><jupyter_text>---  Data Processing[(go to top)](Undertale-&-Deltarune-Soundtrack-Generator)  Data Loading[(go to top)](Undertale-&-Deltarune-Soundtrack-Generator)Read the text files in the target directory.Do some processing to make sure the texts are clean.<jupyter_code>def get_texts(texts_dir):

    if not os.path.isdir(texts_dir):
        raise FileNotFoundError("given text directory not found: {}".format(texts_dir))

    texts = []
    
    for text_path in (file.path for file in os.scandir(texts_dir) if file.is_file() and file.name.endswith(".txt")):
        with open(file=text_path, mode='r', encoding="utf-8") as text_file:
            
            text = text_file.read().strip()

            if not text.replace(' ', '').isdigit():
                raise RuntimeError("one or more characters other than digits and white spaces are detected: {}".format(text_path))

            while "  " in text:
                text = text.replace("  ", ' ')
            
            texts.append((text_path, text))
    
    return dict(texts)


[(os.path.split(text_path)[1], text[:20]) for text_path, text in get_texts("./source/converted_texts").items()]<jupyter_output><empty_output><jupyter_text>Data Preprocessing[(go to top)](Undertale-&-Deltarune-Soundtrack-Generator) Get integers out of the text and make lists of ints.These lists can be used for the input of the models, or be further processed to compress or simplify the sequences.In this notebook, I'll leave the data as it is and do note-by-note. (Similar to Character-By-Character approach)<jupyter_code>def texts_to_intlists(text_list):
    
    intlists = []
    
    for i, text in enumerate(iterable=text_list):
        
        int_strings = text.split(' ')
        
        if not all(int_str.isdigit() for int_str in int_strings):
            raise RuntimeError("non-digit string detected in text {}".format(i))

        ints = [int(int_str) for int_str in int_strings]
        
        intlists.append(ints)
        
    return intlists


print([ints[:10] for ints in texts_to_intlists(get_texts("./source/converted_texts").values())])<jupyter_output>[[42, 46, 49, 53, 0, 42, 46, 49, 53, 0], [73, 89, 0, 73, 89, 0, 73, 89, 0, 73], [39, 51, 0, 39, 51, 0, 39, 51, 0, 39], [48, 0, 48, 0, 48, 0, 48, 0, 48, 0], [39, 0, 39, 0, 39, 0, 39, 0, 39, 0], [30, 0, 30, 0, 30, 0, 30, 0, 30, 0], [48, 55, 0, 48, 55, 0, 48, 55, 0, 48], [27, 39, 0, 27, 39, 0, 27, 39, 0, 27], [61, 64, 71, 75, 0, 61, 64, 71, 75, 0], [77, 0, 77, 0, 77, 0, 77, 0, 77, 0], [74, 0, 74, 0, 74, 0, 74, 0, 74, 0], [32, 36, 39, 68, 0, 32, 36, 39, 68, 0], [62, 0, 62, 0, 62, 0, 0, 0, 0, 65], [0, 0, 0, 62, 0, 62, 0, 62, 0, 62], [49, 0, 49, 0, 49, 0, 49, 0, 49, 0], [31, 43, 0, 31, 43, 0, 31, 43, 0, 31], [24, 31, 0, 24, 31, 0, 24, 31, 0, 24], [45, 57, 0, 45, 57, 0, 45, 57, 0, 45], [39, 0, 39, 0, 39, 0, 39, 0, 39, 0], [46, 0, 46, 0, 46, 0, 46, 0, 46, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [58, 70, 0, 58, 70, 0, 58, 70, 0, 58], [37, 49, 0, 37, 49, 0, 37, 49, 0, 37], [44, 68, 0, 44, 68, 0, 44, 68, 0, 44], [67, 0, 67, 0, 67, 0, 67, 0, 67, 0], [61, 0, 61, 0, 61, 0, 61, 0, 61, 0], [49, 0, 49, 0, [...]<jupyter_text>Dataset & Dataloader Definition[(go to top)](Undertale-&-Deltarune-Soundtrack-Generator) Create a Dataset class from which training data can be sampled.This Dataset should convert the encoded sequence above into tensorsand have a method for shuffling the order of multiple sequences whileleaving the patterns inside of each sequence untouched.<jupyter_code>class UndertaleDeltaruneDataset(Dataset):
    def __init__(self, texts_dir, batch_size=1):
        self.texts = get_texts(texts_dir) # read and get a dictionary of {file_paths: text_contents}
        self.sequences = texts_to_intlists(self.texts.values())

        self.texts_dir = texts_dir
        self.batch_size = batch_size

    def __len__(self):
        return self.batch_size

    def data_len(self):
        return sum([len(sequence) for sequence in self.sequences])

    def __getitem__(self, index):
        shuffled_list = list(itertools.chain(*random.sample(self.sequences, len(self.sequences))))
        inputs = torch.LongTensor(shuffled_list[:-1])
        labels = torch.LongTensor(shuffled_list[1:])
        return inputs, labels<jupyter_output><empty_output><jupyter_text>Create a custom class that loads the data from the dataset above andallows iteration over the dataset, yielding a small sequence batch at a time.<jupyter_code>class UDBatchLoader:
    def __init__(self, ud_dataset, batch_size, sequence_len, drop_last=False, batch_first=True):
        self.ud_dataset = ud_dataset
        self.batch_size = batch_size
        self.sequence_len = sequence_len
        self.drop_last = drop_last
        self.batch_first = batch_first
    
    def __len__(self):
        if self.drop_last:
            return math.floor((self.ud_dataset.data_len() - 1) / self.sequence_len)
        return math.ceil((self.ud_dataset.data_len() - 1) / self.sequence_len)
    
    def generator(self):
        seq_len = self.sequence_len
        n_seq_batches = self.__len__()
        batch_first = self.batch_first
        
        input_batch, target_batch = next(iter(DataLoader(self.ud_dataset, self.batch_size)))
        if not batch_first:
            input_batch = input_batch.transpose(0, 1).contiguous()
            target_batch = target_batch.transpose(0, 1).contiguous()
        
        for start, end in zip(range(0, seq_len * n_seq_batches, seq_len), range(seq_len, (seq_len + 1) * n_seq_batches, seq_len)):
            if batch_first:
                yield (input_batch[:, start:end].contiguous(), target_batch[:, start:end].contiguous())
            else:
                yield (input_batch[start:end], target_batch[start:end])
    
    def __iter__(self):
        return self.generator()<jupyter_output><empty_output><jupyter_text>---  Model Definition[(go to top)](Undertale-&-Deltarune-Soundtrack-Generator)Define the model architectures.<jupyter_code>class UDNet(nn.Module):
    def __init__(self, hidden_size, num_layers, dropout):
        super(UDNet, self).__init__()

        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.dropout = dropout

        self.init_hiddens = nn.Parameter(torch.randn(num_layers, 1, hidden_size))
        self.init_cells   = nn.Parameter(torch.randn(num_layers, 1, hidden_size))

        self.embed = nn.Embedding(num_embeddings=129, embedding_dim=hidden_size)

        self.lstm = nn.LSTM(input_size=hidden_size, hidden_size=hidden_size, num_layers=num_layers, dropout=dropout, batch_first=True)

        self.fc0 = nn.Sequential(
            nn.LayerNorm(hidden_size),
            nn.Dropout(p=dropout),
            nn.Linear(in_features=hidden_size, out_features=256)
        )

        self.fc1 = nn.Sequential(
            nn.ReLU(),
            nn.LayerNorm(256),
            nn.Dropout(p=dropout),
            nn.Linear(in_features=256, out_features=512)
        )

        self.fc2 = nn.Sequential(
            nn.ReLU(),
            nn.LayerNorm(512),
            nn.Dropout(p=dropout),
            nn.Linear(in_features=512, out_features=129)
        )

    def forward(self, x, hiddens=None):
        if hiddens is None:
            hiddens = self.get_init_hiddens(x.size(0))

        x = self.embed(x)

        x, new_hiddens = self.lstm(x, hiddens)

        x = self.fc0(x)
        x = self.fc1(x)
        x = self.fc2(x)

        return x, new_hiddens

    def get_init_hiddens(self, n_batches):
        return [self.init_hiddens.repeat(1, n_batches, 1), self.init_cells.repeat(1, n_batches, 1)]<jupyter_output><empty_output><jupyter_text>---  Training[(go to top)](Undertale-&-Deltarune-Soundtrack-Generator)<jupyter_code>seed                   = 0
batch_size             = 4
sequence_length        = 12800
lr                     = 1e-3
factor                 = 0.5
patience               = 5

n_logs                 = 30

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

random.seed(seed)
torch.manual_seed(seed)

ud_dataset = UndertaleDeltaruneDataset("./source/converted_texts", batch_size)
ud_loader = UDBatchLoader(ud_dataset, batch_size, sequence_length, drop_last=True, batch_first=True)

model = UDNet(hidden_size=256, num_layers=10, dropout=0.2).to(device)

optimizer = optim.Adam(model.parameters(), lr=lr)

lr_scheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer, factor=factor, patience=patience, verbose=True)

print()
print('Data Sequence Total Length:', ud_dataset.data_len())
print()
print(model)
model.train()
logs = {'epoch': [], 'lr':[], 'loss_avg': [], 'acc_1': [], 'acc_5': []}
i_epoch = 0
while True:
    
    hiddens = model.get_init_hiddens(batch_size)

    running_loss = 0
    n_top1_corrects = 0
    n_top5_corrects = 0
    n_instances = 0
    for i, (inputs, labels) in enumerate(ud_loader):
        print("{:d}/{:d}".format(i, len(ud_loader)-1), end='\r')

        inputs = inputs.to(device)
        labels = labels.view(-1).to(device)

        outputs, hidden_states = model(inputs, hiddens)
        outputs = outputs.view(-1, outputs.size(-1))

        hiddens = [hiddens[0].detach(), hiddens[1].detach()]

        loss = F.cross_entropy(outputs, labels)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        n_instances     += labels.size(0)
        running_loss    += loss.item() * labels.size(0)
        top5_match       = outputs.data.topk(k=5, dim=1)[1].eq(labels.unsqueeze(1))
        n_top1_corrects += top5_match[:, 0].sum().item()
        n_top5_corrects += top5_match.sum().item()
        del top5_match

    loss_avg = running_loss / n_instances
    acc_1    = n_top1_corrects / n_instances
    acc_5    = n_top5_corrects / n_instances

    logs['epoch'].append(i_epoch)
    logs['lr'].append(optimizer.param_groups[0]['lr'])
    logs['loss_avg'].append(loss_avg)
    logs['acc_1'].append(acc_1)
    logs['acc_5'].append(acc_5)

    clear_output(wait=True)
    print('\n\n'.join(["Epoch {:d} - LR={:e}\n===============================================\n".format(i_e, lr)
                       + "Average Loss: {:f}\nAverage Top-1 Accuracy: {:f}\nAverage Top-5 Accuracy: {:f}\n".format(l_a, a_1, a_5)
                       + "==============================================="
                       for i_e, lr, l_a, a_1, a_5 in list(zip(*list(logs.values())))[-n_logs:]]), end='\n\n')

    if (i_epoch + 1) % 10 == 0:
        torch.save({'logs': logs, 'state_dict': model.state_dict(), 'optim_dict': optimizer.state_dict(), 'lr_dict': lr_scheduler.state_dict()},
                   "deep/{:d}.pth".format(i_epoch))

    lr_scheduler.step(loss_avg)
    i_epoch += 1<jupyter_output>Epoch 574 - LR=7.812500e-06
===============================================
Average Loss: 0.111057
Average Top-1 Accuracy: 0.963636
Average Top-5 Accuracy: 0.998402
===============================================

Epoch 575 - LR=7.812500e-06
===============================================
Average Loss: 0.110529
Average Top-1 Accuracy: 0.963950
Average Top-5 Accuracy: 0.998429
===============================================

Epoch 576 - LR=3.906250e-06
===============================================
Average Loss: 0.110452
Average Top-1 Accuracy: 0.963755
Average Top-5 Accuracy: 0.998426
===============================================

Epoch 577 - LR=3.906250e-06
===============================================
Average Loss: 0.110985
Average Top-1 Accuracy: 0.963664
Average Top-5 Accuracy: 0.998396
===============================================

Epoch 578 - LR=3.906250e-06
===============================================
Average Loss: 0.110438
Average Top-1 Accuracy: 0.963867
Average Top-[...]
{'fraction_non_alphanumeric': 0.16676606530490531, 'fraction_numerical': 0.053824362606232294, 'mean_word_length': 3.0345864661654134, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 27, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 13, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '18282', 'license': 'MIT', 'n_tokens_mistral': 5112, 'n_tokens_neox': 4398, 'n_words': 1093, 'path': 'Undertale_Deltarune_Soundtrack_Generator-Deep.ipynb', 'repo_name': 'dragonoken/undertale_deltarune_soundtrack_generator', 'size': '13414'}