<jupyter_start><jupyter_text>1 - Simple Sentiment AnalysisWe'll be building a machine learning model to detect sentiment (i.e. detect if a sentence is positive or negative) using PyTorch and TorchText. This will be done on movie reviews, using the [IMDb dataset](http://ai.stanford.edu/~amaas/data/sentiment/).  Preparing DataIn our sentiment classification task the data consists of both the raw string of the review and the sentiment, either "pos" or "neg".<jupyter_code>import torch
from torchtext import data

SEED = 1234

torch.manual_seed(SEED)
torch.backends.cudnn.deterministic = True

TEXT = data.Field(tokenize = 'spacy')
LABEL = data.LabelField(dtype = torch.float)<jupyter_output><empty_output><jupyter_text>The following code automatically downloads the IMDb dataset and splits it into the canonical train/test splits as `torchtext.datasets` objects.<jupyter_code>from torchtext import datasets

train_data, test_data = datasets.IMDB.splits(TEXT, LABEL)
print(f'Number of training examples: {len(train_data)}')
print(f'Number of testing examples: {len(test_data)}')
print(vars(train_data.examples[0]))<jupyter_output>{'text': ['"', 'The', 'Blob', '"', 'qualifies', 'as', 'a', 'cult', 'sci', '-', 'fi', 'film', 'not', 'only', 'because', 'it', 'launched', '27-year', 'old', 'Steve', 'McQueen', 'on', 'a', 'trajectory', 'to', 'superstardom', ',', 'but', 'also', 'because', 'it', 'exploited', 'the', 'popular', 'themes', 'both', 'of', 'alien', 'invasion', 'and', 'teenage', 'delinquency', 'that', 'were', 'inseparable', 'in', 'the', '1950s', '.', 'Interestingly', ',', 'nobody', 'in', 'the', 'Kay', 'Linaker', '&', 'Theodore', 'Simonson', 'screenplay', 'ever', 'refers', 'to', 'the', 'amorphous', ',', 'scarlet', '-', 'red', 'protoplasm', 'that', 'plummeted', 'to', 'Earth', 'in', 'a', 'meteor', 'and', 'menaced', 'everybody', 'in', 'the', 'small', 'town', 'of', 'Downingtown', 'Pennsylvania', 'on', 'a', 'Friday', 'night', 'as', '"', 'The', 'Blob', '.', '"', 'Steve', 'McQueen', 'won', 'the', 'role', 'of', 'Josh', 'Randall', ',', 'the', 'old', 'West', 'bounty', 'hunter', 'in', '"', 'Wanted', ':', 'Dead', 'or', 'Alive'[...]<jupyter_text>By default this splits 70/30.<jupyter_code>import random

train_data, valid_data = train_data.split(random_state=random.seed(SEED))
print(f'Number of training examples: {len(train_data)}')
print(f'Number of validation examples: {len(valid_data)}')
print(f'Number of testing examples: {len(test_data)}')<jupyter_output>Number of training examples: 17500
Number of validation examples: 7500
Number of testing examples: 25000<jupyter_text>Next, we have to build a _vocabulary_. This is a effectively a look up table where every unique word in your data set has a corresponding _index_ (an integer).Will will keep the top 25,000 frequently occuring words.Other words are replaced with a special _unknown_ or `` token.<jupyter_code>MAX_VOCAB_SIZE = 25_000

TEXT.build_vocab(train_data, max_size = MAX_VOCAB_SIZE)
LABEL.build_vocab(train_data)
print(f"Unique tokens in TEXT vocabulary: {len(TEXT.vocab)}")
print(f"Unique tokens in LABEL vocabulary: {len(LABEL.vocab)}")<jupyter_output>Unique tokens in TEXT vocabulary: 25002
Unique tokens in LABEL vocabulary: 2<jupyter_text>The vocab size 25002 because of the addition tokens `` and ``.We can also view the most common words in the vocabulary and their frequencies.<jupyter_code>print(TEXT.vocab.freqs.most_common(20))<jupyter_output>[('the', 202946), (',', 192199), ('.', 165679), ('and', 109101), ('a', 109053), ('of', 100868), ('to', 93689), ('is', 76235), ('in', 61202), ('I', 54694), ('it', 53717), ('that', 49349), ('"', 44319), ("'s", 43396), ('this', 42128), ('-', 37113), ('/><br', 35607), ('was', 34975), ('as', 30510), ('with', 29816)]<jupyter_text>We can also see the vocabulary directly using either the `stoi` (**s**tring **to** **i**nt) or `itos` (**i**nt **to**  **s**tring) method.<jupyter_code>print(TEXT.vocab.itos[:10])<jupyter_output>['<unk>', '<pad>', 'the', ',', '.', 'and', 'a', 'of', 'to', 'is']<jupyter_text>We can also check the labels, ensuring 0 is for negative and 1 is for positive.<jupyter_code>print(LABEL.vocab.stoi)<jupyter_output>defaultdict(<function _default_unk_index at 0x7f6ddd717ae8>, {'neg': 0, 'pos': 1})<jupyter_text>Now we will create the data iterator.<jupyter_code>BATCH_SIZE = 64

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

train_iterator, valid_iterator, test_iterator = data.BucketIterator.splits(
    (train_data, valid_data, test_data), 
    batch_size = BATCH_SIZE,
    device = device)<jupyter_output><empty_output><jupyter_text>Build the Model<jupyter_code>import torch.nn as nn

class RNN(nn.Module):
    def __init__(self, input_dim, embedding_dim, hidden_dim, output_dim):
        
        super().__init__()
        
        self.embedding = nn.Embedding(input_dim, embedding_dim)
        
        self.rnn = nn.RNN(embedding_dim, hidden_dim)
        
        self.fc = nn.Linear(hidden_dim, output_dim)
        
    def forward(self, text):

        #text = [sent len, batch size]
        
        embedded = self.embedding(text)
        
        #embedded = [sent len, batch size, emb dim]
        
        output, hidden = self.rnn(embedded)
        
        #output = [sent len, batch size, hid dim]
        #hidden = [1, batch size, hid dim]
        
        assert torch.equal(output[-1,:,:], hidden.squeeze(0))
        
        return self.fc(hidden.squeeze(0))<jupyter_output><empty_output><jupyter_text>We now create an instance of our RNN class. The input dimension is the dimension of the one-hot vectors, which is equal to the vocabulary size. The embedding dimension is the size of the dense word vectors. This is usually around 50-250 dimensions, but depends on the size of the vocabulary.The hidden dimension is the size of the hidden states. This is usually around 100-500 dimensions, but also depends on factors such as on the vocabulary size, the size of the dense vectors and the complexity of the task.The output dimension is usually the number of classes, however in the case of only 2 classes the output value is between 0 and 1 and thus can be 1-dimensional, i.e. a single scalar real number.<jupyter_code>INPUT_DIM = len(TEXT.vocab)
EMBEDDING_DIM = 100
HIDDEN_DIM = 256
OUTPUT_DIM = 1

model = RNN(INPUT_DIM, EMBEDDING_DIM, HIDDEN_DIM, OUTPUT_DIM)<jupyter_output><empty_output><jupyter_text>Let's also create a function that will tell us how many trainable parameters our model has so we can compare the number of parameters across different models.<jupyter_code>def count_parameters(model):
    return sum(p.numel() for p in model.parameters() if p.requires_grad)

print(f'The model has {count_parameters(model):,} trainable parameters')<jupyter_output>The model has 2,592,105 trainable parameters<jupyter_text>Train the Model Now we'll set up the training and then train the model.First, we'll create an optimizer. Here, we'll use _stochastic gradient descent_ (SGD).<jupyter_code>import torch.optim as optim

optimizer = optim.SGD(model.parameters(), lr=1e-3)<jupyter_output><empty_output><jupyter_text>Next, we'll define our loss function. The loss function here is _binary cross entropy with logits_.<jupyter_code>criterion = nn.BCEWithLogitsLoss()
model = model.to(device)
criterion = criterion.to(device)<jupyter_output><empty_output><jupyter_text>Function for calculating model accuracy.<jupyter_code>def binary_accuracy(preds, y):
    """
    Returns accuracy per batch, i.e. if you get 8/10 right, this returns 0.8, NOT 8
    """

    #round predictions to the closest integer
    rounded_preds = torch.round(torch.sigmoid(preds))
    correct = (rounded_preds == y).float() #convert into float for division 
    acc = correct.sum() / len(correct)
    return acc<jupyter_output><empty_output><jupyter_text>Train the model<jupyter_code>def train(model, iterator, optimizer, criterion):
    
    epoch_loss = 0
    epoch_acc = 0
    
    model.train()
    
    for batch in iterator:
        
        optimizer.zero_grad()
                
        predictions = model(batch.text).squeeze(1)
        
        loss = criterion(predictions, batch.label)
        
        acc = binary_accuracy(predictions, batch.label)
        
        loss.backward()
        
        optimizer.step()
        
        epoch_loss += loss.item()
        epoch_acc += acc.item()
        
    return epoch_loss / len(iterator), epoch_acc / len(iterator)<jupyter_output><empty_output><jupyter_text>Evaluate the model<jupyter_code>def evaluate(model, iterator, criterion):
    
    epoch_loss = 0
    epoch_acc = 0
    
    model.eval()
    
    with torch.no_grad():
    
        for batch in iterator:

            predictions = model(batch.text).squeeze(1)
            
            loss = criterion(predictions, batch.label)
            
            acc = binary_accuracy(predictions, batch.label)

            epoch_loss += loss.item()
            epoch_acc += acc.item()
        
    return epoch_loss / len(iterator), epoch_acc / len(iterator)<jupyter_output><empty_output><jupyter_text>We'll also create a function to tell us how long an epoch takes to compare training times between models.<jupyter_code>import time

def epoch_time(start_time, end_time):
    elapsed_time = end_time - start_time
    elapsed_mins = int(elapsed_time / 60)
    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))
    return elapsed_mins, elapsed_secs
N_EPOCHS = 5

best_valid_loss = float('inf')

for epoch in range(N_EPOCHS):

    start_time = time.time()
    
    train_loss, train_acc = train(model, train_iterator, optimizer, criterion)
    valid_loss, valid_acc = evaluate(model, valid_iterator, criterion)
    
    end_time = time.time()

    epoch_mins, epoch_secs = epoch_time(start_time, end_time)
    
    if valid_loss < best_valid_loss:
        best_valid_loss = valid_loss
        torch.save(model.state_dict(), 'tut1-model.pt')
    
    print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')
    print(f'\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')
    print(f'\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')<jupyter_output>Epoch: 01 | Epoch Time: 0m 15s
	Train Loss: 0.694 | Train Acc: 50.09%
	 Val. Loss: 0.697 |  Val. Acc: 49.54%
Epoch: 02 | Epoch Time: 0m 14s
	Train Loss: 0.693 | Train Acc: 49.60%
	 Val. Loss: 0.697 |  Val. Acc: 49.49%
Epoch: 03 | Epoch Time: 0m 14s
	Train Loss: 0.693 | Train Acc: 50.10%
	 Val. Loss: 0.697 |  Val. Acc: 51.10%
Epoch: 04 | Epoch Time: 0m 14s
	Train Loss: 0.693 | Train Acc: 49.91%
	 Val. Loss: 0.697 |  Val. Acc: 49.46%
Epoch: 05 | Epoch Time: 0m 14s
	Train Loss: 0.693 | Train Acc: 50.13%
	 Val. Loss: 0.697 |  Val. Acc: 50.87%<jupyter_text>The model does not perform well due to the simpleness of model architecture.Finally, the metric we actually care about, the test loss and accuracy, which we get from our parameters that gave us the best validation loss.<jupyter_code>model.load_state_dict(torch.load('tut1-model.pt'))

test_loss, test_acc = evaluate(model, test_iterator, criterion)

print(f'Test Loss: {test_loss:.3f} | Test Acc: {test_acc*100:.2f}%')<jupyter_output>Test Loss: 0.712 | Test Acc: 45.94%<jupyter_text>Save the ModelLet's save the model weights<jupyter_code>cpu_model = model.to('cpu')
torch.save(cpu_model.state_dict(), 'simple_sentiment_analysis.pt')<jupyter_output><empty_output><jupyter_text>Save model metadata<jupyter_code>import pickle

with open('simple_sentiment_analysis_metadata.pkl', 'wb') as f:
    metadata = {
        'input_stoi': TEXT.vocab.stoi,
        'label_itos': LABEL.vocab.itos,
    }

    pickle.dump(metadata, f)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13590072742832693, 'fraction_numerical': 0.030723149336756527, 'mean_word_length': 4.343392775491541, 'pattern_counts': {'":': 0, '<': 77, '<?xml version=': 0, '>': 76, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '655953', 'license': 'MIT', 'n_tokens_mistral': 4279, 'n_tokens_neox': 3916, 'n_words': 1111, 'path': '09 - Neural Embedding/1_Simple_Sentiment_Analysis.ipynb', 'repo_name': 'shan18/EVA4-Phase-2', 'size': '11685'}