<jupyter_start><jupyter_text>Multi-class Sentiment Analyisis using Bidirectional `RNN`.Up to now we have been classifying text with two outcomes either positive or otherwise. In this Notebook we are going to take it futher and be able to classify multiclass text.When we have more than 2 examples, our output must be a $C$ dimensional vector, where $C$ is the number of classes.We are going to use the dataset with **6** (TREC dataset) classes it's a dataset of questions and the task is to classify what category the question belongs to. We do not need to set the dtype in the ``LABEL`` field. When doing a ``mutli-class`` problem, PyTorch expects the labels to be numericalized **LongTensors**.The fine_grained argument allows us to use the fine-grained labels (of which there are 50 classes) or not (in which case they'll be 6 classes). You can change this how you please.<jupyter_code>import torch
from torchtext.legacy import data, datasets
import numpy as np
import random
SEED = 42
random.seed(SEED)
torch.manual_seed(SEED)
np.random.seed(SEED)
torch.backends.cudnn.deterministic = True

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
TEXT = data.Field(tokenize="spacy", 
                  tokenizer_language="en_core_web_sm",
                  include_lengths = True
                  )
LABEL = data.LabelField()
train_data, test_data = datasets.TREC.splits(TEXT, LABEL, fine_grained=False)
validation_data, test_data = test_data.split(random_state=random.seed(SEED))
print(f"TRAINING: \t {len(train_data)}")
print(f"TESTING: \t {len(test_data)}")
print(f"VALIDATION: \t {len(validation_data)}")<jupyter_output>TRAINING: 	 5452
TESTING: 	 150
VALIDATION: 	 350<jupyter_text>Let's look at some example of the training set.<jupyter_code>print(vars(train_data[1]))<jupyter_output>{'text': ['What', 'films', 'featured', 'the', 'character', 'Popeye', 'Doyle', '?'], 'label': 'ENTY'}<jupyter_text>Building a vocabulary.<jupyter_code>MAX_VOCAB_SIZE = 25_000
TEXT.build_vocab(
    train_data,
    max_size = MAX_VOCAB_SIZE,
    vectors= "glove.6B.100d",
    unk_init = torch.Tensor.normal_
)
LABEL.build_vocab(train_data)<jupyter_output><empty_output><jupyter_text>Checking Labels.The 6 labels (for the non-fine-grained case) correspond to the 6 types of questions in the dataset:* HUM for questions about humans* ENTY for questions about entities* DESC for questions asking you for a description* NUM for questions where the answer is numerical* LOC for questions where the answer is a location* ABBR for questions asking about abbreviations<jupyter_code>LABEL.vocab.stoi<jupyter_output><empty_output><jupyter_text>Creating iterators.As usual we want to use the `BucketIterator` to create iterators for all sets.<jupyter_code>BATCH_SIZE = 64

train_iterator, validation_iterator, test_iterator = data.BucketIterator.splits(
    (train_data, validation_data, test_data),
    batch_size = BATCH_SIZE,
    device = device
)<jupyter_output><empty_output><jupyter_text>Creating A `CNN` model to classify Text.<jupyter_code>import torch.nn as nn
from torch.nn import  functional as F
class RNNLSTM(nn.Module):
  def __init__(self, vocab_size, embedding_dim, hidden_dim, output_dim, n_layers, 
                bidirectional, dropout, pad_idx):
    super(RNNLSTM, self).__init__()

    self.embedding = nn.Embedding(vocab_size, embedding_dim=embedding_dim, padding_idx=pad_idx)
    self.lstm = nn.LSTM(embedding_dim, hidden_size=hidden_dim, 
                        bidirectional=bidirectional, dropout=dropout,
                        num_layers=n_layers)
    self.fc = nn.Linear(2 * hidden_dim, output_dim)
    self.dropout = nn.Dropout(dropout)

  def forward(self, text, text_lengths):
    #text = [sent len, batch size]
    embedded = self.dropout(self.embedding(text))
    # lengths need to be on CPU!
    packed_embedded = nn.utils.rnn.pack_padded_sequence(embedded, text_lengths.to('cpu'), enforce_sorted=False)
    packed_output, (h_0, c_0) = self.lstm(packed_embedded)
    
    #unpack sequence
    output, output_lengths = nn.utils.rnn.pad_packed_sequence(packed_output)
    h_0 = self.dropout(torch.cat((h_0[-2,:,:], h_0[-1,:,:]), dim = 1))        
    #h_0 = [batch size, hid dim * num directions]     
    return self.fc(h_0)<jupyter_output><empty_output><jupyter_text>Hyper parameters<jupyter_code>INPUT_DIM = len(TEXT.vocab) # # 25002
EMBEDDING_DIM = 100
HIDDEN_DIM = 256
OUTPUT_DIM = len(LABEL.vocab.stoi)
N_LAYERS = 2
BIDIRECTIONAL = True
DROPOUT = 0.5
PAD_IDX = TEXT.vocab.stoi[TEXT.pad_token] # 0

model = RNNLSTM(INPUT_DIM, 
            EMBEDDING_DIM, 
            HIDDEN_DIM, 
            OUTPUT_DIM, 
            N_LAYERS, 
            BIDIRECTIONAL, 
            DROPOUT, 
            PAD_IDX)
model<jupyter_output><empty_output><jupyter_text>Trainable parameters<jupyter_code>def count_trainable_params(model):
  return sum(p.numel() for p in model.parameters() if p.requires_grad)

print(f'The model has  {count_trainable_params(model):,} trainable parameters')<jupyter_output>The model has  3,247,522 trainable parameters<jupyter_text>Loading the pretrainned embeddings<jupyter_code>pretrained_embeddings = TEXT.vocab.vectors
model.embedding.weight.data.copy_(pretrained_embeddings)<jupyter_output><empty_output><jupyter_text>Make the `` and `` layers zeros.The first two layers should have zeros.<jupyter_code>for i in range(2):
  model.embedding.weight.data[i] = torch.zeros(EMBEDDING_DIM)

model.embedding.weight.data<jupyter_output><empty_output><jupyter_text>Trainning the Model. Generally:``CrossEntropyLoss`` is used when our examples exclusively belong to one of $C$ classes``BCEWithLogitsLoss`` is used when our examples exclusively belong to only 2 classes (0 and 1) and is also used in the case where our examples belong to between 0 and $C$ classes (aka multilabel classification).In this Notebook we are going to use ``CrossEntropyLoss``.<jupyter_code>optimizer = torch.optim.Adam(model.parameters())
criterion = nn.CrossEntropyLoss()<jupyter_output><empty_output><jupyter_text>Pushing the model and Criterion to the device.<jupyter_code>model = model.to(device)
criterion = criterion.to(device)<jupyter_output><empty_output><jupyter_text>Accuracy On Multi-Class ClassificationBefore, we had a function that calculated accuracy in the binary label case, where we said if the value was over ``0.5`` then we would assume it is positive. In the case where we have more than 2 classes, our model outputs a $C$ dimensional vector, where the value of each element is the beleief that the example belongs to that class.For example, in our labels we have: ``**'HUM' = 0, 'ENTY' = 1, 'DESC' = 2, 'NUM' = 3, 'LOC' = 4 and 'ABBR' = 5**.``If the output of our model was something like: ``[5.1, 0.3, 0.1, 2.1, 0.2, 0.6]`` this means that the model strongly believes the example belongs to class ``0``, a question about a human, and slightly believes the example belongs to class 3, a numerical question.We calculate the accuracy by performing an argmax to get the index of the maximum value in the prediction for each element in the batch, and then counting how many times this equals the actual label. We then average this across the batch.<jupyter_code>def categorical_accuracy(preds, y):
    top_pred = preds.argmax(1, keepdim = True)
    correct = top_pred.eq(y.view_as(top_pred)).sum()
    acc = correct.float() / y.shape[0]
    return acc<jupyter_output><empty_output><jupyter_text>The training loop and evaluating is similar to before, without the need to squeeze the model predictions as CrossEntropyLoss expects the input to be ``[batch size, n classes]`` and the label to be ``[batch size]``.`<jupyter_code>def train(model, iterator, optimizer, criterion):
    epoch_loss = 0
    epoch_acc = 0
    model.train()
    for batch in iterator:
        optimizer.zero_grad()
        text, text_lengths = batch.text
        predictions = model(text, text_lengths).squeeze(1)
        loss = criterion(predictions, batch.label)
        acc = categorical_accuracy(predictions, batch.label)
        loss.backward()
        optimizer.step()
        epoch_loss += loss.item()
        epoch_acc += acc.item()
    return epoch_loss / len(iterator), epoch_acc / len(iterator)

def evaluate(model, iterator, criterion):
    epoch_loss = 0
    epoch_acc = 0
    model.eval()
    with torch.no_grad():
        for batch in iterator:
            text, text_lengths = batch.text
            predictions = model(text, text_lengths).squeeze(1)
            loss = criterion(predictions, batch.label)
            acc = categorical_accuracy(predictions, batch.label)
            epoch_loss += loss.item()
            epoch_acc += acc.item()
    return epoch_loss / len(iterator), epoch_acc / len(iterator)<jupyter_output><empty_output><jupyter_text>We'll also create a function to tell us how long an epoch takes to compare training times between models.<jupyter_code>import time
def epoch_time(start_time, end_time):
    elapsed_time = end_time - start_time
    elapsed_mins = int(elapsed_time / 60)
    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))
    return elapsed_mins, elapsed_secs

N_EPOCHS = 5
best_valid_loss = float('inf')
for epoch in range(N_EPOCHS):
    start_time = time.time()
    train_loss, train_acc = train(model, train_iterator, optimizer, criterion)
    valid_loss, valid_acc = evaluate(model, validation_iterator, criterion)
    end_time = time.time()
    epoch_mins, epoch_secs = epoch_time(start_time, end_time)
    if valid_loss < best_valid_loss:
        best_valid_loss = valid_loss
        torch.save(model.state_dict(), 'best-model.pt')
    print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')
    print(f'\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')
    print(f'\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')<jupyter_output>Epoch: 01 | Epoch Time: 0m 3s
	Train Loss: 1.174 | Train Acc: 52.64%
	 Val. Loss: 0.795 |  Val. Acc: 70.45%
Epoch: 02 | Epoch Time: 0m 3s
	Train Loss: 0.793 | Train Acc: 70.19%
	 Val. Loss: 0.559 |  Val. Acc: 78.63%
Epoch: 03 | Epoch Time: 0m 3s
	Train Loss: 0.673 | Train Acc: 75.18%
	 Val. Loss: 0.404 |  Val. Acc: 87.14%
Epoch: 04 | Epoch Time: 0m 3s
	Train Loss: 0.562 | Train Acc: 79.68%
	 Val. Loss: 0.369 |  Val. Acc: 84.88%
Epoch: 05 | Epoch Time: 0m 3s
	Train Loss: 0.506 | Train Acc: 81.49%
	 Val. Loss: 0.329 |  Val. Acc: 88.73%<jupyter_text>Evaluating the `model`<jupyter_code>model.load_state_dict(torch.load('best-model.pt'))

test_loss, test_acc = evaluate(model, test_iterator, criterion)

print(f'Test Loss: {test_loss:.3f} | Test Acc: {test_acc*100:.2f}%')<jupyter_output>Test Loss: 0.403 | Test Acc: 87.12%<jupyter_text>User Input.<jupyter_code>import spacy
import en_core_web_sm
nlp = en_core_web_sm.load()

def predict_class(model, sent):
  model.eval()

  tokenized = [tok.text for tok in nlp.tokenizer(sent)]
  indexed = [TEXT.vocab.stoi[t] for t in tokenized]
  length = [len(indexed)]

  tensor = torch.LongTensor(indexed).to(device)
  tensor = tensor.unsqueeze(1)
  length_tensor = torch.LongTensor(length)
  prediction = torch.argmax(model(tensor, length_tensor), dim=1)
  return prediction.item()
labels = dict(LABEL.vocab.stoi)
labels = dict([(v, k) for (k, v) in labels.items()])<jupyter_output><empty_output><jupyter_text>Getting the user input.<jupyter_code>while True:
  qn = input("Enter a question\nOR 'exit' to close:\n")

  if qn.lower() == "exit":
    break
  pred_class = predict_class(model, qn)
  print(f'Predicted class is: {pred_class} = {labels[pred_class]}')<jupyter_output>Enter a question
OR 'exit' to close:
What is your name?
Predicted class is: 2 = DESC
Enter a question
OR 'exit' to close:
exit
{'fraction_non_alphanumeric': 0.1070359921347354, 'fraction_numerical': 0.020005129520389842, 'mean_word_length': 4.41574074074074, 'pattern_counts': {'":': 1, '<': 68, '<?xml version=': 0, '>': 67, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '340445', 'license': 'MIT', 'n_tokens_mistral': 4182, 'n_tokens_neox': 3907, 'n_words': 1115, 'path': '09_NLP/02_Sentiment_Analyisis_Series/06_Muticlass_Sentiment_Analyisis_RNN.ipynb', 'repo_name': 'CrispenGari/pytorch-python', 'size': '11697'}