<jupyter_start><jupyter_text>Questions Classification Custom dataset.In this notebook we are going to learn how to load the questions dataset using torchtext and prepare it for sentiment classification in pytorch. We are going to use [this series](https://github.com/CrispenGari/pytorch-python/tree/main/09_TorchText/02_Sentiment_Analyisis_Series) as the base of our code.In this series we will learn the following:1. Creating our own dataset using torchtext2. Using RNN and packed padded sequences for sentiment analysis3. Using Fasttext for sentiment analyisis4. Using ConvNet's to do sentiment analyisis.  1. Data preparation using torchtext.* Refer to [this](https://github.com/CrispenGari/pytorch-python/blob/main/09_TorchText/02_Sentiment_Analyisis_Series/09_TorchText_with_custom_data.ipynb)  and [this](https://github.com/CrispenGari/pytorch-python/blob/main/09_TorchText/01_Introduction/01_TorchText.ipynb) notebook for claryification.I've already uploaded the files which we are going to use in my google drive. So the first thing we should do is to mount the google drive.<jupyter_code>from google.colab import drive
drive.mount('/content/drive')<jupyter_output>Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).<jupyter_text>Imports<jupyter_code>import time
from prettytable import PrettyTable
import numpy as np
from matplotlib import pyplot as plt
import pandas as pd

import torch, os, random
from torch import nn
import torch.nn.functional as F

torch.__version__<jupyter_output><empty_output><jupyter_text>Setting up the seeds<jupyter_code>SEED = 42

np.random.seed(SEED)
random.seed(SEED)
torch.manual_seed(SEED)
torch.cuda.manual_seed(SEED)
torch.backends.cudnn.deteministic = True<jupyter_output><empty_output><jupyter_text>Splitting setsWe are going to create 3 files which are:```train.csvtest.csvval.csv```<jupyter_code>base_path = '/content/drive/MyDrive/NLP Data/questions-classification'
file_path = os.path.join(base_path, "Question_Classification_Dataset.csv")
dataframe = pd.read_csv(file_path)
dataframe.head(1)
dataframe.isnull().any()<jupyter_output><empty_output><jupyter_text>Splitting the sets.<jupyter_code>from sklearn.model_selection import train_test_split

train, valid = train_test_split(dataframe, test_size=.05)
valid, test = train_test_split(valid, test_size=.10)
len(train), len(test), len(valid)<jupyter_output><empty_output><jupyter_text>Saving the files.<jupyter_code>train_path = 'train.csv'
test_path = 'test.csv'
val_path = 'val.csv'
if not os.path.exists(os.path.join(base_path, "pytorch")):
  os.makedirs(os.path.join(base_path, "pytorch"))

valid.to_csv(os.path.join(base_path, "pytorch", val_path), index=False)
test.to_csv(os.path.join(base_path, "pytorch", test_path), index=False)
train.to_csv(os.path.join(base_path, "pytorch", train_path), index=False)

print("files saved")<jupyter_output>files saved<jupyter_text>Loading files.Now we have 3 files for three sets that were created which are:```train.csvtest.csvval.csv```We are going to use torchtext to load these files. Paths<jupyter_code>files_path = '/content/drive/MyDrive/NLP Data/questions-classification/pytorch'<jupyter_output><empty_output><jupyter_text>Creating `.json` files for all the three sets.<jupyter_code>train_path_json = 'train.json'
test_path_json = 'test.json'
val_path_json = 'val.json'<jupyter_output><empty_output><jupyter_text>Creating dataframes for each set so that we can easily convert the files to `.json` files.<jupyter_code>train_dataframe = pd.read_csv(os.path.join(files_path, train_path))
test_dataframe = pd.read_csv(os.path.join(files_path, test_path))
val_dataframe = pd.read_csv(os.path.join(files_path, val_path))

train_dataframe.to_json(os.path.join(files_path, train_path_json),  orient="records", lines=True)
test_dataframe.to_json(os.path.join(files_path, test_path_json),  orient="records", lines=True)
val_dataframe.to_json(os.path.join(files_path, val_path_json),  orient="records", lines=True)

print("saved!")<jupyter_output>saved!<jupyter_text>Creating the Fields.In this first notebook our task is simple we are going to classify one label based on a single question. So we need a `Label` and a `Text` Field. On the `Text` field we are going to pass the arg `include_lengths=True` since we are going to used packed padded sequences.<jupyter_code>from torchtext.legacy import data, datasets
TEXT = data.Field(
   tokenize="spacy",
   include_lengths = True,
  tokenizer_language = 'en_core_web_sm',
)
LABEL = data.LabelField()
fields = {
  "Questions": ('text', TEXT),
  "Category1": ('label', LABEL)
}<jupyter_output><empty_output><jupyter_text>Creating the dataset.We ar going to use the `TabularDataset.split()` to create the datasets.<jupyter_code>train_data, val_data, test_data = data.TabularDataset.splits(
   files_path,
   train=train_path_json,
   test= test_path_json,
   validation= val_path_json,
   format = "json",
   fields=fields
)
len(train_data), len(test_data), len(val_data)
print(vars(train_data.examples[0]))<jupyter_output>{'text': ['What', 'is', 'the', 'name', 'of', 'Miss', 'India', '1994', '?'], 'label': 'HUM'}<jupyter_text>Building the Vocabulary and Loading the `pretrained` word vectors.We are going to use the `glove.6B.100d` word vectors which was trained with 6 billion words and each word is a 100 dimesional vector.**Note** We should only build the vocabulary on the `train` dataset only.<jupyter_code>MAX_VOCAB_SIZE = 100_000_000

TEXT.build_vocab(
    train_data,
     max_size = MAX_VOCAB_SIZE,
    vectors = "glove.6B.100d",
    unk_init = torch.Tensor.normal_
)
LABEL.build_vocab(train_data)<jupyter_output><empty_output><jupyter_text>Device.<jupyter_code>device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
device
LABEL.vocab.stoi<jupyter_output><empty_output><jupyter_text>Creating iterators.We are going to use our favorite iterator known as the `BucketIterator` to create iterators for all the sets that we have.<jupyter_code>sort_key = lambda x: len(x.text)

BATCH_SIZE = 64

train_iter, val_iter, test_iter = data.BucketIterator.splits(
    (train_data, val_data, test_data),
    device = device,
    batch_size = BATCH_SIZE,
    sort_key = sort_key,
)<jupyter_output><empty_output><jupyter_text>Creating the Model.<jupyter_code>class QuestionsLSTMRNN(nn.Module):
  def __init__(self, 
               vocab_size,
               embedding_size,
               hidden_size,
               output_size,
               num_layers,
               pad_index,
               bidirectional = True,
               dropout=.5
               ):
    super(QuestionsLSTMRNN, self).__init__()
    self.embedding = nn.Embedding(
        vocab_size,
        embedding_size,
        padding_idx = pad_index
    )
    self.lstm = nn.LSTM(
        embedding_size,
        hidden_size  = hidden_size,
        bidirectional = bidirectional,
        num_layers = num_layers,
        dropout = dropout
    )
    self.fc_1 = nn.Linear(
        hidden_size * 2 if bidirectional else hidden_size,
        out_features = 512
    )
    self.fc_2 = nn.Linear(
        512,
        out_features = 256
    )
    self.out = nn.Linear(
        256,
        out_features = output_size
    )
    self.dropout = nn.Dropout(dropout)

  def forward(self, text, text_lengths):
    embedded = self.dropout(self.embedding(text))
    packed_embedded = nn.utils.rnn.pack_padded_sequence(
        embedded, text_lengths.to('cpu'), enforce_sorted=False
    )
    packed_output, (h_0, c_0) = self.lstm(packed_embedded)
    output, output_lengths = nn.utils.rnn.pad_packed_sequence(packed_output)
    h_0 = self.dropout(torch.cat((h_0[-2,:,:], h_0[-1,:,:]), dim = 1))
    out = self.dropout(self.fc_1(h_0))
    out = self.dropout(self.fc_2(h_0))
    return self.out(out)<jupyter_output><empty_output><jupyter_text>Creating the model instance.<jupyter_code>INPUT_DIM = len(TEXT.vocab)
EMBEDDING_DIM = 100
HIDDEN_DIM = 256
OUTPUT_DIM =  6
N_LAYERS = 2
BIDIRECTIONAL = True
DROPOUT = 0.5
PAD_IDX = TEXT.vocab.stoi[TEXT.pad_token] 

questions_model = QuestionsLSTMRNN(
            INPUT_DIM, 
            EMBEDDING_DIM, 
            HIDDEN_DIM, 
            OUTPUT_DIM, 
            N_LAYERS, 
            bidirectional = BIDIRECTIONAL, 
            dropout = DROPOUT, 
            pad_index = PAD_IDX
            ).to(device)
questions_model<jupyter_output><empty_output><jupyter_text>Model parameters<jupyter_code>def count_trainable_params(model):
  return sum(p.numel() for p in model.parameters()), sum(p.numel() for p in model.parameters() if p.requires_grad)

n_params, trainable_params = count_trainable_params(questions_model)
print(f"Total number of paramaters: {n_params:,}\nTotal tainable parameters: {trainable_params:,}")<jupyter_output>Total number of paramaters: 3,610,970
Total tainable parameters: 3,610,970<jupyter_text>Loading pretrained vextors to the embedding layer.<jupyter_code>pretrained_embeddings  = TEXT.vocab.vectors
questions_model.embedding.weight.data.copy_(pretrained_embeddings)<jupyter_output><empty_output><jupyter_text>Zeroing the `` and `` tokens.<jupyter_code>UNK_IDX = TEXT.vocab.stoi[TEXT.unk_token] or TEXT.vocab.stoi["<unk>"]
questions_model.embedding.weight.data[UNK_IDX] = torch.zeros(EMBEDDING_DIM)
questions_model.embedding.weight.data[PAD_IDX] = torch.zeros(EMBEDDING_DIM)
questions_model.embedding.weight.data<jupyter_output><empty_output><jupyter_text>Loss and optimizer.We are going to use the Adam as our optimizer with the default leaning rate. We are also going to use `CrossEntropyLoss()` as our loss function.<jupyter_code>optimizer = torch.optim.Adam(questions_model.parameters())
criterion = nn.CrossEntropyLoss().to(device)<jupyter_output><empty_output><jupyter_text>Accuracy function.We are going to create the `categorical_accuracy()` function that will calculate the categorical accuracy for predicted labels and actual labels.<jupyter_code>def categorical_accuracy(preds, y):
  top_pred = preds.argmax(1, keepdim = True)
  correct = top_pred.eq(y.view_as(top_pred)).sum()
  return correct.float() / y.shape[0]<jupyter_output><empty_output><jupyter_text>Training and Evaluation functions.<jupyter_code>def train(model, iterator, optimizer, criterion):
    epoch_loss = 0
    epoch_acc = 0
    model.train()
    for batch in iterator:
        optimizer.zero_grad()
        text, text_lengths = batch.text
        predictions = model(text, text_lengths).squeeze(1)
        loss = criterion(predictions, batch.label)
        acc = categorical_accuracy(predictions, batch.label)
        loss.backward()
        optimizer.step()
        epoch_loss += loss.item()
        epoch_acc += acc.item()
    return epoch_loss / len(iterator), epoch_acc / len(iterator)

def evaluate(model, iterator, criterion):
    epoch_loss = 0
    epoch_acc = 0
    model.eval()
    with torch.no_grad():
        for batch in iterator:
            text, text_lengths = batch.text
            predictions = model(text, text_lengths)
            loss = criterion(predictions, batch.label)
            acc = categorical_accuracy(predictions, batch.label)
            epoch_loss += loss.item()
            epoch_acc += acc.item()
    return epoch_loss / len(iterator), epoch_acc / len(iterator)<jupyter_output><empty_output><jupyter_text>Training loop.We are going to create helper functions that will help us to visualize our training.1. Time to string<jupyter_code>def hms_string(sec_elapsed):
    h = int(sec_elapsed / (60 * 60))
    m = int((sec_elapsed % (60 * 60)) / 60)
    s = sec_elapsed % 60
    return "{}:{:>02}:{:>05.2f}".format(h, m, s)<jupyter_output><empty_output><jupyter_text>2. tabulate training epoch.<jupyter_code>def visualize_training(start, end, train_loss, train_accuracy, val_loss, val_accuracy, title):
  data = [
       ["Training", f'{train_loss:.3f}', f'{train_accuracy:.3f}', f"{hms_string(end - start)}" ],
       ["Validation", f'{val_loss:.3f}', f'{val_accuracy:.3f}', "" ],       
  ]
  table = PrettyTable(["CATEGORY", "LOSS", "ACCURACY", "ETA"])
  table.align["CATEGORY"] = 'l'
  table.align["LOSS"] = 'r'
  table.align["ACCURACY"] = 'r'
  table.align["ETA"] = 'r'
  table.title = title
  for row in data:
    table.add_row(row)
  print(table)

N_EPOCHS = 100
best_valid_loss = float('inf')
for epoch in range(N_EPOCHS):
    start = time.time()
    train_loss, train_acc = train(questions_model, train_iter, optimizer, criterion)
    valid_loss, valid_acc = evaluate(questions_model, val_iter, criterion)
    title = f"EPOCH: {epoch+1:02}/{N_EPOCHS:02} {'saving best model...' if valid_loss < best_valid_loss else 'not saving...'}"
    if valid_loss < best_valid_loss:
        best_valid_loss = valid_loss
        torch.save(questions_model.state_dict(), 'best-model.pt')
    end = time.time()
    visualize_training(start, end, train_loss, train_acc, valid_loss, valid_acc, title)<jupyter_output>+--------------------------------------------+
|     EPOCH: 01/100 saving best model...     |
+------------+-------+----------+------------+
| CATEGORY   |  LOSS | ACCURACY |        ETA |
+------------+-------+----------+------------+
| Training   | 1.202 |    0.509 | 0:00:01.16 |
| Validation | 0.882 |    0.696 |            |
+------------+-------+----------+------------+
+--------------------------------------------+
|     EPOCH: 02/100 saving best model...     |
+------------+-------+----------+------------+
| CATEGORY   |  LOSS | ACCURACY |        ETA |
+------------+-------+----------+------------+
| Training   | 0.815 |    0.692 | 0:00:01.06 |
| Validation | 0.731 |    0.706 |            |
+------------+-------+----------+------------+
+--------------------------------------------+
|     EPOCH: 03/100 saving best model...     |
+------------+-------+----------+------------+
| CATEGORY   |  LOSS | ACCURACY |        ETA |
+------------+-------+----------+------------+
| Training   [...]<jupyter_text>Model Evaluation.<jupyter_code>questions_model.load_state_dict(torch.load('best-model.pt'))

test_loss, test_acc = evaluate(questions_model, test_iter, criterion)
print(f'Test Loss: {test_loss:.3f} | Test Acc: {test_acc*100:.2f}%')<jupyter_output>Test Loss: 0.568 | Test Acc: 85.71%<jupyter_text>Model Inference.We are now ready to make predictions with our model.<jupyter_code>import en_core_web_sm
nlp = en_core_web_sm.load()
reversed_labels = dict([(v, k) for (k, v) in LABEL.vocab.stoi.items()])
reversed_labels
def tabulate(column_names, data, title="QUESTIONS PREDICTIONS TABLE"):
  table = PrettyTable(column_names)
  table.align[column_names[0]] = "l"
  table.align[column_names[1]] = "l"
  for row in data:
    table.add_row(row)
  print(table)

def predict_question_type(model, sentence, min_len = 5, actual_class=0):
    model.eval()
    with torch.no_grad():
      tokenized = [tok.text for tok in nlp.tokenizer(sentence)]
     
      if len(tokenized) < min_len:
          tokenized += ['<pad>'] * (min_len - len(tokenized))
      indexed = [TEXT.vocab.stoi[t] for t in tokenized]
      length =  [len(indexed)]
      tensor = torch.LongTensor(indexed).to(device)
      tensor = tensor.unsqueeze(1)
      length_tensor = torch.LongTensor(length)
      probabilities = model(tensor, length_tensor)
      prediction = torch.argmax(probabilities, dim=1)
      prediction = prediction.item()
    
      table_headers =["KEY", "VALUE"]
      table_data = [
          ["PREDICTED CLASS",  prediction],
          ["ACTUAL CLASS", actual_class],
          ["PREDICTED CLASS NAME",  reversed_labels[prediction]],    
      ]
      tabulate(table_headers, table_data)

reversed_labels<jupyter_output><empty_output><jupyter_text>Location<jupyter_code>predict_question_type(questions_model, "What are the largest libraries in the US ?", actual_class=4)<jupyter_output>+----------------------+-------+
| KEY                  | VALUE |
+----------------------+-------+
| PREDICTED CLASS      | 4     |
| ACTUAL CLASS         | 4     |
| PREDICTED CLASS NAME | LOC   |
+----------------------+-------+<jupyter_text>Human<jupyter_code>predict_question_type(questions_model, "Who is <NAME> , 1767-1834 ?", actual_class=1)<jupyter_output>+----------------------+-------+
| KEY                  | VALUE |
+----------------------+-------+
| PREDICTED CLASS      | 1     |
| ACTUAL CLASS         | 1     |
| PREDICTED CLASS NAME | HUM   |
+----------------------+-------+<jupyter_text>DESCRIPTION<jupyter_code>predict_question_type(questions_model, "What is the root of all evil ? ", actual_class=2)<jupyter_output>+----------------------+-------+
| KEY                  | VALUE |
+----------------------+-------+
| PREDICTED CLASS      | 2     |
| ACTUAL CLASS         | 2     |
| PREDICTED CLASS NAME | DESC  |
+----------------------+-------+<jupyter_text>Numeric<jupyter_code>predict_question_type(questions_model, "How many watts make a kilowatt ?", actual_class=3)<jupyter_output>+----------------------+-------+
| KEY                  | VALUE |
+----------------------+-------+
| PREDICTED CLASS      | 3     |
| ACTUAL CLASS         | 3     |
| PREDICTED CLASS NAME | NUM   |
+----------------------+-------+<jupyter_text>ENTITY<jupyter_code>predict_question_type(questions_model, "What films featured the character Popeye Doyle ?", actual_class=0)<jupyter_output>+----------------------+-------+
| KEY                  | VALUE |
+----------------------+-------+
| PREDICTED CLASS      | 0     |
| ACTUAL CLASS         | 0     |
| PREDICTED CLASS NAME | ENTY  |
+----------------------+-------+<jupyter_text>ABBREVIATION<jupyter_code>predict_question_type(questions_model, "What does NECROSIS mean ?", actual_class=5)<jupyter_output>+----------------------+-------+
| KEY                  | VALUE |
+----------------------+-------+
| PREDICTED CLASS      | 5     |
| ACTUAL CLASS         | 5     |
| PREDICTED CLASS NAME | ABBR  |
+----------------------+-------+
{'fraction_non_alphanumeric': 0.16744861483467383, 'fraction_numerical': 0.014242627345844505, 'mean_word_length': 3.207990599294947, 'pattern_counts': {'":': 2, '<': 122, '<?xml version=': 0, '>': 121, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '32', 'dir': 'jupyter-structured-clean-dedup', 'id': '60663', 'license': 'MIT', 'n_tokens_mistral': 6245, 'n_tokens_neox': 5928, 'n_words': 1227, 'path': '04_Questions_Classification/01_Questions_Classification.ipynb', 'repo_name': 'CrispenGari/NLP-PyTorch', 'size': '17912'}