<jupyter_start><jupyter_text>Yelp review rating prediction  Millions of people share a great number of reviews about business on [Yelp.com](https://www.yelp.com/) and Yelp mobile app everyday. These reviews and ratings help other users to make a choice. We used [Yelp APIs (application programming interface)](https://www.yelp.ca/developers) to collect over 35,000 reviews of 1,000 restaurants in New York City. We split this dataset into 90\% TRAIN set (28,000 reviews), 10\% DEV set (3,500 reviews), and 10\% TEST set (3,500 reviews). Each review has text review content and a corresponding label (i.e., 5-level rating star). This table shows the class ditribution of TRAIN and DEV sets.|    Rating  |    of Train   reviews|  of Dev reviews    |  | ---------- | -----------------  |-----|  | 1star      | 5,619              | 683 |  | 2star      | 5,616              | 677 |  | 3star      | 5,583              | 713 |  | 4star      | 5,532              | 733 |  | 5star      | 5,650              | 694 |  In directory `./data/yelp_review/`, we provide the `TRAIN` and `DEV` sets with the corresponding labels.We use the TRAIN and DEV sets to develop a classification system for this task.   Mounting the drive on colab<jupyter_code>from google.colab import drive
drive.mount('/content/drive')<jupyter_output>Mounted at /content/drive<jupyter_text>Data Preprocessing<jupyter_code>import nltk
nltk.download('stopwords')
from nltk.corpus import stopwords
import spacy
import string
spacy_en = spacy.load('en_core_web_sm')
# punctuation = set(string.punctuation)
# stop_words = set(stopwords.words('english'))
def tokenize_en(text):
    """
    Tokenizes English text from a string into a list of strings (tokens)
    """
    tokens = []
    for tok in spacy_en.tokenizer(text):
      tokens.append(tok.text)

    return tokens
TEXT = Field(sequential=True, tokenize=tokenize_en, lower=False)
LABEL = Field(sequential=False, unk_token = None)
train,val,_ = TabularDataset.splits(path= "/yelp/",train='train.tsv', validation="val.tsv", test="test.tsv", # file names
    format='tsv',
    skip_header=True, # if your tsv file has a header, make sure to pass this to ensure it doesn't get proceesed as data!
    fields=[('tweet', TEXT), ('label', LABEL)])

_,_,test = TabularDataset.splits(path= "/yelp/",train='train.tsv', validation="val.tsv", test="test.tsv", # file names
    format='tsv',
    skip_header=True, # if your tsv file has a header, make sure to pass this to ensure it doesn't get proceesed as data!
    fields=[('tweet', TEXT)])
TEXT.build_vocab(train) # builds vocabulary based on all the words that occur at least twice in the training set
LABEL.build_vocab(train)
print(len(TEXT.vocab.stoi))
print(len(LABEL.vocab.stoi))
print(LABEL.vocab.stoi)
#Creating the val, test and train iterators.

val_iter = Iterator(
 dataset = val, # we pass in the datasets we want the iterator to draw data from
 batch_size= 128,
 sort_key= None, 
 sort= False,
# A key to use for sorting examples in order to batch together examples with similar lengths and minimize padding. 
 sort_within_batch= False,
 train= False
)


test_iter = Iterator(
 dataset = test, # we pass in the datasets we want the iterator to draw data from
 batch_size= 128,
 sort_key= None, 
 sort= False,
# A key to use for sorting examples in order to batch together examples with similar lengths and minimize padding. 
 sort_within_batch= False,
 train= False
)


train_iter,_,_ = BucketIterator.splits(
 (train,val, test), # we pass in the datasets we want the iterator to draw data from
 batch_sizes= (128,128,128),
 sort_key=lambda x: len(x.tweet), 
 sort= True,
# A key to use for sorting examples in order to batch together examples with similar lengths and minimize padding. 
 sort_within_batch= True
)<jupyter_output><empty_output><jupyter_text>Model Evaluation<jupyter_code>## GRU model for evaluation.

import torch.nn as nn
class GRUmodel(nn.Module):
  
  def __init__(self, embedding_size, vocab_size, output_size, hidden_size, layers):
    # In the constructor we define the layers for our model
    super(GRUmodel, self).__init__()
    # word embedding lookup table
    self.embedding = nn.Embedding(num_embeddings=vocab_size, embedding_dim=embedding_size)
    # core GRU module
    self.GRU_layer = nn.GRU(input_size=embedding_size, hidden_size= hidden_size, num_layers=layers) 
    # activation function
    self.activation_fn = nn.Tanh()
    # classification related modules
    self.linear_layer = nn.Linear(hidden_size, output_size) 
    self.softmax_layer = nn.LogSoftmax(dim=1)
    self.debug = False
  
  def forward(self, x):
    # In the forward function we define the forward propagation logic
    if self.debug:
        print("input word indices shape = ", x.size())
    out = self.embedding(x)
    if self.debug:
        print("word embeddings shape = ", out.size())
    out, _ = self.GRU_layer(out) # since we are not feeding h_0 explicitly, h_0 will be initialized to zeros by default
    if self.debug:
        print("RNN output (features from last layer of RNN for all timesteps) shape = ", out.size())
    # classify based on the hidden representation after RNN processes the last token
    out = out[-1]
    if self.debug:
        print("Tweet embeddings or RNN output (features from last layer of RNN for the last timestep only) shape = ", out.size())
    out = self.activation_fn(out)
    if self.debug:
        print("ReLU output shape = ", out.size())
    out = self.linear_layer(out)
    if self.debug:
        print("linear layer output shape = ", out.size())
    out = self.softmax_layer(out) # accepts 2D or more dimensional inputs
    if self.debug:
        print("softmax layer output shape = ", out.size())
    return out
## Evaluation function

import numpy as np
from sklearn.metrics import accuracy_score
from sklearn.metrics import f1_score

def train(loader,model,criterion,optimizer,device):
    total_loss = 0.0
    # iterate throught the data loader
    num_sample = 0
    for batch in loader:
        # load the current batch
        batch_input = batch.tweet
        batch_output = batch.label
        
        batch_input = batch_input.to(device)
        batch_output = batch_output.to(device)
        # forward propagation
        # pass the data through the model
        model_outputs = model(batch_input)
        # compute the loss
        cur_loss = criterion(model_outputs, batch_output)
        total_loss += cur_loss.item()

        # backward propagation (compute the gradients and update the model)
        # clear the buffer
        optimizer.zero_grad()
        # compute the gradients
        cur_loss.backward()
        # update the weights
        optimizer.step()

        num_sample += batch_output.shape[0]
    return total_loss/num_sample

# evaluation logic based on classification accuracy
def evaluate(loader,model,criterion,device):
    all_pred=[]
    all_label = []
    with torch.no_grad(): # impacts the autograd engine and deactivate it. reduces memory usage and speeds up computation
        for batch in loader:
             # load the current batch
            batch_input = batch.tweet
            batch_output = batch.label

            batch_input = batch_input.to(device)
            # forward propagation
            # pass the data through the model
            model_outputs = model(batch_input)
            # print(model_outputs.shape)
            # identify the predicted class for each example in the batch
            probabilities, predicted = torch.max(model_outputs.cpu().data, 1)
            # put all the true labels and predictions to two lists

            all_pred.extend(predicted)
            all_label.extend(batch_output)
      
    accuracy = accuracy_score(all_label, all_pred)
    f1score = f1_score(all_label, all_pred, average='macro') 
    return accuracy,f1score<jupyter_output><empty_output><jupyter_text>Hyper-parameter optimization<jupyter_code># from numpy.lib.histograms import histogramdd
# from torch._C import JitType
import scipy.stats

LEARNING_RATE= 0.1
MAX_EPOCHS=10
EMBEDDING_SIZE = 300
VOCAB_SIZE = len(TEXT.vocab.stoi)
NUM_CLASSES = len(LABEL.vocab.stoi)

def random_search(hidden_units_list, layers_list):
    results = []
    for i in layers_list:
        for j in hidden_units_list:
            config = {
                #define hyperparameters here
                "layers": i,
                "hidden_size": j
                }
            print(config)
            # model = ConvNet(config["layers"],3,config["filters"],nn.ReLU(),output_size=3, VOCAB_SIZE=VOCAB_SIZE, WORD_VEC_SIZE=WORD_VEC_SIZE)
            model = LSTMmodel(EMBEDDING_SIZE, VOCAB_SIZE, NUM_CLASSES, config["hidden_size"], config["layers"]) 
            model.to(device)
            #print﴾model﴿
            criterion = nn.NLLLoss()
            optimizer = torch.optim.SGD(model.parameters(), lr=LEARNING_RATE)
            max_val = 0
            best_epoch = 0
            for epoch in range(MAX_EPOCHS):
                # train the model for one pass over the data
                train_loss = train(train_iter,model,criterion,optimizer,device)
                # compute the training accuracy
                train_acc = evaluate(train_iter,model,criterion,device)
                # compute the validation accuracy
                val_acc = evaluate(val_iter,model,criterion,device)
                if val_acc[0] > max_val:
                    max_val = val_acc[0]
                    best_epoch = epoch+1
                # print the loss for every epoch
                print('epoch ',epoch+1,'loss ', train_loss,'Train Accuracy & F1',train_acc,'Validation Accuracy & F1 ', val_acc)
                # Append the results for every epoch
                results.append((max_val,best_epoch,config))
    return results
# Calling the hyperparamater optimization for LSTM
layers_list = [1, 2, 3]
hidden_size_list = [100, 200, 300, 400, 500]
random_search(hidden_size_list,layers_list)
# Calling the hyperparamater optimization for GRU
layers_list = [1, 2, 3]
hidden_size_list = [100, 200, 300, 400, 500]
random_search(hidden_size_list,layers_list)
# Calling the hyperparamater optimization for Ensemble of GRU,LSTM
layers_list = [1,2]
hidden_size_list = [100,200]
random_search(hidden_size_list,layers_list)<jupyter_output>{'layers': 1, 'hidden_size': 100}
epoch  1 loss  0.012522491071905409 Train Accuracy & F1 (0.21735714285714286, 0.15361606304701242) Validation Accuracy & F1  (0.19542857142857142, 0.06754167280268557)
epoch  2 loss  0.012450981387070247 Train Accuracy & F1 (0.24471428571428572, 0.21402334972287748) Validation Accuracy & F1  (0.19485714285714287, 0.06744818503972153)
epoch  3 loss  0.012249883941241673 Train Accuracy & F1 (0.28214285714285714, 0.2544283262226445) Validation Accuracy & F1  (0.19514285714285715, 0.06798020554900973)
epoch  4 loss  0.011688113033771515 Train Accuracy & F1 (0.3407857142857143, 0.32119285104039097) Validation Accuracy & F1  (0.1957142857142857, 0.06809411879594567)
epoch  5 loss  0.011006993519408361 Train Accuracy & F1 (0.39964285714285713, 0.362908521237034) Validation Accuracy & F1  (0.1957142857142857, 0.06805006334620607)
epoch  6 loss  0.010523114174604416 Train Accuracy & F1 (0.4205, 0.38319304061201426) Validation Accuracy & F1  (0.196, 0.0686036433[...]<jupyter_text>Final model training and evaluation {'layers': 2, 'hidden_size': 100}<jupyter_code>EMBEDDING_SIZE = 300
VOCAB_SIZE = len(TEXT.vocab.stoi)
NUM_CLASSES = len(LABEL.vocab.stoi)
HIDDEN_SIZE = 100
NUM_LAYERS = 2


model = GRUmodel(EMBEDDING_SIZE, VOCAB_SIZE, NUM_CLASSES, HIDDEN_SIZE, NUM_LAYERS) 
print(model)
model.to(device)
LEARNING_RATE = 0.01
criterion = nn.NLLLoss()
# create an instance of SGD with required hyperparameters
optimizer = torch.optim.Adam(model.parameters(), lr=LEARNING_RATE)
import os
if not os.path.exists("./drive/MyDrive/checkpoint/"): # check if the directory doesn't exist already
    os.mkdir("./drive/MyDrive/checkpoint/")

MAX_EPOCHS = 10
for epoch in range(MAX_EPOCHS):
    # train the model for one pass over the data
    train_loss = train(train_iter,model,criterion,optimizer,device)  
    # compute the training accuracy
    train_acc = evaluate(train_iter,model,criterion,device)
    # compute the validation accuracy
    val_acc = evaluate(val_iter,model,criterion,device)
    
    # print the loss for every epoch
    print('epoch ',epoch+1,'loss ', train_loss,'Train Accuracy & F1',train_acc,'Validation Accuracy & F1 ', val_acc)
    
    # save model, optimizer, and number of epoch to a dictionary
    model_save = {
            'epoch': epoch,  # number of epoch
            'model_state_dict': model.state_dict(), # model parameters 
            'optimizer_state_dict': optimizer.state_dict(), # save optimizer 
            'loss': train_loss # training loss
            }
    
    # use torch.save to store 
    torch.save(model_save, "./drive/MyDrive/checkpoint/model_{}.pt".format(epoch))
# The best epoch is 2 an we will be generating our predictions based on the best model.

EMBEDDING_SIZE = 300
VOCAB_SIZE = 53723
NUM_CLASSES = len(LABEL.vocab.stoi)
HIDDEN_SIZE = 100
NUM_LAYERS = 2
# define a new model

model = GRUmodel(EMBEDDING_SIZE, VOCAB_SIZE, NUM_CLASSES, HIDDEN_SIZE, NUM_LAYERS) 

# load checkpoint

checkpoint = torch.load("./drive/MyDrive/checkpoint/model_1.pt")

# assign the parameters of checkpoint to this new model

model.load_state_dict(checkpoint['model_state_dict'])
# model.to(device)

print(model) # can be used for inference or for further training
## Funtion to generate the predictions.

def inference(loader):
    all_pred=[]
    all_label = []
    with torch.no_grad(): # impacts the autograd engine and deactivate it. reduces memory usage and speeds up computation
        for batch in loader:
            # load the current batch
            batch_input = batch.tweet
            # forward propagation
            # pass the data through the model
            model_outputs = model(batch_input)
            # identify the predicted class for each example in the batch
            probabilities, predicted = torch.max(model_outputs.cpu().data, 1)
            # put all the true labels and predictions to two lists
            all_pred.extend(predicted)

    return all_pred
pred_list = inference(val_iter)

test_list = inference(test_iter)


pred_list = [x.item() for x in pred_list]
pred_list = [LABEL.vocab.itos[x] for x in pred_list]
test_list = [x.item() for x in test_list]
test_list = [LABEL.vocab.itos[x] for x in test_list]<jupyter_output><empty_output><jupyter_text>Generating Submission File<jupyter_code>def out_prediction(first_name, last_name, prediction_list, dataset, submit_number):
    """
    out_prediction takes three input varibles: first_name, last_name, prediction_list
    <first_name>, string, your first name, e.g., Tom
    <last_name>, string, your last name, e.g., Smith
    <prediction_list>, list of string which includes all your predications of TEST samples
                        e.g., ['1star','5star','3star']
    <dataset> dev or test
    <submit_number> index of your submission
                        
    Generate a file is named with <yourfirstname>_<yourlastname>_PRED.txt in current directory
    """
    output_file = open("{}_{}_{}_{}.txt".format(first_name,last_name, dataset, submit_number),'w')
    for item in prediction_list:
        output_file.write(item+"\n")
    output_file.close()<jupyter_output><empty_output><jupyter_text>A example of using `out_prediction` funtion. You can find a file `Tom_Smith_PRED.txt` in your diretory.<jupyter_code>out_prediction("Varadraj", "Poojari",test_list, "test", "4")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09351278943675448, 'fraction_numerical': 0.04624609300248772, 'mean_word_length': 3.3965227145260797, 'pattern_counts': {'":': 2, '<': 34, '<?xml version=': 0, '>': 35, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '546115', 'license': 'MIT', 'n_tokens_mistral': 5413, 'n_tokens_neox': 4854, 'n_words': 1427, 'path': 'Yelp_review_colab.ipynb', 'repo_name': 'VaradrajPoojari/Yelp-review-rating-prediction-', 'size': '15677'}