<jupyter_start><jupyter_text>Modeling: Aspect-Based Sentiment Analysis (Aspect Extraction) BerTweet *Building a multi-label classifier for ABSA using a BERTweet model* **`Overview:`** As seen mentioned by [(Karimi et al., 2021)](https://arxiv.org/pdf/2001.11316.pdf), aspect-based sentiment analysis (ABSA) can be broken down into two subtasks of *aspect extraction (AE)* and *aspect sentiment classification (ASC)* (Hu and Liu, 2004). This notebook focuses on the first subtask of aspect extraction.\\Here ABSA is framed as a multi-label classification problem and a BERTweet model is fine-tuned to learn and predict the predetermined aspects (price, speed, coverage, customer service, and reliability) for our Nigerian Internet Service domain.**`Initial results:`** Initial results indicate poor extraction performance of the fine-tuned model. This might be as a result of limited data. I look towards a binary relevance approach (using a set of independent classifier which each predict a different aspect) for the same multi-label classification problem in 6.2.2  1. Library Importation<jupyter_code>#Data manipulation and visualization packages
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

#Modeling packages
import torch
from tqdm.auto import tqdm
import pytorch_lightning as pl
from torchmetrics import Precision, Recall, FBeta, AUROC
from torch.utils.data import Dataset, DataLoader
from pytorch_lightning.loggers import TensorBoardLogger
from pytorch_lightning.callbacks import ModelCheckpoint, EarlyStopping
from transformers import AdamW, get_linear_schedule_with_warmup
from transformers import AutoTokenizer, AutoModel

#Model evaluation and metrics
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, multilabel_confusion_matrix

#Set seed
RANDOM_SEED = 123
pl.seed_everything(RANDOM_SEED)<jupyter_output>Global seed set to 123<jupyter_text>2. Loading the data<jupyter_code>#Load the dataframe with punctuated tweets. 
aspect_df = pd.read_csv("../data/processed/absa_text_annotated_price.csv")

#Load the dataframe from the 2nd absa annotation phase
#- Contains mostly tweets with aspect annotation (unlike the dataframe above)
aspect_df2 = pd.read_csv("../data/processed/second_tweet_absa_annotated.csv")

print(f"aspect_df size: {aspect_df.shape}")
print(f"aspect_df2 size: {aspect_df2.shape}")
#Merge both dataframes together
merged_absa_df = pd.concat([aspect_df,aspect_df2.drop(columns = ['Cleaned tweets'])])
print(merged_absa_df.shape)
merged_absa_df.head()<jupyter_output>(488, 3)<jupyter_text>3. Reformat the data  a. Convert lists from string format to list<jupyter_code>#Convert Aspects column
merged_absa_df.Aspects = merged_absa_df.Aspects.apply(lambda x: eval(x) if (pd.notnull(x)) else x)

#Convert Sentiment column
merged_absa_df.Sentiment = merged_absa_df.Sentiment.apply(lambda x: eval(x) if (pd.notnull(x)) else x)

#Make a copy of the dataframe
processed_aspect_df = merged_absa_df.copy()

#Quick preview
processed_aspect_df.head()<jupyter_output><empty_output><jupyter_text>b. Fill NaNs with [None]<jupyter_code>processed_aspect_df = processed_aspect_df.apply(lambda s: s.fillna({i: [None] for i in processed_aspect_df.index}))
processed_aspect_df.head()<jupyter_output><empty_output><jupyter_text>c. Binarizing the aspects<jupyter_code>#List aspects determined during the annotation phase
#Note: This might not be exhaustive! But it should cover most cases. It is also subjective!
#Also using synonyms of these words will likely yield different results
ASPECTS = ['price','speed','reliability','coverage', 'customer service']

#Iterate through all the aspects and if the aspect is not in the tweet, record 0 else record 1
for aspect in ASPECTS[::-1]:
    processed_aspect_df.insert(1,aspect,processed_aspect_df.Aspects.apply(lambda x: 1 if aspect in x else 0))
    
#Drop the Aspects column
processed_aspect_df.drop(columns=['Aspects'] , inplace=True)

#Quick preview
processed_aspect_df.head()
#Save the binarized dataframe
processed_aspect_df.to_csv("../data/processed/absa_binarized.csv",index=False)<jupyter_output><empty_output><jupyter_text>4. Quick EDA<jupyter_code>with plt.style.context(['notebook','no-latex','grid']):
    plt.figure(figsize = (12,7), dpi=300)
    processed_aspect_df[ASPECTS].sum().sort_values()\
                                            .plot(kind = "barh", color = 'cornflowerblue')\
                                            .set(xlabel = "Number of times detected in tweets",
                                                ylabel = "Aspect category")
                        
    plt.show()<jupyter_output><empty_output><jupyter_text>From the above, we see that the labels are quite imbalanced. Trustworthiness is especially important to point out. To account for this, we look into reweighting our loss function. However, before doing so, we look more broadly into aspects vs no aspects tweets distribution:<jupyter_code>#Get the rows with aspects tagged and no aspects tagged
aspects_tagged = processed_aspect_df[processed_aspect_df[ASPECTS].sum(axis=1) > 0]
no_aspects_tagged = processed_aspect_df[processed_aspect_df[ASPECTS].sum(axis=1) == 0]

with plt.style.context(['notebook','no-latex','grid']):
    plt.figure(figsize = (12,7), dpi=300)
    plt.barh(['Aspect(s) tagged','No aspect tagged'],
             [len(aspects_tagged),len(no_aspects_tagged)],color=['mediumslateblue','royalblue'])<jupyter_output><empty_output><jupyter_text>Roughly balanced.  5. Split data into training, validation and test set  a. Split to training and test set<jupyter_code>#Split the data 80:20 (training and test)
train_df, test_df = train_test_split(processed_aspect_df,test_size=0.20,
                                    stratify = processed_aspect_df.coverage)
print(f"Train size: {train_df.shape}", f"Test size: {test_df.shape}")<jupyter_output>Train size: (390, 7) Test size: (98, 7)<jupyter_text>b. Split training set to training and validation set<jupyter_code>train_df, val_df = train_test_split(train_df,test_size=0.25,
                                    stratify = train_df.coverage)
print(f"Train size: {train_df.shape}", f"Validation size: {val_df.shape}")<jupyter_output>Train size: (292, 7) Validation size: (98, 7)<jupyter_text>c. Check to see that all the sets have all the aspect categories<jupyter_code>df_sets = [('Training',train_df),('Validation', val_df),('Test',test_df)]
    
plt.figure(figsize = (12,6))
for idx,df in enumerate(df_sets):
    
    plt.subplot(1,3,idx+1)
    plt.tight_layout()

    with plt.style.context(['notebook','no-latex','grid']):
        
        plt.title(df[0])
        df[1][ASPECTS].sum().plot(kind = "bar", color = 'cornflowerblue')
        plt.xticks(rotation=90)
        
        if idx == 0:
            plt.ylabel('Number of tags in the tweets', fontsize=16)
        
    
plt.show()<jupyter_output><empty_output><jupyter_text>6. Modeling - Aspect Extraction v1**`Default BCE Loss Function`** Dataset from above has been changed so the below section won't reproduce the results if reran!  a. Load the tokenizer and the BERTweet model<jupyter_code>#Load BerTweet tokenizer
TOKENIZER = AutoTokenizer.from_pretrained("vinai/bertweet-base", normalization=True)

#Load the BERTweet model
BERTWEET_MODEL = AutoModel.from_pretrained("vinai/bertweet-base", from_tf = True, return_dict = True)<jupyter_output>Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
2021-12-10 12:09:39.471568: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
All TF 2.0 model weights were used when initializing RobertaModel.

All the weights of RobertaModel were initialized from the TF 2.0 model.
If your task is similar to the task the model of the checkpoint was trained on, you can already use RobertaModel for predictions without further training.<jupyter_text>b. Prepare the training, validation, and test sets<jupyter_code>TRAIN_BATCH_SIZE = 32 #Increase batch size so the model has an increased chance of seeing as many aspects
TEST_BATCH_SIZE = 8
class Generate_PyTorch_Dataset(torch.utils.data.Dataset):

    def __init__(self, dataframe, tokenizer):
        self.dataframe  = dataframe
        self.tokenizer  = tokenizer
        self.max_len    = tokenizer.model_max_length
  
    def __len__(self):
        return len(self.dataframe)
  
    def __getitem__(self, idx):
        
        #Get each row of the dataframe
        data_row = self.dataframe.iloc[idx]
        
        #Get the tweet
        text = str(data_row.Text)
        
        #Get the aspect labels
        labels = data_row[ASPECTS]
        
        #Encode the tweet
        encoded_text = self.tokenizer.encode_plus(
            text,
            add_special_tokens    = True,
            truncation            = True,
            return_attention_mask = True,
            return_token_type_ids = False,
            return_length         = True,
            max_length            = self.max_len,
            return_tensors        = 'pt',
            padding               = "max_length"
        )
        
        
        return {
            'input_ids': encoded_text['input_ids'].flatten(),
            'attention_mask': encoded_text['attention_mask'].flatten(),
            'labels': torch.tensor(labels, dtype=torch.float)
        }
    
class PyTorchDataModule(pl.LightningDataModule):
    
    def __init__(self, train_df, val_df, test_df, tokenizer, train_batch_size=16, test_batch_size=8):
        
        super().__init__()
        self.train_df = train_df
        self.val_df = train_df
        self.test_df = test_df
        self.tokenizer = tokenizer
        self.train_batch_size = train_batch_size
        self.test_batch_size = test_batch_size
        self.max_len = self.tokenizer.model_max_length
        
    def setup(self):
        
        """
        Setup the training, validation and test sets by converting them to Pytorch datasets
        """
        
        self.train_dataset = Generate_PyTorch_Dataset(self.train_df,self.tokenizer)
        self.val_dataset = Generate_PyTorch_Dataset(self.val_df,self.tokenizer)
        self.test_dataset = Generate_PyTorch_Dataset(self.test_df,self.tokenizer)
        
        
    def train_dataloader(self):
        
        """
        Training set dataloader
        """
        
        return DataLoader(self.train_dataset,
                          batch_size  = self.train_batch_size,
                          shuffle     = True, 
                          num_workers = 2)
         
    def val_dataloader(self):
        
        """
        Validation set dataloader
        """
        
        return DataLoader(self.val_dataset, batch_size = self.test_batch_size, num_workers=2)
        
    def test_dataloader(self):
        
        """
        Test set dataloader
        """
        
        return DataLoader(self.test_dataset, batch_size = self.test_batch_size, num_workers=2)
        
#Instantiate and set up the data_module
data_module = PyTorchDataModule(train_df,val_df,test_df,TOKENIZER, TRAIN_BATCH_SIZE, TEST_BATCH_SIZE)
data_module.setup()
class ISP_TweetAspectClassifier(pl.LightningModule):
    
    """
    Adapted from Multi-label Text Classification with BERT and PyTorch Lightning. (2021). 
    Curiousily. https://curiousily.com/posts/multi-label-text-classification-with-bert-and-pytorch-lightning/
    """
    
    #Set the aspect classifier
    def __init__(self, n_classes=5, n_training_steps=None, n_warmup_steps=None, lr=2e-5):
        
        super().__init__()
        self.lr = lr
        self.n_warmup_steps = n_warmup_steps
        self.n_training_steps = n_training_steps
        self.bert = BERTWEET_MODEL
        self.classifier = torch.nn.Linear(self.bert.config.hidden_size, n_classes)
        self.criterion = torch.nn.BCELoss()
        
    def forward(self, input_ids, attention_mask, labels = None):
        output = self.bert(input_ids, attention_mask=attention_mask)
        output = self.classifier(output.pooler_output)
        output = torch.sigmoid(output)
        
        loss = 0
        
        if labels is not None:
            loss = self.criterion(output, labels)
            
        return loss, output
    
    def training_step(self, batch, batch_idx):
        
        input_ids = batch["input_ids"]
        attention_mask = batch["attention_mask"]
        labels = batch["labels"]
        
        loss, outputs = self(input_ids, attention_mask, labels)
        self.log("train_loss", loss, prog_bar=True, logger=True)
        
        return {"loss": loss, "predictions": outputs.detach(), "labels": labels}
    
    def validation_step(self, batch, batch_idx):
        input_ids = batch["input_ids"]
        attention_mask = batch["attention_mask"]
        labels = batch["labels"]
        
        loss, outputs = self(input_ids, attention_mask,labels)
        self.log("val_loss", loss, prog_bar=True, logger=True)
        
        return loss
    
    
    def test_step(self, batch, batch_idx):
        input_ids = batch["input_ids"]
        attention_mask = batch["attention_mask"]
        labels = batch["labels"]
        
        loss, outputs = self(input_ids, attention_mask, labels)
        self.log("test_loss", loss, prog_bar=True, logger=True)
        
        return loss
    
    
    def training_epoch_end(self, outputs):
            
        #List to store the true labels and the model's predictions
        labels = []
        predictions = []
        
        #Iterate through all the outputs and get the true vs. predicted label
        for output in outputs:
            
            for label in output["labels"].detach().cpu():
                labels.append(label)
            for pred in output["predictions"].detach().cpu():
                predictions.append(pred)
        
        #Stack the tensors
        labels = torch.stack(labels).int()
        predictions = torch.stack(predictions)
        
        #Record the AUROC for each aspect after each training epoch
        for idx, name in enumerate(ASPECTS):
            metric = AUROC()
            class_roc_auc = metric(predictions[:,idx], labels[:,idx])
            self.logger.experiment.add_scalar(f"{name}_roc_auc/Train", class_roc_auc, self.current_epoch)
        
        #Calculate the micro-F1
        f1_metric = F1()
        micro_f1 = f1_metric(predictions, labels)
        self.logger.experiment.add_scalar(f"micro_f1_score", micro_f1, self.current_epoch)
   
    
    def configure_optimizers(self):
        
        optimizer = AdamW(self.parameters(), lr = self.lr)
        
        scheduler = get_linear_schedule_with_warmup(
            optimizer,
            num_warmup_steps=self.n_warmup_steps,
            num_training_steps=self.n_training_steps)
        
          
        return {'optimizer': optimizer, 'lr_scheduler':{'scheduler':scheduler,'interval':'step'}}
    
#Define variables
N_EPOCHS = 20
STEPS_PER_EPOCH =len(train_df)//TRAIN_BATCH_SIZE
TOTAL_TRAIN_STEPS = STEPS_PER_EPOCH * N_EPOCHS
N_WARMUP_STEPS = TOTAL_TRAIN_STEPS // 4

#Instantiate the classifier model
model = ISP_TweetAspectClassifier(n_training_steps = TOTAL_TRAIN_STEPS,
                                  n_warmup_steps   = N_WARMUP_STEPS)<jupyter_output><empty_output><jupyter_text>d. Training<jupyter_code>#Setup callback to perform saves during training
checkpoint_callback = ModelCheckpoint(
  dirpath    = "../models/absa-aspect-extraction/bertweet",
  filename   = "ae-{epoch:02d}-{val_loss:.2f}",
  save_top_k = 3, #Save the top 3 models
  verbose    = True,
  monitor    = "val_loss",
  mode       = "min" #Minimize val loss
)

#Log progress in Tensorboard
logger = TensorBoardLogger("../models/absa-aspect-extraction/bertweet/lightning_logs", name = "isp-tweets")
trainer = pl.Trainer(
    #gpus                       = 1,
    logger                      = logger,
    log_every_n_steps           = 10,
    callbacks                   = [checkpoint_callback],
    max_epochs                  =  N_EPOCHS,
    progress_bar_refresh_rate   =  30
)

trainer.fit(model, data_module)<jupyter_output>GPU available: False, used: False
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs

  | Name       | Type         | Params
--------------------------------------------
0 | bert       | RobertaModel | 134 M 
1 | classifier | Linear       | 3.8 K 
2 | criterion  | BCELoss      | 0     
--------------------------------------------
134 M     Trainable params
0         Non-trainable params
134 M     Total params
539.615   Total estimated model params size (MB)<jupyter_text>e. Model Evaluation<jupyter_code>#Evaluate the model's performance on the test dataset
trainer.test(model,data_module)<jupyter_output>/Users/koredeakande/opt/anaconda3/envs/capstone/lib/python3.7/site-packages/pytorch_lightning/trainer/data_loading.py:111: UserWarning: The dataloader, test_dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 8 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
  f"The dataloader, {name}, does not have many workers which may be a bottleneck."<jupyter_text>(i) Evaluate model performance on a generated example<jupyter_code>#Load the best model based on validation loss
trained_model = ISP_TweetAspectClassifier.load_from_checkpoint(
    trainer.checkpoint_callback.best_model_path,
    n_classes=len(ASPECTS)
)
#Load the best model based on validation loss
trained_model = ISP_TweetAspectClassifier.load_from_checkpoint(
    "../models/absa-aspect-extraction/bertweet/ae-epoch=19-val_loss=0.33.ckpt",
    n_classes=len(ASPECTS)
)
#Put model into evaluation mode
trained_model.eval()
trained_model.freeze()
test_comment = "The internet is so slow and it's so expensive"

encoding = TOKENIZER.encode_plus(
  test_comment,
  add_special_tokens=True,
  max_length=TOKENIZER.model_max_length,
  return_token_type_ids=False,
  padding="max_length",
  return_attention_mask=True,
  return_tensors='pt',
)

_, test_prediction = trained_model(encoding["input_ids"], encoding["attention_mask"])
test_prediction = test_prediction.flatten().numpy()
for label, prediction in zip(ASPECTS, test_prediction):
    print(f"{label}: {prediction}")<jupyter_output>price: 0.1331108808517456
speed: 0.19577057659626007
reliability: 0.15056845545768738
coverage: 0.10842109471559525
customer service: 0.09471900761127472<jupyter_text>Doesn't seem to do a good job distinguishing the different aspects for this case.  (ii) Evaluate model performance on the validation set<jupyter_code>device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

trained_model = trained_model.to(device)

#Prepare the validation set
val_dataset = Generate_PyTorch_Dataset(
    val_df,
    TOKENIZER,
)

#Lists to store the model predictions and the true labels
model_preds = []
true_labels = []

for item in tqdm(val_dataset):
    _, pred = trained_model(
        item["input_ids"].unsqueeze(dim=0).to(device),
        item["attention_mask"].unsqueeze(dim=0).to(device)
    )
    model_preds.append(pred.flatten())
    true_labels.append(item["labels"].int())

model_preds = torch.stack(model_preds).detach().cpu()
true_labels = torch.stack(true_labels).detach().cpu()
auroc_metric = AUROC(pos_label=1)
print("AUROC per tag")
for i, name in enumerate(ASPECTS):
    tag_auroc = auroc_metric(model_preds[:, i], true_labels[:, i])
    print(f"{name}: {tag_auroc}")<jupyter_output>AUROC per tag
price: 0.7131783366203308
speed: 0.5746606588363647
reliability: 0.7471264004707336
coverage: 0.6379525661468506
customer service: 0.9108231663703918<jupyter_text>Price AUROC is extremely low  Classification Report<jupyter_code>y_pred = model_preds.numpy()
y_true = true_labels.numpy()

y_pred = np.where(y_pred > 0.3, 1, 0)

print(classification_report(
  y_true,
  y_pred,
  target_names=ASPECTS,
  zero_division=0
))<jupyter_output>precision    recall  f1-score   support

           price       0.24      0.42      0.30        12
           speed       0.33      0.31      0.32        13
     reliability       0.12      0.18      0.15        11
        coverage       0.33      0.11      0.17         9
customer service       0.50      0.75      0.60        16

       micro avg       0.32      0.39      0.35        61
       macro avg       0.31      0.35      0.31        61
    weighted avg       0.32      0.39      0.34        61
     samples avg       0.08      0.16      0.10        61<jupyter_text>- Oops. It turns out the support for price is lower than expected. I might have made a mistake with the labelling – I did- Model seems to overfit on dominant classes. Reweighting the loss function might be a good thing to try.  Optimize the threshold<jupyter_code>#Get the true values and model probs
y_pred = model_preds
y_true = true_labels

#Initialize a threshold and optimal F2 value
optimal_thresh = (0,0)

#Iterate through the different thresholds
for threshold in np.arange(0.01,0.6,0.01):
    
    #Define the FBeta metric
    f_beta = FBeta(num_classes=5, beta=0.5, average='weighted', threshold = threshold)
    
    #Compute the FBeta score
    score = f_beta(y_pred,y_true)
    
    #If greater than current optimal FBeta score
    if score > optimal_thresh[1]:
        
        #Replace the threshold
        optimal_thresh = (threshold,score)
        
optimal_thresh<jupyter_output><empty_output><jupyter_text>---  7. Modeling - Aspect Extraction v2**`Reweighting BCE Loss Function with ENS weighting`**  Redefine the model<jupyter_code>class ISP_TweetAspectClassifier(pl.LightningModule):
    
    #Set the aspect classifier
    def __init__(self, 
                 n_classes=6, 
                 samples_per_cls = None,
                 n_training_steps=None, 
                 n_warmup_steps=None, 
                 lr=2e-5):
        
        super().__init__()
        
        #Training parameters
        self.lr = lr
        self.n_warmup_steps = n_warmup_steps
        self.n_training_steps = n_training_steps
        
        #Model and model layers
        self.bert = BERTWEET_MODEL
        self.classifier = torch.nn.Linear(self.bert.config.hidden_size, n_classes)
        
        #If other param for ENS provided, reweight loss function to account for imbalance
        if samples_per_cls:
            self.class_weights = self.get_ens_weights(n_classes, samples_per_cls)
            self.criterion = torch.nn.BCELoss(weight = self.class_weights)
        else:
            self.criterion = torch.nn.BCELoss()
        
    def get_ens_weights(self, no_of_classes,samples_per_cls,beta=0.99):
    
        """
        Compute class weights using effective number of samples strategy
        """

        effective_num = 1.0 - np.power(beta, samples_per_cls)
        weights = (1.0 - beta) / np.array(effective_num)
        weights = weights / np.sum(weights) * no_of_classes
        weights = torch.tensor(weights.astype(np.float32))
        
        return weights
    
        
    def forward(self, input_ids, attention_mask, labels = None):
        output = self.bert(input_ids, attention_mask=attention_mask)
        output = self.classifier(output.pooler_output)
        output = torch.sigmoid(output)
        
        loss = 0
        
        if labels is not None:
            loss = self.criterion(output, labels)
            
        return loss, output
    
    def training_step(self, batch, batch_idx):
        
        input_ids = batch["input_ids"]
        attention_mask = batch["attention_mask"]
        labels = batch["labels"]
        
        loss, outputs = self(input_ids, attention_mask, labels)
        self.log("train_loss", loss, prog_bar=True, logger=True)
        
        return {"loss": loss, "predictions": outputs.detach(), "labels": labels}
    
    def validation_step(self, batch, batch_idx):
        input_ids = batch["input_ids"]
        attention_mask = batch["attention_mask"]
        labels = batch["labels"]
        
        loss, outputs = self(input_ids, attention_mask,labels)
        self.log("val_loss", loss, prog_bar=True, logger=True)
        
        return loss
    
    
    def test_step(self, batch, batch_idx):
        input_ids = batch["input_ids"]
        attention_mask = batch["attention_mask"]
        labels = batch["labels"]
        
        loss, outputs = self(input_ids, attention_mask, labels)
        self.log("test_loss", loss, prog_bar=True, logger=True)
        
        return loss
    
    
    def training_epoch_end(self, outputs):
            
        #List to store the true labels and the model's predictions
        labels = []
        predictions = []
        
        #Iterate through all the outputs and get the true vs. predicted label
        for output in outputs:
            
            for label in output["labels"].detach().cpu():
                labels.append(label)
            for pred in output["predictions"].detach().cpu():
                predictions.append(pred)
        
        #Stack the tensors
        labels = torch.stack(labels).int()
        predictions = torch.stack(predictions)
        
        #Record the AUROC for each aspect after each training epoch
        for idx, name in enumerate(ASPECTS):
            metric = AUROC()
            class_roc_auc = metric(predictions[:,idx], labels[:,idx])
            self.logger.experiment.add_scalar(f"{name}_roc_auc/Train", class_roc_auc, self.current_epoch)
        
        #Calculate the micro-F1
        f1_metric = F1()
        micro_f1 = f1_metric(predictions, labels)
        self.logger.experiment.add_scalar(f"micro_f1_score", micro_f1, self.current_epoch)
   
    
    def configure_optimizers(self):
        
        optimizer = AdamW(self.parameters(), lr = self.lr)
        
        scheduler = get_linear_schedule_with_warmup(
            optimizer,
            num_warmup_steps=self.n_warmup_steps,
            num_training_steps=self.n_training_steps)
        
          
        return {'optimizer': optimizer, 'lr_scheduler':{'scheduler':scheduler,'interval':'step'}}
    
#Define variables
N_EPOCHS = 30
STEPS_PER_EPOCH =len(train_df)//TRAIN_BATCH_SIZE
TOTAL_TRAIN_STEPS = STEPS_PER_EPOCH * N_EPOCHS
N_WARMUP_STEPS = TOTAL_TRAIN_STEPS // 4
SAMPLES_PER_LABEL = processed_aspect_df[ASPECTS].sum().tolist()

#Instantiate the classifier model
model = ISP_TweetAspectClassifier(samples_per_cls  = SAMPLES_PER_LABEL,
                                  n_training_steps = TOTAL_TRAIN_STEPS,
                                  n_warmup_steps   = N_WARMUP_STEPS)<jupyter_output><empty_output><jupyter_text>d. Training<jupyter_code>#Setup callback to perform saves during training
checkpoint_callback = ModelCheckpoint(
  dirpath    = "../models/absa-aspect-extraction/bertweet",
  filename   = "ae-{epoch:02d}-{val_loss:.2f}",
  save_top_k = 3, #Save the top 3 models
  verbose    = True,
  monitor    = "val_loss",
  mode       = "min" #Minimize val loss
)

#Log progress in Tensorboard
logger = TensorBoardLogger("../models/absa-aspect-extraction/bertweet/lightning_logs", name = "isp-tweets")
trainer = pl.Trainer(
    #gpus                       = 1,
    logger                      = logger,
    log_every_n_steps           = 10,
    callbacks                   = [checkpoint_callback],
    max_epochs                  =  N_EPOCHS,
    progress_bar_refresh_rate   =  30
)

trainer.fit(model, data_module)<jupyter_output>GPU available: False, used: False
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs

  | Name       | Type         | Params
--------------------------------------------
0 | bert       | RobertaModel | 134 M 
1 | classifier | Linear       | 4.6 K 
2 | criterion  | BCELoss      | 0     
--------------------------------------------
134 M     Trainable params
0         Non-trainable params
134 M     Total params
539.618   Total estimated model params size (MB)<jupyter_text>e. Model Evaluation<jupyter_code>#Evaluate the model's performance on the test dataset
trainer.test(model,data_module)
# Visualize the logs using tensorboard.
%load_ext tensorboard
%tensorboard --logdir ../models/absa-aspect-extraction/bertweet/lightning_logs/
#%reload_ext tensorboard<jupyter_output>The tensorboard extension is already loaded. To reload it, use:
  %reload_ext tensorboard<jupyter_text>Micro F1 is horrible, although this totally depends on the threshold  (i) Evaluate model performance on a generated example<jupyter_code>#Load the best model based on validation loss
trained_model = ISP_TweetAspectClassifier.load_from_checkpoint(
    trainer.checkpoint_callback.best_model_path,
    n_classes=len(ASPECTS),
    samples_per_cls=SAMPLES_PER_LABEL
    
)

trained_model
#Put model into evaluation mode
trained_model.eval()
trained_model.freeze()
test_comment = "The internet is so slow and it's so expensive"

encoding = TOKENIZER.encode_plus(
  test_comment,
  add_special_tokens=True,
  max_length=TOKENIZER.model_max_length,
  return_token_type_ids=False,
  padding="max_length",
  return_attention_mask=True,
  return_tensors='pt',
)

_, test_prediction = trained_model(encoding["input_ids"], encoding["attention_mask"])
test_prediction = test_prediction.flatten().numpy()
for label, prediction in zip(ASPECTS, test_prediction):
    print(f"{label}: {prediction}")<jupyter_output>price: 0.10340679436922073
speed: 0.21158915758132935
reliability: 0.16090767085552216
coverage: 0.11477973312139511
customer service: 0.18621541559696198
trustworthiness: 0.04301103204488754<jupyter_text>Doesn't seem to do a good job distinguishing the different aspects for this case.  (ii) Evaluate model performance on the validation set<jupyter_code>device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

trained_model = trained_model.to(device)

#Prepare the validation set
val_dataset = Generate_PyTorch_Dataset(
    val_df,
    TOKENIZER,
)

#Lists to store the model predictions and the true labels
model_preds = []
true_labels = []

for item in tqdm(val_dataset):
    _, pred = trained_model(
        item["input_ids"].unsqueeze(dim=0).to(device),
        item["attention_mask"].unsqueeze(dim=0).to(device)
    )
    model_preds.append(pred.flatten())
    true_labels.append(item["labels"].int())

model_preds = torch.stack(model_preds).detach().cpu()
true_labels = torch.stack(true_labels).detach().cpu()
#Compute the accuracy on the validation set
acc_metric = Accuracy()
acc_metric(model_preds, true_labels)<jupyter_output><empty_output><jupyter_text>We get a fairly high accuracy on the validation set. However, we note that the dataset is quite imbalanced. So it would be important to also check the model's performance on the different aspects (remember: trustworthiness had very few samples)<jupyter_code>auroc_metric = AUROC(pos_label=1)
print("AUROC per tag")
for i, name in enumerate(ASPECTS):
    tag_auroc = auroc_metric(model_preds[:, i], true_labels[:, i])
    print(f"{name}: {tag_auroc}")<jupyter_output>AUROC per tag
price: 0.619623601436615
speed: 0.6294116973876953
reliability: 0.7241379618644714
coverage: 0.7284946441650391
customer service: 0.8817204236984253
trustworthiness: 0.47278910875320435<jupyter_text>Classification Report<jupyter_code>y_pred = model_preds.numpy()
y_true = true_labels.numpy()

y_pred = np.where(y_pred > 0.15, 1, 0)

print(classification_report(
  y_true,
  y_pred,
  target_names=ASPECTS,
  zero_division=0
))
ASPECTS
x = 11
print(val_df.iloc[x,:].Text)
print(model_preds.numpy()[x])
print(y_true[x])<jupyter_output>ngcomcommission consumersncc spectranet_ng waiting, responded that i should call back. my question is how did 8gb got exhausted without heavy downloads or streaming? i demand an answer, this was the same reason i stopped using my spectranet account since august last year.
[0.14580019 0.21396242 0.14961557 0.11340807 0.24158876 0.04609388]
[0 0 0 0 0 1]<jupyter_text>Still predicting dominant classes significantly more. Next we try oversampling the minority classes ---  7. Modeling - Aspect Extraction v3**`Oversampling minority classes`**<jupyter_code>df_sets = [('Training',train_df),('Validation', val_df),('Test',test_df)]
    
plt.figure(figsize = (12,6))
for idx,df in enumerate(df_sets):
    
    plt.subplot(1,3,idx+1)
    plt.tight_layout()

    with plt.style.context(['notebook','no-latex','grid']):
        
        plt.title(df[0])
        df[1][ASPECTS].sum().plot(kind = "bar", color = 'cornflowerblue')
        plt.xticks(rotation=90)
        
        if idx == 0:
            plt.ylabel('Number of tags in the tweets', fontsize=16)
        
    
plt.show()
#Make a copy of the training set for oversampling
train_oversampled = train_df.copy()<jupyter_output><empty_output><jupyter_text>a. Oversample minority classes**Note:** I had individually run it and know how the distributions turn out. The function below was done to condense the oversampling and save space in the notebook<jupyter_code>for aspect in ['trustworthiness', 'price', 'coverage', 'reliability']:
    
    not_aspect = train_oversampled.query(f" {aspect} == 0 ")
    aspect_tweets = train_oversampled.query(f" {aspect} == 1 ")

    #Oversample aspect tweets to size of class with maximum number of observations
    #and create new training dataset
    train_oversampled = pd.concat([
      not_aspect,
      aspect_tweets .sample(max(train_df[ASPECTS].sum().tolist()), replace=True)
    ])
    
#Plot new distribution
train_oversampled[ASPECTS].sum().plot(kind = "bar", color = 'cornflowerblue')
plt.show()<jupyter_output><empty_output><jupyter_text>b. Oversample tweets with no aspects<jupyter_code>#Get rows with atleast one aspect tagged and rows with no aspect tagged
oversampled_aspects_tagged = train_oversampled[train_oversampled[ASPECTS].sum(axis=1) > 0]
oversampled_no_aspects = train_oversampled[train_oversampled[ASPECTS].sum(axis=1) == 0]

#Oversample the rows with no tagged aspects and then shuffle the dataframe
train_oversampled = pd.concat([
  oversampled_aspects_tagged,
  oversampled_no_aspects.sample(len(oversampled_aspects_tagged), replace=True)
]).sample(frac=1)

#Check that the counts are balanced now
print(f"No aspects count: {len(train_oversampled[train_oversampled[ASPECTS].sum(axis=1) == 0])}.",
      f"Aspects count: {len(train_oversampled[train_oversampled[ASPECTS].sum(axis=1) > 0])}")

print(f"Train size: {train_oversampled.shape}")<jupyter_output>No aspects count: 242. Aspects count: 242
Train size: (484, 8)<jupyter_text>d. Setup the data module with the oversample training set<jupyter_code>#Instantiate and set up the data_module
data_module2 = PyTorchDataModule(train_oversampled,val_df,test_df,TOKENIZER, TRAIN_BATCH_SIZE, TEST_BATCH_SIZE)
data_module2.setup()<jupyter_output><empty_output><jupyter_text>e. Define the model<jupyter_code>class ISP_TweetAspectClassifier3(pl.LightningModule):
    
    #Set the aspect classifier
    def __init__(self, n_classes=6, n_training_steps=None, n_warmup_steps=None, lr=2e-5):
        
        super().__init__()
        self.lr = lr
        self.n_classes = n_classes
        self.n_warmup_steps = n_warmup_steps
        self.n_training_steps = n_training_steps
        self.bert = BERTWEET_MODEL
        self.classifier = torch.nn.Linear(self.bert.config.hidden_size, n_classes)
        self.criterion = torch.nn.BCELoss()
        
    def forward(self, input_ids, attention_mask, labels = None):
        output = self.bert(input_ids, attention_mask=attention_mask)
        output = self.classifier(output.pooler_output)
        output = torch.sigmoid(output)
        
        loss = 0
        
        if labels is not None:
            loss = self.criterion(output, labels)
            
        return loss, output
    
    def training_step(self, batch, batch_idx):
        
        input_ids = batch["input_ids"]
        attention_mask = batch["attention_mask"]
        labels = batch["labels"]
        
        loss, outputs = self(input_ids, attention_mask, labels)
        self.log("train_loss", loss, prog_bar=True, logger=True)
        
        return {"loss": loss, "predictions": outputs.detach(), "labels": labels}
    
    def validation_step(self, batch, batch_idx):
        input_ids = batch["input_ids"]
        attention_mask = batch["attention_mask"]
        labels = batch["labels"]
        
        loss, outputs = self(input_ids, attention_mask,labels)
        self.log("val_loss", loss, prog_bar=True, logger=True)
        
        return loss
    
    
    def test_step(self, batch, batch_idx):
        input_ids = batch["input_ids"]
        attention_mask = batch["attention_mask"]
        labels = batch["labels"]
        
        loss, outputs = self(input_ids, attention_mask, labels)
        self.log("test_loss", loss, prog_bar=True, logger=True)
        
        return loss
    
    
    def training_epoch_end(self, outputs):
            
        #List to store the true labels and the model's predictions
        labels = []
        predictions = []
        
        #Iterate through all the outputs and get the true vs. predicted label
        for output in outputs:
            
            for label in output["labels"].detach().cpu():
                labels.append(label)
            for pred in output["predictions"].detach().cpu():
                predictions.append(pred)
        
        #Stack the tensors
        labels = torch.stack(labels).int()
        predictions = torch.stack(predictions)
        
        #Record the AUROC for each aspect after each training epoch
        for idx, name in enumerate(ASPECTS):
            metric = AUROC()
            class_roc_auc = metric(predictions[:,idx], labels[:,idx])
            self.logger.experiment.add_scalar(f"{name}_roc_auc/Train", class_roc_auc, self.current_epoch)
        
        #Calculate the overall AUROC - not sure about this computation
        micro_auroc_setup = AUROC(num_classes = self.n_classes, pos_label = 1, average='micro')
        micro_auroc = micro_auroc_setup(predictions, labels)
        self.logger.experiment.add_scalar(f"micro_auroc", micro_auroc, self.current_epoch)
   
    
    def configure_optimizers(self):
        
        optimizer = AdamW(self.parameters(), lr = self.lr)
        
        scheduler = get_linear_schedule_with_warmup(
            optimizer,
            num_warmup_steps=self.n_warmup_steps,
            num_training_steps=self.n_training_steps)
        
          
        return {'optimizer': optimizer, 'lr_scheduler':{'scheduler':scheduler,'interval':'step'}}
    
#Define variables
N_EPOCHS = 20
STEPS_PER_EPOCH =len(train_df)//TRAIN_BATCH_SIZE
TOTAL_TRAIN_STEPS = STEPS_PER_EPOCH * N_EPOCHS
N_WARMUP_STEPS = TOTAL_TRAIN_STEPS // 4

#Instantiate the classifier model
model3 = ISP_TweetAspectClassifier3(n_training_steps = TOTAL_TRAIN_STEPS,
                                   n_warmup_steps   = N_WARMUP_STEPS)<jupyter_output><empty_output><jupyter_text>d. Training<jupyter_code>#Setup callback to perform saves during training
checkpoint_callback3 = ModelCheckpoint(
  dirpath    = "../models/absa-aspect-extraction/bertweet_oversampled",
  filename   = "ae-{epoch:02d}-{val_loss:.2f}",
  save_top_k = 3, #Save the top 3 models
  verbose    = True,
  monitor    = "val_loss",
  mode       = "min" #Minimize val loss
)

#Log progress in Tensorboard
logger3 = TensorBoardLogger("../models/absa-aspect-extraction/bertweet_oversampled/lightning_logs", name = "isp-tweets")
trainer3 = pl.Trainer(
    #gpus                       = 1,
    logger                      = logger3,
    log_every_n_steps           = 10,
    callbacks                   = [checkpoint_callback3],
    max_epochs                  =  N_EPOCHS,
    progress_bar_refresh_rate   =  30
)

trainer3.fit(model3, data_module2)<jupyter_output>GPU available: False, used: False
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs

  | Name       | Type         | Params
--------------------------------------------
0 | bert       | RobertaModel | 134 M 
1 | classifier | Linear       | 4.6 K 
2 | criterion  | BCELoss      | 0     
--------------------------------------------
4.6 K     Trainable params
134 M     Non-trainable params
134 M     Total params
539.618   Total estimated model params size (MB)<jupyter_text>Already very skeptical about this model since the loss did not decrease significantly  e. Model Evaluation<jupyter_code>#Evaluate the model's performance on the test dataset
trainer3.test(model3,data_module2)
# Visualize the logs using tensorboard.
%load_ext tensorboard
%tensorboard --logdir ../models/absa-aspect-extraction/bertweet_oversampled/lightning_logs/
#%reload_ext tensorboard<jupyter_output>The tensorboard extension is already loaded. To reload it, use:
  %reload_ext tensorboard<jupyter_text>Micro AUROC (around 0.5) is bad. Suggests random guessing at best – i.e. poor classifier  (i) Evaluate model performance on a generated example<jupyter_code>#Load the best model based on validation loss
trained_model3 = ISP_TweetAspectClassifier3.load_from_checkpoint(
    trainer3.checkpoint_callback.best_model_path,
    n_classes=len(ASPECTS),
    
)
#Put model into evaluation mode
trained_model3.eval()
trained_model3.freeze()
test_comment = "The internet is so slow and it's so expensive"

encoding = TOKENIZER.encode_plus(
  test_comment,
  add_special_tokens=True,
  max_length=TOKENIZER.model_max_length,
  return_token_type_ids=False,
  padding="max_length",
  return_attention_mask=True,
  return_tensors='pt',
)

_, test_prediction = trained_model3(encoding["input_ids"], encoding["attention_mask"])
test_prediction = test_prediction.flatten().numpy()
for label, prediction in zip(ASPECTS, test_prediction):
    print(f"{label}: {prediction}")<jupyter_output>price: 0.4770582616329193
speed: 0.4313480257987976
reliability: 0.41877180337905884
coverage: 0.36569344997406006
customer service: 0.41532450914382935
trustworthiness: 0.34830161929130554<jupyter_text>Doesn't seem to do a good job distinguishing the different aspects for this case. Although the values are finally higher! Real difference seems to have been increased training sample size. Or increased samples for the different labels  (ii) Evaluate model performance on the validation set<jupyter_code>device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

trained_model3 = trained_model3.to(device)

#Prepare the validation set
val_dataset = Generate_PyTorch_Dataset(
    val_df,
    TOKENIZER,
)

#Lists to store the model predictions and the true labels
model_preds = []
true_labels = []

for item in tqdm(val_dataset):
    _, pred = trained_model3(
        item["input_ids"].unsqueeze(dim=0).to(device),
        item["attention_mask"].unsqueeze(dim=0).to(device)
    )
    model_preds.append(pred.flatten())
    true_labels.append(item["labels"].int())

model_preds = torch.stack(model_preds).detach().cpu()
true_labels = torch.stack(true_labels).detach().cpu()
#Compute the accuracy on the validation set
acc_metric = Accuracy()
acc_metric(model_preds, true_labels)<jupyter_output><empty_output><jupyter_text>We get a fairly high accuracy on the validation set. However, we note that the dataset is quite imbalanced. So it would be important to also check the model's performance on the different aspects (remember: trustworthiness had very few samples)<jupyter_code>auroc_metric = AUROC(pos_label=1)
print("AUROC per tag")
for i, name in enumerate(ASPECTS):
    tag_auroc = auroc_metric(model_preds[:, i], true_labels[:, i])
    print(f"{name}: {tag_auroc}")<jupyter_output>AUROC per tag
price: 0.6263440847396851
speed: 0.5874999165534973
reliability: 0.47454842925071716
coverage: 0.4448924958705902
customer service: 0.8682795763015747
trustworthiness: 0.5748299956321716<jupyter_text>Classification Report<jupyter_code>y_pred = model_preds.numpy()
y_true = true_labels.numpy()

y_pred = np.where(y_pred > 0.43, 1, 0)

print(classification_report(
  y_true,
  y_pred,
  target_names=ASPECTS,
  zero_division=0
))<jupyter_output>precision    recall  f1-score   support

           price       0.08      1.00      0.15         8
           speed       0.13      0.19      0.15        16
     reliability       0.11      0.14      0.12        14
        coverage       0.09      0.12      0.11         8
customer service       0.36      0.62      0.45         8
 trustworthiness       0.00      0.00      0.00         3

       micro avg       0.11      0.33      0.17        57
       macro avg       0.13      0.35      0.16        57
    weighted avg       0.14      0.33      0.17        57
     samples avg       0.08      0.17      0.10        57
{'fraction_non_alphanumeric': 0.09748287823534643, 'fraction_numerical': 0.028284265765365114, 'mean_word_length': 3.1457411504424777, 'pattern_counts': {'":': 9, '<': 124, '<?xml version=': 0, '>': 131, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 21, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '36', 'dir': 'jupyter-structured-clean-dedup', 'id': '151030', 'license': 'MIT', 'n_tokens_mistral': 15125, 'n_tokens_neox': 14091, 'n_words': 3415, 'path': 'notebooks/6.2.1-modeling-absa(ae)-multi-label-classifier.ipynb', 'repo_name': 'KoredeAkande/nigerian_isp_sentiment_analysis', 'size': '44972'}