<jupyter_start><jupyter_text>Text PreprocessingBERT tokenizer and text preprocessing from the hugging face library to prepare the data for the BERT classification model. We also look at why the text needs to be pre-processed and used for the BERT classification model<jupyter_code>from google.colab import drive
drive.mount('/content/gdrive')
root_path = "/content/gdrive/MyDrive/Collab-Notebooks/Sentiment-Analysis/"
!pip install transformers
import transformers
from transformers import BertModel, BertTokenizer, AdamW, get_linear_schedule_with_warmup
import torch

import numpy as np
import pandas as pd
import seaborn as sns

from pylab import rcParams

import matplotlib.pyplot as plt
from matplotlib import rc

from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix, classification_report

from collections import defaultdict
from textwrap import wrap
from torch import nn, optim
from torch.utils.data import Dataset, DataLoader

%matplotlib inline
%config InlineBackend.figure_format='retina'

sns.set(style='whitegrid', palette='muted', font_scale=1.2)
HAPPY_COLORS_PALETTE = ["#01BEFE", "#FFDD00", "#FF7D00", "#FF006D", "#ADFF02", "#8F00FF"]
sns.set_palette(sns.color_palette(HAPPY_COLORS_PALETTE))
rcParams['figure.figsize'] = 6,6

RANDOM_SEED = 42
np.random.seed(RANDOM_SEED)
torch.manual_seed(RANDOM_SEED)

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
df = pd.read_csv(root_path+"reviews.csv")
df.head()
df.shape
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 16354 entries, 0 to 16353
Data columns (total 12 columns):
 #   Column                Non-Null Count  Dtype 
---  ------                --------------  ----- 
 0   reviewId              16354 non-null  object
 1   userName              <PASSWORD> non-null  object
 2   userImage             16354 non-null  object
 3   content               16354 non-null  object
 4   score                 16354 non-null  int64 
 5   thumbsUpCount         16354 non-null  int64 
 6   reviewCreatedVersion  13988 non-null  object
 7   at                    16354 non-null  object
 8   replyContent          8740 non-null   object
 9   repliedAt             8740 non-null   object
 10  sortOrder             16354 non-null  object
 11  appId                 16354 non-null  object
dtypes: int64(2), object(10)
memory usage: 1.5+ MB<jupyter_text>To check whether or not our dataset is balanced in terms of the dataset distribution that we have. We can check this with respect to the count of the different scores.The distribution seems reasonable since (1,2) will be grouped and (4,5) will be grouped and 3 is double of that<jupyter_code>sns.countplot(df.score)
plt.xlabel('review score')
def to_sentiment(rating):
  rating = int(rating)
  if rating <=2:
    return 0
  elif rating == 3:
    return 1
  else:
    return 2
df['sentiment'] = df.score.apply(to_sentiment)
class_names = ['negative','neutral','positive']
ax =sns.countplot(df.sentiment)
plt.xlabel('review sentiment')
ax.set_xticklabels(class_names)<jupyter_output>/usr/local/lib/python3.6/dist-packages/seaborn/_decorators.py:43: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  FutureWarning<jupyter_text>Data PreprocessingBERT transformer requires some special work on the text as preprocessing. Pretrained on v large dataset; and we are using it for sequence classification<jupyter_code>#starting with the tokenizer
tokenizer = transformers.BertTokenizer.from_pretrained('bert-base-uncased')<jupyter_output><empty_output><jupyter_text>BERT requires special tokens which indicate separation between sentences.This comes from the fact that BERT was initially built for Q/A systems and whiledoing classification of sequences BERT requires a tokensSeuqences must be of equal length, which means we'll have to do padding which willappend tokens of 0 meaning to the end of our sentences that we pass into the modelWe also require an attention mask - each token that has a meaning should have a valueof 1 in the attention mask, else if it is a padding token attention mask should be 0* Use a small piece of text to show how the tokenizer splits and works with the text* Building a dataset, that will take the reviews, use the tokenizer internally, and build a dataset that is compatible with pytorch<jupyter_code>sample_txt = "When was I last outside? I am stuck at home for 2 weeks."
tokens = tokenizer.tokenize(sample_txt)
print(tokens)
#keeps all the punctuation here
#converting tokens to ids to pass to the bert model later

token_ids = tokenizer.convert_tokens_to_ids(tokens)
print(len(token_ids))
print(len(tokens))
print(token_ids)<jupyter_output>15
15
[2043, 2001, 1045, 2197, 2648, 1029, 1045, 2572, 5881, 2012, 2188, 2005, 1016, 3134, 1012]<jupyter_text>Special Tokens* SEP tokens* Classification token* Padding token* Unknown token for something that is unrecognizedWe use the ecode plus method to create all the steps for tokenization and data pre processing<jupyter_code>tokenizer.sep_token, tokenizer.sep_token_id
tokenizer.cls_token, tokenizer.cls_token_id
tokenizer.pad_token, tokenizer.pad_token_id
tokenizer.unk_token, tokenizer.unk_token_id
encoding = tokenizer.encode_plus(
    sample_txt,
    max_length = 32,
    add_special_tokens=True,
    pad_to_max_length = True,
    return_attention_mask = True,
    return_token_type_ids = False,
    return_tensors = 'pt'
)

encoding.keys()
encoding['input_ids']
len(encoding['input_ids'][0])
encoding['attention_mask']<jupyter_output><empty_output><jupyter_text>Choosing Sequence LengthWe can look at the distribution of the sentences in our token lengths and the choose one ideal oneWe will just use the encode method of the tokenizer here<jupyter_code>token_lens = []

for txt in df.content:
  tokens = tokenizer.encode(txt, max_length=512)
  token_lens.append(len(tokens))
sns.distplot(token_lens)<jupyter_output>/usr/local/lib/python3.6/dist-packages/seaborn/distributions.py:2551: FutureWarning: `distplot` is a deprecated function and will be removed in a future version. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `histplot` (an axes-level function for histograms).
  warnings.warn(msg, FutureWarning)<jupyter_text>Here the max length of 128 seems suitable enough for our distribution of tokens and initially, we can choose a len that may clip some tokens but allows the model to train relatively faster and also helps build a faster prototype for testing  Creating a DATSET<jupyter_code>class GPReviewDataset(Dataset):


  def __init__(self, review, target, tokenizer, max_len):
    self.review = review
    self.target = target
    self.tokenizer = tokenizer
    self.max_len = max_len

  def __len__(self):
    return len(self.review)

  #takes the index of the element from the dataset
  def __getitem__(self, item):
    review = str(self.review[item])
    encoding = tokenizer.encode_plus(
      review,
      max_length = self.max_len,
      add_special_tokens=True,
      pad_to_max_length = True,
      return_attention_mask = True,
      return_token_type_ids = False,
      return_tensors = 'pt'
    )
    return {
        'input_ids' : encoding['input_ids'],
        'attention_mask' : encoding['attention_mask'],
        'targets' : torch.tensor(self.target[item], dtype=torch.long)
    }
  
MAX_LEN = 128
BATCH_SIZE = 8
EPOCHS = 20
#splitting into train, test, val

df_train, df_test = train_test_split(df, test_size=0.2, random_state=RANDOM_SEED)
df_val, df_test = train_test_split(df_test, test_size = 0.5, random_state=RANDOM_SEED)
df_train.shape, df_val.shape, df_test.shape
'''
wrapping all the dataframes into the pytorch dataloaders 
to be able to load and use in the bert pretrained model
'''

def create_data_loader(df, tokenizer, max_len, batch_size):
  ds = GPReviewDataset(
      review = df.content.to_numpy(),
      target = df.sentiment.to_numpy(),
      tokenizer = tokenizer,
      max_len = max_len
  )

  return DataLoader(
      ds,
      batch_size = batch_size,
      num_workers = 4
  )
train_data_loader = create_data_loader(df_train, tokenizer, MAX_LEN, BATCH_SIZE)
val_data_loader = create_data_loader(df_val, tokenizer, MAX_LEN, BATCH_SIZE)
test_data_loader = create_data_loader(df_test, tokenizer, MAX_LEN, BATCH_SIZE)
data = next(iter(train_data_loader))
data.keys()
print(data['input_ids'].shape)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07884637748575415, 'fraction_numerical': 0.02604954064426096, 'mean_word_length': 4.449936628643853, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 27, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '421262', 'license': 'MIT', 'n_tokens_mistral': 2823, 'n_tokens_neox': 2629, 'n_words': 866, 'path': 'text-preprocessing.ipynb', 'repo_name': 'mallika2011/Sentiment-Analysis', 'size': '8594'}