<jupyter_start><jupyter_text>Dive into Abusive Language with SnorkelAuthor: <NAME> Updated: 2021-08-02----------  Transfer Learning with BERT ClassifierThe output of the Snorkel `LabelModel` is a set of labels that can be used with most popular supervised learning models such as Logistic Regression, XGBoost, and neural networks.Google’s BERT (Bidirectional Encoder Representations from Transformers, 2018) combines context embedding (ELMO) and several Transformers, plus it’s bidirectional (which was a big novelty for Transformers). The vector BERT assigns to a word is a function of the entire sentence, therefore, a word can have different vectors based on the contexts. In order to complete a text classification task, you can use BERT in 3 different ways:* Train it all from scratches and use it as classifier* Extract the word embeddings and use them in an embedding layer (like I did with Word2Vec)* Fine-tune the pre-trained model (transfer learning)**We will focus on transfer learning to maximize the benefits of Snorkel.**<jupyter_code># Imports and setup for Google Colab

# Mount Google Drive
from google.colab import drive ## module to use Google Drive with Python
drive.mount('/content/drive') ## mount to access contents

# Install python libraries
! pip install --upgrade tensorflow --quiet
! pip install snorkel --quiet
! pip install tensorboard==1.15.0 --quiet
! pip install transformers --quiet
# Imports for data and plotting
import numpy as np
import pandas as pd

import matplotlib.pyplot as plt
plt.style.use('fivethirtyeight')
%matplotlib inline 
import seaborn as sns

import pickle
import os
import re

# Imports for snorkel analysis and multi-task learning
from snorkel.labeling.model import LabelModel
from snorkel.labeling import filter_unlabeled_dataframe

# Imports for bert language model
from sklearn.model_selection import train_test_split
from sklearn.utils.class_weight import compute_class_weight
from sklearn import metrics

import transformers

import torch
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
from torch.utils.data import TensorDataset, DataLoader
from torch.utils.data import RandomSampler, SequentialSampler

import time
import datetime
import random
# Load labeled dataset for training 
df_train = pd.read_pickle('../data/processed/df_train.pkl')
df_train.reset_index(drop=True, inplace=True)

df_valid = pd.read_pickle('../data/processed/df_valid.pkl')
df_valid.reset_index(drop=True, inplace=True)

df_test = pd.read_pickle('../data/processed/df_test.pkl')
df_test.reset_index(drop=True, inplace=True)

pd.set_option('display.max_colwidth', 500)
df_train.head()
# Load matrix of votes obtained from all the labeling functions
l_train = pd.read_pickle('../models/lf_train_final.pkl')
l_valid = pd.read_pickle('../models/lf_valid_final.pkl')
l_test = pd.read_pickle('../models/lf_test_final.pkl')<jupyter_output><empty_output><jupyter_text>Snorkel Label Model`Majority Vote` is the simpliest function Snorkel uses to generate a class label, based on the most voted label from all of the labeling functions. However, in order to take advantage of Snorkel's full functionality, `Label Model` predicts new class labels by learning to estimate the accuracy and correlations of the labeling functions based on their conflicts and overlaps. We will use the `Label Model` function to generate a single confidence-weighted label given a matrix of predictions.<jupyter_code># Apply LabelModel, no gold labels for training
label_model_train = LabelModel(cardinality=2, verbose=True) ## number of classes     
label_model_train.fit(L_train=l_train,
                      n_epochs=2000, 
                      log_freq=100, 
                      seed=42)

# Get our predictions for the unlabeled dataset
probs_train = label_model_train.predict(L=l_train)
# Calculate Label Model accuracy using validation data
label_model_acc = label_model_train.score(
    L=l_valid, Y=df_valid.label, tie_break_policy='random'
    )['accuracy']

print(f"{'Label Model Accuracy:': <25} {label_model_acc * 100:.1f}%")
# Filter out unlabeled data points
df_train_filtered, probs_train_filtered = filter_unlabeled_dataframe(
    X=df_train, y=probs_train, L=l_train
    )

print('Shape of X_train: {} Shape of X_train_probs: {}'.format(
    df_train_filtered.shape, 
    probs_train_filtered.shape
    )
) # Confirm 100% coverage from labeling functions
# Explore conflicts between gold labels and LM labels
df_train_filtered['lm_label'] = probs_train_filtered
df_train_filtered.loc[
    df_train_filtered['lm_label'] != df_train_filtered['label'], :].head(50)<jupyter_output><empty_output><jupyter_text>Train a BERT ClassifierBERT provides a Universal Language Model, pre-trained on unlabeled data extracted from non-Twitter sources:1.   BooksCorpus with 800M words2.   English Wikipedia with 2,500M wordsWe will fine-tune the Language Model for the task at hand with a large corpus of unlabeled data points. We will then train a discriminative classifier by fine-tuning it with gradual unfreezing.<jupyter_code># Fine tune the Language Model using Twitter Sentiment140 dataset from Kaggle
# BERT will then be able to generalize in the Twitter domain

df_twitter = pd.read_pickle('../data/processed/twitter_sentiment140.pkl')
df_twitter['target'] = df_twitter['target'].map({0: 1, 4: 0})
df_twitter.head()
# Split Twitter Sentiment140 dataset into train and validation
# No need for test dataset for fine-tuning

X_ts140_train, X_ts140_valid, y_ts140_train, y_ts140_valid = train_test_split(
    df_twitter['tweet'], 
    df_twitter['target'], 
    random_state=42, 
    test_size=0.3, 
    stratify=df_twitter['target']
    )
# Padding will add tokens up to a max_length
# Padding tokens to not help the model learn anything useful (slower training)
seq_len = [len(i.split()) for i in df_twitter.tweet]

fig,ax = plt.subplots()

ax.hist(seq_len, bins=30, label='histogram')
ax.set_ylabel('Frequency')
ax.set_xlabel('Tweet Word Count')

plt.show(); # Most tweets have about 30 words or less
# Create BERT tokenizer (original BERT of 110M parameters)
# BERT tokenizer can handle punctuation, simleys, etc.
# Previously replaced mentions and urls with special tokens (#has_url, #has_mention)

bert_token = transformers.BertTokenizerFast.from_pretrained(
    'bert-base-uncased', 
    do_lower_case=True) 

# Create helper function for text parsing

def bert_encode(tweet_df, tokenizer):
    ## add '[CLS]' token as prefix to flag start of text
    ## append '[SEP]' token to flag end of text
    ## append '[PAD]' token to fill uneven text
    bert_tokens = tokenizer.batch_encode_plus(
        tweet_df['tweet'].to_list(),
        padding='max_length', 
        truncation=True,
        max_length=30
        )
    
    ## convert list to tensors
    input_word_ids = torch.tensor(bert_tokens['input_ids'])
    input_masks = torch.tensor(bert_tokens['attention_mask'])
    input_type_ids = torch.tensor(bert_tokens['token_type_ids'])

    inputs = {
        'input_word_ids': input_word_ids,
        'input_masks': input_masks,
        'input_type_ids': input_type_ids
        }

    return inputs
'''
# Save the BERT encodes for training and testing
SAVE_PATH = '../models/'
os.chdir(SAVE_PATH)

# Apply BERT encoding 
X_ts140_train_bert = bert_encode(
    pd.DataFrame(X_ts140_train, columns=['tweet']), 
    bert_token
    )

model_name1 = 'bert_tokens_ts140_xtrain.pkl'
with open(model_name1, 'wb') as file:
    pickle.dump(X_ts140_train_bert, file)

X_ts140_test_bert = bert_encode(
    pd.DataFrame(X_ts140_test, columns=['tweet']), 
    bert_token)

model_name2 = 'bert_tokens_ts140_xtest.pkl'
with open(model_name2, 'wb') as file:
    pickle.dump(X_ts140_test_bert, file)
'''
'''
# Apply BERT encoding 
X_train = bert_encode(df_train, bert_token)
y_train = df_train.label.values

# Save the BERT encodes for training
model_name3 = 'bert_tokens_xtrain.pkl'
with open(model_name3, 'wb') as file:
    pickle.dump(X_train, file)

X_valid = bert_encode(df_valid, bert_token)
y_valid = df_valid.label.values

model_name4 = 'bert_tokens_xvalid.pkl'
with open(model_name4, 'wb') as file:
    pickle.dump(X_valid, file)

X_test = bert_encode(df_test, bert_token)
y_test = df_test.label.values

model_name5 = 'bert_tokens_xtest.pkl'
with open(model_name5, 'wb') as file:
    pickle.dump(X_test, file)
'''
# Load BERT encodes
SAVE_PATH = '../models/'
os.chdir(SAVE_PATH)

X_ts140_train_bert = pd.read_pickle('bert_tokens_ts140_xtrain.pkl')
X_ts140_valid_bert = pd.read_pickle('bert_tokens_ts140_xvalid.pkl')
# Create data loaders for train and test set
# Pass batches of train data and test data as input to the model during training

batch_size = 32 ## define batch size, authors recommend 16 or 32 for fine-tuning

# Wrap tensors for train
ts140_train_data = TensorDataset(
    X_ts140_train_bert['input_word_ids'], 
    X_ts140_train_bert['input_masks'], 
    torch.tensor(y_ts140_train.to_list())
    )

# Make sampler for sampling the data during training
ts140_train_sampler = RandomSampler(ts140_train_data)

# Make dataLoader for train set
ts140_train_dataloader = DataLoader(
    ts140_train_data, 
    sampler=ts140_train_sampler, 
    batch_size=batch_size
    )

# Wrap tensors for valid
ts140_valid_data = TensorDataset(
    X_ts140_valid_bert['input_word_ids'], 
    X_ts140_valid_bert['input_masks'], 
    torch.tensor(y_ts140_valid.to_list())
    )

# Make sampler for sampling the data during training
ts140_valid_sampler = SequentialSampler(ts140_valid_data)

# Make dataLoader for validation set
ts140_valid_dataloader = DataLoader(
    ts140_valid_data, 
    sampler=ts140_valid_sampler, 
    batch_size=batch_size
    )
# Define model architecture for fine-tuning

# Code adapted from <NAME>
# >>> https://mccormickml.com/2019/07/22/BERT-fine-tuning/#32-required-formatting

nlp_bert = transformers.BertForSequenceClassification.from_pretrained(
    'bert-base-uncased', ## use 12-layer BERT model, uncased vocab
    num_labels=2, ## binary classfication
    output_attentions = False, ## model returns attentions weights
    output_hidden_states = False, ## model returns all hidden-states
    )

nlp_bert.cuda()
# Get all of the model's parameters as a list of tuples

# Code adapted from <NAME>
# >>> https://mccormickml.com/2019/07/22/BERT-fine-tuning/#32-required-formatting

params = list(nlp_bert.named_parameters())

print('The BERT model has {:} different named parameters.\n'.format(len(params)))

print('==== Embedding Layer ====\n')

for p in params[0:5]:
    print("{:<55} {:>12}".format(p[0], str(tuple(p[1].size()))))

print('\n==== First Transformer ====\n')

for p in params[5:21]:
    print("{:<55} {:>12}".format(p[0], str(tuple(p[1].size()))))

print('\n==== Output Layer ====\n')

for p in params[-4:]:
    print("{:<55} {:>12}".format(p[0], str(tuple(p[1].size()))))

# Define optimizer 
optimizer = transformers.AdamW(nlp_bert.parameters(),
                               lr=2e-5, ## learning rate
                               eps=1e-8 # small number prevent divison by zero
                               ) 

epochs = 4 ## define number of training epochs

# Check the class weights for imbalanced classes
class_weights = compute_class_weight(
    'balanced', 
    np.unique(y_ts140_train), 
    y_ts140_train
    )

print("Class Weights:", class_weights)

'''
# Update model architecture to handle imbalanced classes
weights= torch.tensor(class_weights, dtype=torch.float) ## convert to tensor
weights = weights.to(device) ## push to GPU
cross_entropy  = torch.nn.NLLLoss(weight=weights) ## define the loss function
'''
# Define learning rate scheduler
total_steps = len(ts140_train_dataloader) * epochs ## batches X epochs

scheduler = transformers.get_linear_schedule_with_warmup(
    optimizer,
    num_warmup_steps=0,
    num_training_steps=total_steps
)
# Define helper functions for calculating accuracy

# Code adapted from <NAME>
# >>> https://mccormickml.com/2019/07/22/BERT-fine-tuning/#32-required-formatting

# Calculate accuracy for predictions vs labels
def flat_accuracy(preds, labels):
    pred_flat = np.argmax(preds, axis=1).flatten()
    labels_flat = labels.flatten()
    return np.sum(pred_flat == labels_flat) / len(labels_flat)

# Return time in seconds as string hh:mm:ss
def format_time(elapsed):
    ## round to the nearest second
    elapsed_rounded = int(round((elapsed)))
    ## format as hh:mm:ssa
    return str(datetime.timedelta(seconds=elapsed_rounded))
# Run training loop

# Code adapted from <NAME>
# >>> https://mccormickml.com/2019/07/22/BERT-fine-tuning/#32-required-formatting

seed_val = 42

random.seed(seed_val)
np.random.seed(seed_val)
torch.manual_seed(seed_val)
torch.cuda.manual_seed_all(seed_val)

# Store quantities such as training and validation loss, 
# validation accuracy, and timings
training_stats = []

# Measure the total training time for the whole run
total_t0 = time.time()

# Loop each epoch...
for epoch_i in range(0, epochs):
    ## perform one full pass over the training set
    print("")
    print('======== Epoch {:} / {:} ========'.format(epoch_i + 1, epochs))
    print('Training...')

    ## measure how long the training epoch takes.
    t0 = time.time()

    ## reset the total loss for this epoch.
    total_train_loss = 0

    ## put the model into training mode     
    nlp_bert.train()

    ## loop each batch of training data...
    for step, batch in enumerate(ts140_train_dataloader):

        ## update progress every 40 batches
        if step % 40 == 0 and not step == 0:
            ## calculate elapsed time in minutes
            elapsed = format_time(time.time() - t0)
            
            ## report progress
            print('  Batch {:>5,}  of  {:>5,}.    Elapsed: {:}.'.format(
                step, len(ts140_train_dataloader), elapsed))

        ## unpack this training batch from our dataloader
        ## NOTE: `model.to(device) is in-place operation
        ## batch (Tensor) to GPU is not in-place operation
        b_input_ids = batch[0].to('cuda') ## input ids
        b_input_mask = batch[1].to('cuda') ## attention masks
        b_labels = batch[2].to('cuda') ## labels

        ## Clear previously calculated gradients before performing backward pass         
        nlp_bert.zero_grad()        

        ## perform a forward pass (evaluate the model on this training batch)
        loss_logit_dict = nlp_bert(b_input_ids,
                                   token_type_ids=None,
                                   attention_mask=b_input_mask,
                                   labels=b_labels
                                   )
        loss =  loss_logit_dict['loss']
        logits = loss_logit_dict['logits']

        ## accumulate the training loss over all of the batches
        ## calculate the average loss at the end 
        total_train_loss += loss.item()

        ## perform a backward pass to calculate the gradients
        loss.backward()

        ## clip the norm of the gradients to 1.0
        ## prevent the "exploding gradients" problem
        torch.nn.utils.clip_grad_norm_(nlp_bert.parameters(), 1.0)

        ## update parameters and take a step using the computed gradient
        optimizer.step() ## dictates update rule based on gradients, lr, etc.

        ## update the learning rate
        scheduler.step()

    ## calculate the average loss over all of the batches
    avg_train_loss = total_train_loss / len(ts140_train_dataloader)            
    
    ## measure how long this epoch took
    training_time = format_time(time.time() - t0)

    print("")
    print("  Average training loss: {0:.2f}".format(avg_train_loss))
    print("  Training epoch took: {:}".format(training_time))
        
    ## start validation

    print("")
    print("Running Validation...")

    t0 = time.time()

    ## put the model in evaluation mode--the dropout layers behave differently
    nlp_bert.eval()

    ## define tracking variables 
    total_eval_accuracy = 0
    total_eval_loss = 0
    nb_eval_steps = 0

    ## evaluate data for one epoch
    for batch in ts140_valid_dataloader:
        
        ## unpack this training batch from our dataloader
        b_input_ids = batch[0].to('cuda') ## input ids
        b_input_mask = batch[1].to('cuda') ## attention masks
        b_labels = batch[2].to('cuda') ## labels
        
        ## tell pytorch not to bother with constructing the compute graph during
        # the forward pass, since this is only needed for backprop (training)
        with torch.no_grad():        

            ## check forward pass, calculate logit predictions
            loss_logit_dict = nlp_bert(b_input_ids,
                                    token_type_ids=None,
                                    ## segment ids for multiple sentences
                                    attention_mask=b_input_mask,
                                    labels=b_labels
                                    )
            loss =  loss_logit_dict['loss']
            logits = loss_logit_dict['logits']

        ## accumulate the validation loss
        total_eval_loss += loss.item()

        ## move logits and labels to CPU
        logits = logits.detach().cpu().numpy()
        label_ids = b_labels.to('cpu').numpy()

        ## calculate the accuracy for this batch of tweets
        total_eval_accuracy += flat_accuracy(logits, label_ids)
        

    ## report the final accuracy for this validation run
    avg_val_accuracy = total_eval_accuracy / len(ts140_valid_dataloader)
    print("  Accuracy: {0:.2f}".format(avg_val_accuracy))

    ## calculate the average loss over all of the batches
    avg_val_loss = total_eval_loss / len(ts140_valid_dataloader)
    
    ## measure how long the validation run took
    validation_time = format_time(time.time() - t0)
    
    print("  Validation Loss: {0:.2f}".format(avg_val_loss))
    print("  Validation took: {:}".format(validation_time))

    ## record all statistics from this epoch
    training_stats.append(
        {
            'epoch': epoch_i + 1,
            'Training Loss': avg_train_loss,
            'Valid. Loss': avg_val_loss,
            'Valid. Accur.': avg_val_accuracy,
            'Training Time': training_time,
            'Validation Time': validation_time
        }
    )

print("")
print("Training complete!")

print("Total training took {:} (h:mm:ss)".format(
    format_time(time.time()-total_t0)
    )
)

'''
# Save the default names for the model
SAVE_PATH = '../models/'

# Save current state with best parameters
torch.save(nlp_bert.state_dict(), 
           os.path.join(SAVE_PATH, 'model_bert_ts140_statedict.pt')
           )

# Save full model
torch.save(nlp_bert, '../models/model_bert_ts140_full_vf.pt')

# Save checkpoint
torch.save({
            'epoch': epochs,
            'model_state_dict': nlp_bert.state_dict(),
            'optimizer_state_dict': optimizer.state_dict(),
            'loss': loss
            }, '../models/model_bert_ts140_checkpoint_vf.pt')
'''<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09879201680672269, 'fraction_numerical': 0.018119747899159665, 'mean_word_length': 3.5056791292001894, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 35, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '204515', 'license': 'MIT', 'n_tokens_mistral': 6233, 'n_tokens_neox': 5810, 'n_words': 1665, 'path': 'colab_notebooks/ntbk_04a_labeled_finetune_bert_twitter.ipynb', 'repo_name': 'codecaviar/hatespeech2020', 'size': '19083'}