<jupyter_start><jupyter_text>Predicting Search Engine Results Relevance using BERTIn this notebook I'll show you how easy it is to fine tune BERT to predict the relevance of search results and beat a previous state of the art approach.<jupyter_code>from IPython.display import Image
Image(filename='myLateKaggleWinner.SearchRelevance.png')
%load_ext autoreload
%autoreload 2
%matplotlib inline
import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)

import torch
from torch.utils.data import DataLoader
import argparse
import time
from sklearn.metrics import accuracy_score, classification_report
import matplotlib.pyplot as plt
from tqdm import tqdm
import numpy as np
import pandas as pd

from dataloader import SearchRelevanceDataset
from transformers import AdamW
from transformers import BertTokenizer, BertForSequenceClassification
from transformers import get_linear_schedule_with_warmup


device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
device<jupyter_output>I0820 00:31:09.548614 139904012736320 file_utils.py:39] PyTorch version 1.4.0 available.
I0820 00:31:11.283068 139904012736320 file_utils.py:55] TensorFlow version 2.0.0 available.<jupyter_text>To run this notebook, you'll need to agree to the Kaggle license terms and download the data<jupyter_code># the following commented out lines were needed, along with manually editing the column name to convert to utf8 and to avoid a keyword collision
# train_file ='kaggle_ecommerce_complete_Rank_Crowd_AGG_Descriptions_ALL_ROWS.csv'
# train_file_path = f'../data/crowdflower-search-relevance/{train_file}'
# train_df = pd.read_csv(train_file_path, encoding='iso-8859-1' )
# train_df.head()
train_file ='kaggle_ecommerce_complete_Rank_Crowd_AGG_Descriptions_ALL_ROWS_utf8.csv'
train_file_path = f'../data/crowdflower-search-relevance/{train_file}'
# train_df.to_csv(train_file_path, encoding='utf-8', index=False)
# train_df = pd.read_csv(train_file_path)
# train_df.head()
train_df = pd.read_csv(train_file_path)
train_df.head()<jupyter_output><empty_output><jupyter_text>After inspecting the data, we decide to only use values for which we have relevance scores.It is left to the reader as an exercise to determine if imputing the missing values would boost the classifier scores. So much low hanging fruit.<jupyter_code>train_df.query('relevance_score < 0 | relevance_score.isnull()', engine='python').head()<jupyter_output><empty_output><jupyter_text>In another notebook we might predict this nan values and cross validate, and maybe even manually label, etc. There are all sorts of improvements that can be done, but this notebook focuses on how big results can be gained with just fine tuning BERT in a coarse manner.<jupyter_code>len(train_df)
train_df.dropna(subset=['relevance_score'], inplace=True)
len(train_df)<jupyter_output><empty_output><jupyter_text>We'll also collapse the relevance scores into four categories<jupyter_code>train_df['median_relevance'] = [int(round((tmp) )) for tmp in train_df['relevance_score'].tolist()]
train_df.head()                                    
ids = train_df['_unit_id'].tolist()
len(ids), len(set(ids))
titles = train_df['product_title'].tolist()
len(titles), len(set(titles))
len(set( train_df['median_relevance'].tolist()))
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=4)
model.to(device)
device<jupyter_output>I0820 00:31:19.779527 139904012736320 tokenization_utils.py:1015] loading file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt from cache at /home/todd/.cache/torch/transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
I0820 00:31:20.293649 139904012736320 configuration_utils.py:285] loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-config.json from cache at /home/todd/.cache/torch/transformers/4dad0251492946e18ac39290fcfe91b89d370fee250efe9521476438fe8ca185.7156163d5fdc189c3016baca0775ffce230789d7fa2a42ef516483e4ca884517
I0820 00:31:20.294543 139904012736320 configuration_utils.py:321] Model config BertConfig {
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "id2label": {
    "0": "LABEL_0",
    "1": "LAB[...]<jupyter_text>Let's take a look at the class distribution<jupyter_code>print(train_df['median_relevance'].value_counts())
train_df['median_relevance'].value_counts() / len(train_df)<jupyter_output><empty_output><jupyter_text>So our baseline to beat is 53%, which we get if we predict the most common class Let's create dataframes for training and validation<jupyter_code>training_df =  train_df.sample( int(len(train_df) * .9))
validation_df = train_df.query(f'_unit_id not in {training_df._unit_id.tolist() }')
validation_df.head()
training_file='../data/crowdflower-search-relevance/mytraining.csv'
validation_file = '../data/crowdflower-search-relevance/myvalidation.csv'

# Do this once
# training_df.to_csv( training_file, index=False)
# validation_df.to_csv(validation_file, index=False)
parser = argparse.ArgumentParser()
parser.add_argument('-gpu', type = int, default = 0)
parser.add_argument('-freeze_bert', action='store_true', default=False)
parser.add_argument('-maxlen', type = int, default= 512)
parser.add_argument('-batch_size', type = int, default=8)
parser.add_argument('-lr', type = float, default = 2e-5)
parser.add_argument('-print_every', type = int, default= 1000)
parser.add_argument('-max_eps', type = int, default=4)
parser.add_argument('-fp16', type=bool, default=True)
args = parser.parse_args(["-gpu", "0"] )<jupyter_output><empty_output><jupyter_text>Creating dataloaders<jupyter_code>print("Creating train and val dataloaders")
st = time.time()
train_set = SearchRelevanceDataset(filename = training_file, maxlen = args.maxlen,
                                  tokenizer=tokenizer ,
                                  data_columns=('query', 'product_title',  'product_description'))
val_set = SearchRelevanceDataset(filename = validation_file, maxlen = args.maxlen,
                                tokenizer=tokenizer,
                                data_columns=('query', 'product_title',  'product_description')) 
train_loader = DataLoader(train_set, batch_size = args.batch_size, num_workers = 2, shuffle=True) 
val_loader = DataLoader(val_set, batch_size = args.batch_size, num_workers = 1) 
print("Done in {} seconds".format(time.time() - st))
training_stats =[]

def flat_accuracy(preds, labels):
    """Function to calculate the accuracy of our predictions vs labels"""
    pred_flat = np.argmax(preds, axis=1).flatten()
    labels_flat = labels.flatten()
    return np.sum(pred_flat == labels_flat) / len(labels_flat)

def evaluate(model, dataloader, args, ep, avg_train_loss):
    model.eval()
    total_eval_accuracy = 0
    total_eval_loss = 0
    nb_eval_steps = 0
    t0 = time.time() 
    with torch.no_grad():
        for seq, tok_type_ids, attn_masks, labels in dataloader:
            seq, attn_masks = seq.cuda(args.gpu), attn_masks.cuda(args.gpu)            
            tok_type_ids, labels = tok_type_ids.cuda(args.gpu), labels.cuda(args.gpu)                        
            loss, logits = model(seq, 
                             token_type_ids=None, 
                             attention_mask=attn_masks,
                                 labels=labels) 
            # Accumulate the validation loss.
            total_eval_loss += loss.item()
            # Move logits and labels to CPU
            logits = logits.detach().cpu().numpy()
            label_ids = labels.to('cpu').numpy()
            # Calculate the accuracy for this batch & accumulate it over all batches.
            total_eval_accuracy += flat_accuracy(logits, label_ids)       

    # Report the final accuracy for this validation run.
    avg_val_accuracy = total_eval_accuracy / len(dataloader)
    print("  Accuracy: {0:.2f}".format(avg_val_accuracy))
    # Calculate the average loss over all of the batches.
    avg_val_loss = total_eval_loss / len(dataloader)    
    print("  Validation Loss: {0:.2f}".format(avg_val_loss))
    
    # Record all statistics from this epoch.
    training_stats.append(
        {
            'epoch': ep + 1,
            'Training Loss': avg_train_loss,
            'Valid. Loss': avg_val_loss,
            'Valid. Accur.': avg_val_accuracy,
        }
    )
    return avg_val_accuracy, avg_val_loss


def train(model, opti, train_loader, val_loader, args):
    model.zero_grad()
    best_acc = 0        
    for ep in range(args.max_eps):
        # Reset the total loss for this epoch.
        total_train_loss = 0   
        model.train()
        t0 = time.time()  # Measure how long the training epoch takes.
        pbar = tqdm(total=len(train_loader))
        for it, (seq, tok_type_ids, attn_masks, labels) in enumerate(train_loader):                        
            opti.zero_grad()
            seq, attn_masks = seq.cuda(args.gpu), attn_masks.cuda(args.gpu)
            tok_type_ids, labels = tok_type_ids.cuda(args.gpu), labels.cuda(args.gpu)            
            loss, logits = model(seq, 
                             token_type_ids=None, # tok_type_ids
                             attention_mask=attn_masks, 
                             labels=labels)                      

            # Accumulate the training loss for the epoch over all of the batches 
            total_train_loss += loss.item()

            # Perform a backward pass to calculate the gradients.
            if args.fp16:
                with amp.scale_loss(loss, opti) as scaled_loss:
                    scaled_loss.backward()
            else:
                loss.backward()    
            
            # Clip the norm of the gradients to 1.0.; to prevent the "exploding gradients" problem.
            if args.fp16:
                torch.nn.utils.clip_grad_norm_(amp.master_params(opti), 1.0)
            else:
                torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
                
            # Update parameters and take a step using the computed gradient.
            opti.step()
            # Update the learning rate.
            scheduler.step()
            model.zero_grad()
            pbar.update(1)
            if it % args.print_every == 0 and it >0:
                # Move logits and labels to CPU
                logits = logits.detach().cpu().numpy()
                label_ids = labels.to('cpu').numpy()
                acc = flat_accuracy(logits, label_ids)                               
                print("Iteration {} of epoch {} complete. Loss : {} Accuracy : {}".format(it, ep, loss.item(), acc))
        pbar.close()
        
        # Calculate the average loss over all of the batches.
        avg_train_loss = total_train_loss / len(train_loader)            
        print("Average training loss: {0:.2f}".format(avg_train_loss))        
        try:
            val_acc, val_loss = evaluate(model, val_loader, args, ep, avg_train_loss)
            print("Epoch {} complete! Validation Accuracy : {}, Validation Loss : {}".format(ep, val_acc, val_loss))
            if val_acc > best_acc:
                print("Best validation accuracy improved from {} to {}, saving model...".format(best_acc, val_acc))
                best_acc = val_acc
                torch.save(model.state_dict(), 'bert.cls.search.relevance._{}_freeze_{}.dat'.format(ep, args.freeze_bert))
        except Exception as ex:
            LOG.exception('failure %s', str(ex))
# Number of training epochs. The BERT authors recommend between 2 and 4. 
epochs = args.max_eps #  4

# Total number of training steps is [number of batches] x [number of epochs]. 
# (Note that this is not the same as the number of training samples).
total_steps = len(train_loader) * epochs

optimizer = AdamW(model.parameters(), lr = 2e-5, eps = 1e-8)
scheduler = get_linear_schedule_with_warmup(optimizer, 
                                            num_warmup_steps = 0, # Default value in run_glue.py
                                            num_training_steps = total_steps)<jupyter_output><empty_output><jupyter_text>Do you have Apex? If so, training with mixed precision will speed things up<jupyter_code># use Apex if available
try:
    from apex import amp
    model, optimizer = amp.initialize(model, optimizer, opt_level="O1") # s["fp16_opt_level"])
except ImportError:
    raise ImportError("Please install apex from https://www.github.com/nvidia/apex to use fp16 training.")
print("Training...")
st = time.time()
train(model, optimizer, train_loader, val_loader, args)
print("Done in {} seconds".format(time.time() - st))
loss_train = [ep['Training Loss'] for ep in training_stats]
loss_val = [ep['Valid. Loss'] for ep in training_stats]
acc_val = [ep['Valid. Accur.'] for ep in training_stats]
epochs = list(range(1,5))
plt.plot(epochs, loss_train, 'g', label='Training loss')
plt.plot(epochs, loss_val, 'b', label='validation loss')
plt.plot(epochs, acc_val, 'r', label='validation accuracy')
plt.title('Training & Validation loss & accuracy')
plt.xlabel('Epochs')
plt.ylabel('Loss / Acc')
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>The graph shows a lot of improvement could be made, but, sadly, we'll see it's good enough to beat the previous SOTA  Evaluate<jupyter_code>device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Put model in evaluation mode
model.eval()
# Tracking variables 
predictions, true_labels = [], []
# Predict 
for batch in val_loader:
    # Add batch to GPU
    batch = tuple(t.to(device) for t in batch)
    # Unpack the inputs from our dataloader
    b_input_ids, b_tok_types, b_input_mask, b_labels = batch
    # Telling the model not to compute or store gradients, saving memory and speeding up prediction
    with torch.no_grad():
        loss, logits = model(b_input_ids, 
                             token_type_ids=b_tok_types, 
                             attention_mask=b_input_mask, 
                             labels=b_labels)   
        # Move logits and labels to CPU
        logits = logits.detach().cpu().numpy()
        label_ids = b_labels.cpu().numpy()  
        # Store predictions and true labels
        predictions.append(logits)
        true_labels.append(label_ids)

# Flatten the predictions and true values for evaluation on the whole dataset
flat_predictions = [item for sublist in predictions for item in sublist]
flat_predictions = np.argmax(flat_predictions, axis=1).flatten()
flat_true_labels = [item for sublist in true_labels for item in sublist]

accuracy_score(flat_true_labels, flat_predictions)
print('labels', val_set.label_encoder.classes_)

target_names = [ str(tmp) for tmp in val_set.label_encoder.classes_.tolist()]
print(classification_report(flat_true_labels, flat_predictions, target_names=target_names))
model.save_pretrained('mymodel')
tokenizer.save_pretrained('mymodel')<jupyter_output>I0820 01:11:56.681124 139904012736320 configuration_utils.py:144] Configuration saved in mymodel/config.json
I0820 01:11:57.829241 139904012736320 modeling_utils.py:483] Model weights saved in mymodel/pytorch_model.bin<jupyter_text>Let's load the test data and prepare it to make a submission<jupyter_code>test_file ='../data/crowdflower-search-relevance/test.csv'
test_df = pd.read_csv(test_file)
test_df.head()
test_df['product_title'].fillna('', inplace=True)
test_df['product_description'].fillna('', inplace=True)
test_set = SearchRelevanceDataset(filename = test_file, maxlen = args.maxlen,
                                tokenizer=tokenizer,
                                data_columns=('query', 'product_title',  'product_description')
                                 , label_name=None)  
test_loader = DataLoader(test_set, batch_size = args.batch_size, num_workers = 1, shuffle=False)<jupyter_output><empty_output><jupyter_text>Let's prove we can reconstitute our model and make predictions for a submission<jupyter_code>device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Put model in evaluation mode
model.eval()

# Tracking variables 
predictions, true_labels = [], []

# Predict 
for batch in tqdm(test_loader, total=len(test_loader)):
    # Add batch to GPU
    batch = tuple(t.to(device) for t in batch)
    # Unpack the inputs from our dataloader
    b_input_ids, b_tok_types, b_input_mask, b_labels = batch
    # Telling the model not to compute or store gradients, saving memory and speeding up prediction
    with torch.no_grad():
        loss, logits = model(b_input_ids, 
                             token_type_ids=b_tok_types, 
                             attention_mask=b_input_mask, 
                             labels=b_labels)   
        # Move logits and labels to CPU
        logits = logits.detach().cpu().numpy()
        label_ids = b_labels.cpu().numpy()  
        # Store predictions and true labels
        predictions.append(logits)
        true_labels.append(label_ids)

flat_predictions = [item for sublist in predictions for item in sublist]
flat_predictions = np.argmax(flat_predictions, axis=1).flatten()

submission_file = 'my.kaggle.searchRelevance.submission.csv'
ids = test_df['id'].tolist()
corrected_preds =[tmp +1 for tmp in flat_predictions]
my_submission = pd.DataFrame({"id": ids, "prediction": corrected_preds})
my_submission.to_csv(submission_file, index=False)
! ls my.kaggle.searchRelevance.submission.csv
my_submission.head()
print(len(my_submission))
my_submission['prediction'].value_counts() / len(my_submission)<jupyter_output>22513
{'fraction_non_alphanumeric': 0.1007647323436797, 'fraction_numerical': 0.02862123256860099, 'mean_word_length': 3.450700700700701, 'pattern_counts': {'":': 10, '<': 46, '<?xml version=': 0, '>': 47, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '155894', 'license': 'Apache-2.0', 'n_tokens_mistral': 5976, 'n_tokens_neox': 5540, 'n_words': 1338, 'path': 'searching/search_results_relevance_using_BERT.ipynb', 'repo_name': 'todd-cook/ML-You-Can-Use', 'size': '17784'}