<jupyter_start><jupyter_text>Using BERT as Feature Extractor Other than fine-tuning BERT for downstream task such as text classification, we can use pretrained BERT model as a feature extractor, very much the same as we are using pretrained CNN such as ResNet as feature extractors for downstream task such as image classification and object detection.  In this lab, we will see how we use a pretrained DistilBert Model to extract features (or embedding) from text and use the extracted features (embeddings) to train a classifier to classify text. You can contrast this with the other lab where we train the DistilBert end to end for the classification, and compare the performance of both. At the end of this session, you will be able to:- prepare data and use model-specific Tokenizer to format data suitable for use by the model- extract text embeddings from the bert model - use the extracted features for text classification  Install Hugging Face Transformers libraryIf you are running this notebook in Google Colab, you will need to install the Hugging Face transformers library as it is not part of the standard environment.<jupyter_code>!pip install transformers==4.6.1
import numpy as np
import tensorflow as tf
import pandas as pd
import os 
import shutil

from transformers import (
    AutoTokenizer,
    TFAutoModel,
)
from transformers.utils import logging as hf_logging
from sklearn.model_selection import train_test_split

# We enable logging level to info and use default log handler and log formatting
hf_logging.set_verbosity_info()
hf_logging.enable_default_handler()
hf_logging.enable_explicit_format()
# downloaded the datasets.

!wget https://nyp-aicourse.s3-ap-southeast-1.amazonaws.com/datasets/imdb_test.csv
!wget https://nyp-aicourse.s3-ap-southeast-1.amazonaws.com/datasets/imdb_train.csv
train_df = pd.read_csv('imdb_train.csv')
test_df = pd.read_csv('imdb_test.csv')<jupyter_output><empty_output><jupyter_text>The train set has 40000 samples. We will a small subset (e.g. 2000) samples for finetuning our pretrained model. Similarly we will use a smaller test set for evaluating our model. We use dataframe's sample() to randomly select a subset of samples.<jupyter_code>TRAIN_SIZE = 2000
TEST_SIZE = 200 

train_df = train_df.sample(n=TRAIN_SIZE)
test_df = test_df.sample(n=TEST_SIZE)
train_df['sentiment'] =  train_df['sentiment'].apply(lambda x: 0 if x == 'negative' else 1)
test_df['sentiment'] =  test_df['sentiment'].apply(lambda x: 0 if x == 'negative' else 1)
train_df.sentiment.value_counts()
train_texts = train_df['review'].to_list()
train_labels = train_df['sentiment'].to_list()
test_texts = test_df['review'].to_list()
test_labels = test_df['sentiment'].to_list()
train_texts, val_texts, train_labels, val_labels = train_test_split(train_texts, train_labels, test_size=.2)<jupyter_output><empty_output><jupyter_text>TokenizationWe will now load the DistilBert tokenizer for the pretrained model "distillbert-base-uncased".  This is the same as the other lab exercise.<jupyter_code>tokenizer = AutoTokenizer.from_pretrained('distilbert-base-uncased')<jupyter_output><empty_output><jupyter_text>The pretrained DistilBERT [tokenizer](https://huggingface.co/transformers/main_classes/tokenizer.htmltransformers.PreTrainedTokenizer) expects a string or list of string, so we need to convert the data frame (or series) into list.  Here we will tokenize the text string, and pad the text string to the longest sequence in the batch, and also to truncate the sequence if it exceeds the maximum length allowed by the model (in BERT's case, it is 512).<jupyter_code>train_encodings = tokenizer(train_texts, padding=True, truncation=True)
test_encodings = tokenizer(test_texts, padding=True, truncation=True)<jupyter_output><empty_output><jupyter_text>We will create a tensorflow dataset and use it's efficient batching later to obtain the embeddings.<jupyter_code>BATCH_SIZE = 16
train_dataset = tf.data.Dataset.from_tensor_slices((
    train_encodings['input_ids'],
    train_labels
)).batch(BATCH_SIZE)


test_dataset = tf.data.Dataset.from_tensor_slices((
    test_encodings['input_ids'],
    test_labels
)).batch(BATCH_SIZE)
len(train_encodings['input_ids'])<jupyter_output><empty_output><jupyter_text>Here we instantiate a pretrained model from 'distilbert-base-cased' and specify output_hidden_state=True so that we get the output from each of the attention layers.   Feature Extraction using (Distil)BERT. Here we will load the pretrained model for distibert-based-uncased and use it to extract features from the text (i.e. emeddings).<jupyter_code>model = TFAutoModel.from_pretrained("distilbert-base-uncased",output_hidden_states=True)<jupyter_output><empty_output><jupyter_text>The model will produce two outputs: the 1st output `output[0]` is of shape `(16, 512, 768)` which corresponds to the output of the last hidden layer and the second output `output[1]` is a list of 7 outputs of shape `(16, 512, 768)`, corresponding to the output of each of the 7 attention layers. 768 refers to the hidden size.<jupyter_code>train_embeddings = None

train_data = train_dataset.as_numpy_iterator()
count = 0
for batch in train_data:
    output = model.predict(batch[0])
    hidden_states = output[1]
    # here we take the output of the second last attention layer as our embeddings. 
    # We take the average of the embedding value of 512 tokens (at axis=1) to generate sentence embedding  
    sentence_embeddings = tf.reduce_mean(hidden_states[-2], axis=1).numpy()
    if train_embeddings is None:
        train_embeddings = sentence_embeddings
    else:
        train_embeddings = np.vstack([train_embeddings, sentence_embeddings])
train_embeddings.shape
test_embeddings = None

test_data = test_dataset.as_numpy_iterator()

for batch in test_data:
    output = model.predict(batch[0])
    hidden_states = output[1]
    # here we take the output of the second last attention layer as our embeddings. 
    # We take the average of the embedding value of 512 tokens (at axis=1) to generate sentence embedding  
    sentence_embeddings = tf.reduce_mean(hidden_states[-2], axis=1).numpy()
    if test_embeddings is None:
        test_embeddings = sentence_embeddings
    else:
        test_embeddings = np.vstack([test_embeddings, sentence_embeddings])
test_embeddings.shape<jupyter_output><empty_output><jupyter_text>Train a classifier using the extracted features (embeddings)<jupyter_code>X_train = train_embeddings
y_train = train_labels
X_test = test_embeddings
y_test = test_labels
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
clf = LogisticRegression(max_iter=1000)
clf.fit(X_train, y_train)
y_pred = clf.predict(X_test)
print(classification_report(y_test, y_pred))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07518022657054584, 'fraction_numerical': 0.011475651022509931, 'mean_word_length': 5.770916334661354, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '361034', 'license': 'Apache-2.0', 'n_tokens_mistral': 2109, 'n_tokens_neox': 2017, 'n_words': 693, 'path': 'week12/bert-embedding.ipynb', 'repo_name': 'nyp-sit/it3103-tutors', 'size': '6797'}