<jupyter_start><jupyter_text>Sequence Classification with TransformersAdapted from [Hugging Face Transformer Tutorial](https://medium.com/@lysandrejik/using-tensorflow-2-for-state-of-the-art-natural-language-processing-102445cda54a).We will be using two different models as a means of comparison: Google's BERT and Facebook's RoBERTa. Both have the same architecture but have had different pre-training approached.  Installing required dependenciesIn order to import the TensorFlow modules, we must make sure that TF2 is installed in the environment.<jupyter_code>import tensorflow as tf
print(tf.__version__)
# install Hugging Face's transformer library
!pip install transformers<jupyter_output>Requirement already satisfied: transformers in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (2.1.1)
Requirement already satisfied: regex in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from transformers) (2019.11.1)
Requirement already satisfied: sacremoses in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from transformers) (0.0.35)
Requirement already satisfied: numpy in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from transformers) (1.16.2)
Requirement already satisfied: boto3 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from transformers) (1.10.16)
Requirement already satisfied: tqdm in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from transformers) (4.36.1)
Requirement already satisfied: requests in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from transformers) (2.22.0)
Requirement already satisfied: sentencepiece in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from transformers) (0.1[...]<jupyter_text>Initializing the pre-trained modelsLet's initialize the models with pre-trained weights. The list of pre-trained weights is available in [the official documentation](https://huggingface.co/transformers/pretrained_models.html). Downloading the weights may take a bit of time, but it only needs to be done once!<jupyter_code>from transformers import (TFBertForSequenceClassification, 
                          BertTokenizer,
                          TFRobertaForSequenceClassification, 
                          RobertaTokenizer)

bert_model = TFBertForSequenceClassification.from_pretrained("bert-base-cased")
bert_tokenizer = BertTokenizer.from_pretrained("bert-base-cased")

roberta_model = TFRobertaForSequenceClassification.from_pretrained("roberta-base")
roberta_tokenizer = RobertaTokenizer.from_pretrained("roberta-base")<jupyter_output>A sequence with no special tokens has been passed to the RoBERTa model. This model requires special tokens in order to work. Please specify add_special_tokens=True in your encoding.
A sequence with no special tokens has been passed to the RoBERTa model. This model requires special tokens in order to work. Please specify add_special_tokens=True in your encoding.<jupyter_text>TokenizationBERT and RoBERTa are both Transformer models that have the same architecture. As such, they accept only a certain kind of inputs: vectors of integers, each value representing a token. Each string of text must first be converted to a list of indices to be fed to the model. The tokenizer takes care of that for us.BERT and RoBERTa may have the same architecture, but they differ in tokenization. BERT uses a sub-word tokenization, whereas RoBERTa uses the same tokenization than GPT-2: byte-level byte-pair-encoding. Let's see what this means:<jupyter_code>sequence = "Systolic arrays are cool. This 🐳 is cool too."

bert_tokenized_sequence = bert_tokenizer.tokenize(sequence)
roberta_tokenized_sequence = roberta_tokenizer.tokenize(sequence)

print("BERT:", bert_tokenized_sequence)
print("RoBERTa:", roberta_tokenized_sequence)<jupyter_output>BERT: ['S', '##ys', '##to', '##lic', 'array', '##s', 'are', 'cool', '.', 'This', '[UNK]', 'is', 'cool', 'too', '.']
RoBERTa: ['Sy', 'st', 'olic', 'Ġarrays', 'Ġare', 'Ġcool', '.', 'ĠThis', 'ĠðŁ', 'Ĳ', '³', 'Ġis', 'Ġcool', 'Ġtoo', '.']<jupyter_text>**BERT Tokenizer**Here, the BERT tokenizer splits the string into multiple substrings. If the substrings are in its vocabulary, they will stay as is: this is the case for `array`,  `are` and  `cool`. However, if a resulting string is not in its vocabulary, it will be split again until every string is represented by its vocabulary. For example,  `Systolic` is split multiple times until every token is represented in the BERT vocabulary: it is split into four tokens.The BERT tokenizer is lacking when it comes to complex characters spread over multiple bytes, as can be seen with emojis. In the sequence used, an emoji of a whale was added. As the BERT tokenizer cannot interpret this emoji on a byte-level, it replaces it by the unknown token [UNK].**RoBERTa Tokenizer**On the other hand, the RoBERTa tokenizer has a slightly different approach. Here too, the string is split into multiple substrings, which are themselves split into multiple substrings until every substring can be represented by the vocabulary. However, the RoBERTa tokenizer has a **byte-level approach**. This tokenizer can represent every sequence as a combination of bytes, which makes it shine in the case of complex characters spread over multiple bytes, as with the whale emoji. Instead of using the unknown token, this tokenizer can correctly encode the whale emoji as the combination of multiple bytes. This tokenizer therefore does not require an unknown token, as it can handle every byte separately.  Getting State-of-the-Art results on sequence classificationIn order to get State-of-the-Art results on this task, we will fine-tune our models on a given dataset. Fine-tuning a model means that we will slightly train it on top of an already trained checkpoint. The learning rate will be very low, as having it to high would result in catastrophic forgetting -> the model would forget what it had learned until now semantically and syntaxically.We will follow the procedure detailed below:    1) Get the dataset from `tensorflow_datasets`    2) Pre-process this dataset so that it can be used by the model    3) Set-up a training loop using Keras' fit API; train the model on the training data    4) Evaluate the model on the testing data and compare to the actual results  Getting the datasetWe will be using the Microsoft Research Paraphrase Corpus (MRPC) dataset, which is a sequence classification dataset. We get the train and validation data from the `tensorflow_datasets` package. These values are in the form of `tf.data.Dataset`, which is perfect for our use-case.<jupyter_code>!pip install tensorflow_datasets
import tensorflow_datasets
data = tensorflow_datasets.load("glue/mrpc")

train_dataset = data["train"]
validation_dataset = data["validation"]<jupyter_output>INFO - Overwrite dataset info from restored data version.
INFO - Reusing dataset glue (/home/azureuser/tensorflow_datasets/glue/mrpc/0.0.2)
INFO - Constructing tf.data.Dataset for split None, from /home/azureuser/tensorflow_datasets/glue/mrpc/0.0.2<jupyter_text>Let's output the value of the first item of the dataset to get an idea of the data we're dealing with.<jupyter_code>example = list(train_dataset.__iter__())[0]
print('',
    'idx:      ', example['idx'],       '\n',
    'label:    ', example['label'],     '\n',
    'sentence1:', example['sentence1'], '\n',
    'sentence2:', example['sentence2'],
)<jupyter_output>idx:       tf.Tensor(201, shape=(), dtype=int32) 
 label:     tf.Tensor(1, shape=(), dtype=int64) 
 sentence1: tf.Tensor(b'Tibco has used the Rendezvous name since 1994 for several of its technology products , according to the Palo Alto , California company .', shape=(), dtype=string) 
 sentence2: tf.Tensor(b'Tibco has used the Rendezvous name since 1994 for several of its technology products , it said .', shape=(), dtype=string)<jupyter_text>The values are in the form of dictionaries that are done as follow:```pyexample = {    idx: number,    label: number,    sentence1: string,    sentence2: string}```The three fields that are of interest to us are the `label`, the `sentence1` and the `sentence2`. The label is equal to `1` when the two sentences are paraphrases of each other, and is equal to `0` otherwise.We cannot pass this directly to the models as they cannot interpret the meaning of strings (list of characters), so we will see how to convert that example to features that our models can understand. Firstly, we must obtain the token ids that will be fed to the model from the two sequences. The two models (BERT, RoBERTa) have different encoding mechanisms concerning pair sequence encoding, using `sep` and `cls` tokens to specify the end of a sequence or the end of the two sequences. With `A` as the first sequence and `B` as the second, BERT encodes the sequence pair as follows:`[CLS] A [SEP] B [SEP]`while RoBERTa encodes the sequence pair differently:`[CLS] A [SEP][SEP] B [SEP]`Thankfully, our encode method can handle that on its own. Here are the two sentences we are currently dealing with:<jupyter_code>seq0 = example['sentence1'].numpy().decode('utf-8')  # Obtain bytes from tensor and convert it to a string
seq1 = example['sentence2'].numpy().decode('utf-8')  # Obtain bytes from tensor and convert it to a string

print("First sequence:", seq0)
print("Second sequence:", seq1)<jupyter_output>First sequence: Tibco has used the Rendezvous name since 1994 for several of its technology products , according to the Palo Alto , California company .
Second sequence: Tibco has used the Rendezvous name since 1994 for several of its technology products , it said .<jupyter_text>Encoding sequencesIn order to encode the sequence to be understandable by the model, two different methods can come in handy. encode()`encode` is a high-level method that returns the encoded sequence with the special tokens and truncated to a maximum length if need be. Here we identify the special CLS and SEP tokens of RoBERTa and BERT, and explicit them in the encoded sequence as to understand the difference in tokenization.<jupyter_code>encoded_bert_sequence = bert_tokenizer.encode(seq0, seq1, add_special_tokens=True, max_length=128)
encoded_roberta_sequence = roberta_tokenizer.encode(seq0, seq1, add_special_tokens=True, max_length=128)

print("BERT tokenizer separator, cls token id:   ", bert_tokenizer.sep_token_id, bert_tokenizer.cls_token_id)
print("RoBERTa tokenizer separator, cls token id:", roberta_tokenizer.sep_token_id, roberta_tokenizer.cls_token_id)

bert_special_tokens = [bert_tokenizer.sep_token_id, bert_tokenizer.cls_token_id]
roberta_special_tokens = [roberta_tokenizer.sep_token_id, roberta_tokenizer.cls_token_id]

def print_in_red(string):
    print("\033[91m" + str(string) + "\033[0m", end=' ')

print("\nBERT tokenized sequence")
output = [print_in_red(tok) if tok in bert_special_tokens else print(tok, end=' ') for tok in encoded_bert_sequence]

print("\n\nRoBERTa tokenized sequence")
output = [print_in_red(tok) if tok in roberta_special_tokens else print(tok, end=' ') for tok in encoded_roberta_sequence]<jupyter_output>BERT tokenizer separator, cls token id:    102 101
RoBERTa tokenizer separator, cls token id: 2 0

BERT tokenized sequence
[91m101[0m 157 13292 2528 1144 1215 1103 16513 15125 11944 1271 1290 1898 1111 1317 1104 1157 2815 2982 117 2452 1106 1103 19585 2858 17762 117 1756 1419 119 [91m102[0m 157 13292 2528 1144 1215 1103 16513 15125 11944 1271 1290 1898 1111 1317 1104 1157 2815 2982 117 1122 1163 119 [91m102[0m 

RoBERTa tokenized sequence
[91m0[0m 565 1452 876 34 341 5 29110 42057 766 187 8148 13 484 9 63 806 785 2156 309 7 5 21065 18402 2156 886 138 479 [91m2[0m [91m2[0m 565 1452 876 34 341 5 29110 42057 766 187 8148 13 484 9 63 806 785 2156 24 26 479 [91m2[0m<jupyter_text>encode_plus()`encode_plus` is similar to `encode` but it returns additional information: the token type ids as well as several other features that we don't need to manage right now.The token type ids are used by some models in the case of sequence classification. It is a mask indicating to the model which sequence a token is from. For example, let's say we have two sequences A and B with tokens `[a0, a1, a2, a3]` and `[b0, b1, b2, b3, b4]` respectively.The BERT tokenizer would create a single sequence from those two lists of tokens that would look like the following:[tokens]         `[CLS] a0 a1 a2 a3 [SEP] b0 b1 b2 b3 b4 [SEP]`. [token type ids] `  0    0  0  0  0   0    1  1  1  1  1   1`Thanks to the token type ids, the model is aware of which token belongs to which sequence We won't need to use encode_plus in this experiment as directly in the `Transformers` library exists a method to directly convert a dataset to features, and is agnostic to both the GLUE task and the specified tokenizer. This method makes use of `encode_plus` under the hood and is called `glue_convert_examples_to_features`:<jupyter_code>from transformers import glue_convert_examples_to_features

bert_train_dataset = glue_convert_examples_to_features(train_dataset, bert_tokenizer, 128, 'mrpc')
bert_train_dataset = bert_train_dataset.shuffle(100).batch(32).repeat(2)

bert_validation_dataset = glue_convert_examples_to_features(validation_dataset, bert_tokenizer, 128, 'mrpc')
bert_validation_dataset = bert_validation_dataset.batch(64)<jupyter_output><empty_output><jupyter_text>The two BERT datasets are now ready to be used: the training dataset is shuffled and batch, while the validation dataset is only batched.RoBERTa requires a bit more of work as it does not use the `token_type_ids`, which we need to remove. We use the `tf.data.Dataset.map()` method for this.<jupyter_code>def token_type_ids_removal(example, label):
    del example["token_type_ids"]
    return example, label

roberta_train_dataset = glue_convert_examples_to_features(train_dataset, roberta_tokenizer, 128, 'mrpc')
roberta_train_dataset = roberta_train_dataset.map(token_type_ids_removal)
roberta_train_dataset = roberta_train_dataset.shuffle(100).batch(32).repeat(2)

roberta_validation_dataset = glue_convert_examples_to_features(validation_dataset, roberta_tokenizer, 128, 'mrpc')
roberta_validation_dataset = roberta_validation_dataset.map(token_type_ids_removal)
roberta_validation_dataset = roberta_validation_dataset.batch(64)<jupyter_output><empty_output><jupyter_text>Defining the hyper-parametersBefore fine-tuning the model, we must define a few hyperparameters that will be used during the training such as the optimizer, the loss and the evaluation metric.As an optimizer we'll be using Adam, which was the optimizer used during those models' pre-training. As a loss we'll be using the sparse categorical cross-entropy, and the sparse categorical accuracy as the evaluation metric.<jupyter_code>optimizer = tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08, clipnorm=1.0)
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')

bert_model.compile(optimizer=optimizer, loss=loss, metrics=[metric])
roberta_model.compile(optimizer=optimizer, loss=loss, metrics=[metric])<jupyter_output><empty_output><jupyter_text>Training the modelThe beauty of tensorflow/keras lies here: using keras' fit method to fine-tune the model with a single line of code<jupyter_code>print("Fine-tuning BERT on MRPC")
bert_history = bert_model.fit(bert_train_dataset, epochs=3, validation_data=bert_validation_dataset)
print("\nFine-tuning RoBERTa on MRPC")
# this does not work - known bug in Hugging Face's Roberta implementation for TF2
#roberta_history = roberta_model.fit(roberta_train_dataset, epochs=3, validation_data=roberta_validation_dataset)<jupyter_output>Fine-tuning RoBERTa on MRPC<jupyter_text>Keras's simplicity doesn't end hereEvaluating a model is as simple as it is to train it - using the evaluate method<jupyter_code>print("Evaluating the BERT model")
bert_model.evaluate(bert_validation_dataset)

print("Evaluating the RoBERTa model")
#roberta_model.evaluate(roberta_validation_dataset)<jupyter_output>Evaluating the BERT model
7/7 [==============================] - 6s 862ms/step - loss: 0.5210 - accuracy: 0.8652
Evaluating the RoBERTa model
{'fraction_non_alphanumeric': 0.07895873172537166, 'fraction_numerical': 0.03867744124360002, 'mean_word_length': 5.526570048309178, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 41, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '361245', 'license': 'MIT', 'n_tokens_mistral': 5264, 'n_tokens_neox': 4746, 'n_words': 1876, 'path': 'labs/4.Sequence_Classification_Bert/Sequence_Classification_with_Transformers.ipynb', 'repo_name': 'gmarchet/dnnworkshop6', 'size': '16211'}