<jupyter_start><jupyter_text>SetupThis section contains supplementary information, functions, and installs required packages.<jupyter_code>!pip install tensorflow-gpu==2.0 tensorflow_datasets gpustat transformers -Uq<jupyter_output><empty_output><jupyter_text>**About**This notebook is put together by <NAME> (`<EMAIL>`) for the [**PyCon SG**](https://pycon.sg/) 2019 tutorial on [**Improving Deep Learning Performance in TensorFlow**](https://github.com/NVAITC/pycon-sg19-tensorflow-tutorial).**Acknowledgements*** This notebook uses some materials adapted from TensorFlow documentation.* This notebook uses the [HuggingFace Transformers library](https://github.com/huggingface/transformers).* This notebook uses the [GLUE (MRPC) Dataset](https://gluebenchmark.com/) ([TensorFlow Datasets page](https://www.tensorflow.org/datasets/catalog/glue)).**Dataset Citation**```@inproceedings{wang2019glue,  title={ {GLUE}: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding},  author={<NAME> <NAME> <NAME> <NAME> Levy, <NAME> <NAME>.},  note={In the Proceedings of ICLR.},  year={2019}}```<jupyter_code>import tensorflow.compat.v2 as tf
import tensorflow_datasets
import time

class TimeHistory(tf.keras.callbacks.Callback):
    def on_train_begin(self, logs={}):
        self.times = []
    def on_epoch_begin(self, epoch, logs={}):
        self.epoch_time_start = time.time()
    def on_epoch_end(self, epoch, logs={}):
        self.times.append(time.time() - self.epoch_time_start)<jupyter_output><empty_output><jupyter_text>Sequence Classification with BERT in TF 2.0<jupyter_code>!gpustat<jupyter_output><empty_output><jupyter_text>Load BERT Tokenizer<jupyter_code>from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained("bert-base-cased")<jupyter_output><empty_output><jupyter_text>Input Pipeline  Load Dataset<jupyter_code>data, info = tensorflow_datasets.load("glue/mrpc", with_info=True)

train_examples = info.splits["train"].num_examples
valid_examples = info.splits["validation"].num_examples<jupyter_output>INFO:absl:Overwrite dataset info from restored data version.
INFO:absl:Reusing dataset glue (/home/jovyan/tensorflow_datasets/glue/mrpc/0.0.2)
INFO:absl:Constructing tf.data.Dataset for split None, from /home/jovyan/tensorflow_datasets/glue/mrpc/0.0.2
WARNING:absl:Warning: Setting shuffle_files=True because split=TRAIN and shuffle_files=None. This behavior will be deprecated on 2019-08-06, at which point shuffle_files=False will be the default for all splits.<jupyter_text>Build Input Pipeline<jupyter_code>from transformers import glue_convert_examples_to_features

BATCH_SIZE = 32

# Prepare dataset for GLUE as a tf.data.Dataset instance
train_dataset = glue_convert_examples_to_features(data["train"], tokenizer, 128, "mrpc")
train_dataset = train_dataset.shuffle(512).batch(BATCH_SIZE).repeat(-1)

valid_dataset = glue_convert_examples_to_features(data["validation"], tokenizer, 128, "mrpc")
valid_dataset = valid_dataset.batch(BATCH_SIZE)<jupyter_output><empty_output><jupyter_text>Build BERT Model  Load Pre-trained BERT Model<jupyter_code>from transformers import TFBertForSequenceClassification

model = TFBertForSequenceClassification.from_pretrained("bert-base-cased")
opt = tf.keras.optimizers.Adam(learning_rate=3e-5)

loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
acc = tf.keras.metrics.SparseCategoricalAccuracy("accuracy")
model.compile(optimizer=opt,
              loss=loss,
              metrics=[acc])<jupyter_output><empty_output><jupyter_text>Train BERT Model<jupyter_code>time_callback = TimeHistory()
history = model.fit(train_dataset, epochs=4, steps_per_epoch=train_examples//BATCH_SIZE,
                    validation_data=valid_dataset, validation_steps=valid_examples//BATCH_SIZE,
                    validation_freq=3, callbacks=[time_callback])
epoch_time = min(time_callback.times)
egs_per_sec = train_examples//epoch_time

print("Peak Examples/s:", egs_per_sec)<jupyter_output>Peak Examples/s: 38.0
{'fraction_non_alphanumeric': 0.1134558093346574, 'fraction_numerical': 0.012909632571996028, 'mean_word_length': 6.962450592885376, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 39, 'https://': 5, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '379952', 'license': 'MIT', 'n_tokens_mistral': 1368, 'n_tokens_neox': 1292, 'n_words': 247, 'path': 'tf_transformer_base.ipynb', 'repo_name': 'NVAITC/pycon-sg19-tensorflow-tutorial', 'size': '4101'}