<jupyter_start><jupyter_text>Get TensorFlowIn case you have not installed TensorFlow, run the following cell.<jupyter_code># # Requires the latest pip
# !pip3 install --upgrade pip
# !pip3 install tensorflow<jupyter_output><empty_output><jupyter_text>Get DataIn case you have not run the `1 - get_data.ipynb`, run the following cell.<jupyter_code># !wget http://qim.fs.quoracdn.net/quora_duplicate_questions.tsv
# import get_data

# get_data.get_quora_data()<jupyter_output><empty_output><jupyter_text>Preprocess and numericalize text dataIn case you have not run the `2 - preprocess_data.ipynb`, run the following cell.<jupyter_code>from utils import *

# ---- load dataset ----
train_set, dev_set, test_set = load_dataset(['train.txt', 'dev.txt', 'test.txt'])

# ---- numericalize the train set ----
V = TextVectorizer(tokenize) 
text = gather_text(train_set) # for collecting texts from train set
V.build_vocab(text) # for building mapping vocab_to_idx dictionary and text_encoder
train_set_encoded = list(encode_dataset(train_set, encoder=V.text_encoder)) # encodoing train set
dev_set_encoded = list(encode_dataset(dev_set, encoder=V.text_encoder)) # encodoing dev set for validation
test_set_encoded  = list(encode_dataset(test_set, encoder=V.text_encoder)) # encodoing dev set for prediction<jupyter_output>Two vocabulary dictionaries have been built!
Please call [1mX.vocab_to_idx | X.idx_to_vocab[0m to find out more where [X] stands for the name you used for this TextVectorizer class.<jupyter_text>A noteThere are multiple ways to use tensorflow to train a model, but the easiest one is to employ the `fit` method. In this `fit` function, the `inputs` and `targets` (labels) from the train set should be separately provided, and the `inputs` and `targets` from the dev set should be put inside a list or tuple. And both the `inputs` and `targets` should not be batched, as there is another builtin parameter called `batch_size` that will create mini batches for us. Nevertheless, to maintain consistency, this tutorial decided to still use the `build_batches` function that we will build together later in other tutorials. This `build_batches` function will help normalize the text seq length, which tensorflow's `fit` method does not provide. A better way of using packages in the tensorflow ecosystem to preprocess and numericalize text data will be introduced separately, just as what I intended to do for the other two deep learning frameworks.<jupyter_code># ---- build mini batches for the train and dev set ----
train_set_batched = build_batches(train_set_encoded, batch_size=3000, include_seq_len=False)
dev_set_batched = build_batches(dev_set_encoded, batch_size=1000, include_seq_len=False)
test_set_batched = build_batches(test_set_encoded, batch_size=1000, include_seq_len=False)

train_X1, train_X2, train_Y = train_set_batched[0]
dev_X1, dev_X2, dev_Y = dev_set_batched[0]
test_X1, test_X2, test_Y = test_set_batched[0]<jupyter_output><empty_output><jupyter_text>Training and evaluating models  BoW (Bag of Words) model Training<jupyter_code>from tf_models.BoW import BoW
from tensorflow import keras
model = BoW(len(V.vocab_to_idx), 1)
model.compile(optimizer=keras.optimizers.Adam(5e-4),
              loss=keras.losses.BinaryCrossentropy(),
              metrics=[["accuracy"]])

%time model.fit([train_X1, train_X2], train_Y, epochs=5, batch_size=64, validation_data=([dev_X1, dev_X2], dev_Y))<jupyter_output>2022-01-19 23:47:39.949779: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-01-19 23:47:40.685039: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)<jupyter_text>Evaluation on the test set<jupyter_code>model.evaluate([test_X1, test_X2], test_Y)<jupyter_output>32/32 [==============================] - 0s 1ms/step - loss: 0.6094 - accuracy: 0.6680<jupyter_text>CNN (Convolutional Neural Network) model Training<jupyter_code>from tf_models.CNN import CNN

model = CNN(len(V.vocab_to_idx), 1)
model.compile(optimizer=keras.optimizers.Adam(5e-4),
              loss=keras.losses.BinaryCrossentropy(),
              metrics=[["accuracy"]])

%time model.fit([train_X1, train_X2], train_Y, epochs=5, batch_size=64, validation_data=([dev_X1, dev_X2], dev_Y))<jupyter_output>Epoch 1/5
47/47 [==============================] - 2s 32ms/step - loss: 0.6896 - accuracy: 0.5567 - val_loss: 0.6817 - val_accuracy: 0.6200
Epoch 2/5
47/47 [==============================] - 1s 31ms/step - loss: 0.6411 - accuracy: 0.7313 - val_loss: 0.6442 - val_accuracy: 0.6280
Epoch 3/5
47/47 [==============================] - 2s 33ms/step - loss: 0.4901 - accuracy: 0.8133 - val_loss: 0.6240 - val_accuracy: 0.6890
Epoch 4/5
47/47 [==============================] - 1s 30ms/step - loss: 0.2682 - accuracy: 0.9107 - val_loss: 0.7409 - val_accuracy: 0.6540
Epoch 5/5
47/47 [==============================] - 1s 28ms/step - loss: 0.1152 - accuracy: 0.9670 - val_loss: 0.9151 - val_accuracy: 0.6500
CPU times: user 28.3 s, sys: 7.88 s, total: 36.2 s
Wall time: 7.77 s<jupyter_text>Evaluation on the test set<jupyter_code>model.evaluate([test_X1, test_X2], test_Y)<jupyter_output>32/32 [==============================] - 0s 4ms/step - loss: 0.9565 - accuracy: 0.6190<jupyter_text>RNN (Recurrent neural network) models  Simple RNN model Training<jupyter_code>from tf_models.S_RNN import SimpleRNN

model = SimpleRNN(len(V.vocab_to_idx), 1, bidirectional=False)

model.compile(optimizer=keras.optimizers.Adam(5e-4),
              loss=keras.losses.BinaryCrossentropy(),
              metrics=[["accuracy"]])

%time model.fit([train_X1, train_X2], train_Y, epochs=5, batch_size=64, validation_data=([dev_X1, dev_X2], dev_Y))<jupyter_output>Epoch 1/5
47/47 [==============================] - 4s 40ms/step - loss: 0.6783 - accuracy: 0.5617 - val_loss: 0.6489 - val_accuracy: 0.6030
Epoch 2/5
47/47 [==============================] - 2s 37ms/step - loss: 0.5395 - accuracy: 0.7383 - val_loss: 0.6443 - val_accuracy: 0.6370
Epoch 3/5
47/47 [==============================] - 2s 34ms/step - loss: 0.3117 - accuracy: 0.8883 - val_loss: 0.7628 - val_accuracy: 0.6340
Epoch 4/5
47/47 [==============================] - 2s 33ms/step - loss: 0.1100 - accuracy: 0.9677 - val_loss: 0.9975 - val_accuracy: 0.6210
Epoch 5/5
47/47 [==============================] - 2s 33ms/step - loss: 0.0417 - accuracy: 0.9907 - val_loss: 1.1667 - val_accuracy: 0.6370
CPU times: user 33.2 s, sys: 9.99 s, total: 43.2 s
Wall time: 10.4 s<jupyter_text>Evaluation on the test set<jupyter_code>model.evaluate([test_X1, test_X2], test_Y)<jupyter_output>32/32 [==============================] - 0s 4ms/step - loss: 1.2218 - accuracy: 0.6190<jupyter_text>GRU (Gated recurrent units) model  Training<jupyter_code>from tf_models.GRU import GRU

model = GRU(len(V.vocab_to_idx), 1, bidirectional=False)
model.compile(optimizer=keras.optimizers.Adam(5e-4),
              loss=keras.losses.BinaryCrossentropy(),
              metrics=[["accuracy"]])

%time model.fit([train_X1, train_X2], train_Y, epochs=5, batch_size=64, validation_data=([dev_X1, dev_X2], dev_Y))<jupyter_output>Epoch 1/5
47/47 [==============================] - 10s 104ms/step - loss: 0.6915 - accuracy: 0.5353 - val_loss: 0.6884 - val_accuracy: 0.5880
Epoch 2/5
47/47 [==============================] - 3s 68ms/step - loss: 0.6171 - accuracy: 0.7170 - val_loss: 0.6209 - val_accuracy: 0.6610
Epoch 3/5
47/47 [==============================] - 3s 70ms/step - loss: 0.3772 - accuracy: 0.8453 - val_loss: 0.7429 - val_accuracy: 0.6260
Epoch 4/5
47/47 [==============================] - 3s 67ms/step - loss: 0.1799 - accuracy: 0.9407 - val_loss: 0.9252 - val_accuracy: 0.6240
Epoch 5/5
47/47 [==============================] - 3s 67ms/step - loss: 0.0887 - accuracy: 0.9747 - val_loss: 1.2896 - val_accuracy: 0.6290
CPU times: user 1min 15s, sys: 19.8 s, total: 1min 34s
Wall time: 22.4 s<jupyter_text>Evaluation on the test set<jupyter_code>model.evaluate([test_X1, test_X2], test_Y)<jupyter_output>32/32 [==============================] - 2s 8ms/step - loss: 1.3176 - accuracy: 0.6110<jupyter_text>LSTM (Long short-term memory) model Training<jupyter_code>from tf_models.LSTM import LSTM

model = LSTM(len(V.vocab_to_idx), 1, bidirectional=False)
model.compile(optimizer=keras.optimizers.Adam(5e-4),
              loss=keras.losses.BinaryCrossentropy(),
              metrics=[["accuracy"]])

%time model.fit([train_X1, train_X2], train_Y, epochs=5, batch_size=64, validation_data=([dev_X1, dev_X2], dev_Y))<jupyter_output>Epoch 1/5
47/47 [==============================] - 11s 130ms/step - loss: 0.6831 - accuracy: 0.5700 - val_loss: 0.6531 - val_accuracy: 0.6200
Epoch 2/5
47/47 [==============================] - 4s 96ms/step - loss: 0.5764 - accuracy: 0.7153 - val_loss: 0.6018 - val_accuracy: 0.6650
Epoch 3/5
47/47 [==============================] - 4s 87ms/step - loss: 0.4192 - accuracy: 0.8303 - val_loss: 0.6325 - val_accuracy: 0.6590
Epoch 4/5
47/47 [==============================] - 4s 88ms/step - loss: 0.2442 - accuracy: 0.9130 - val_loss: 0.8302 - val_accuracy: 0.6650
Epoch 5/5
47/47 [==============================] - 4s 87ms/step - loss: 0.1267 - accuracy: 0.9623 - val_loss: 1.0756 - val_accuracy: 0.6540
CPU times: user 1min 33s, sys: 26.3 s, total: 1min 59s
Wall time: 28.2 s<jupyter_text>Evaluation on the test set<jupyter_code>model.evaluate([test_X1, test_X2], test_Y)<jupyter_output>32/32 [==============================] - 2s 16ms/step - loss: 1.0836 - accuracy: 0.6570
{'fraction_non_alphanumeric': 0.1882990692441444, 'fraction_numerical': 0.08673417203641198, 'mean_word_length': 4.998773006134969, 'pattern_counts': {'":': 0, '<': 46, '<?xml version=': 0, '>': 46, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '309528', 'license': 'MIT', 'n_tokens_mistral': 4050, 'n_tokens_neox': 3434, 'n_words': 880, 'path': '0.3 - very_quick_starts_with_tensorflow.ipynb', 'repo_name': 'jaaack-wang/text-matching-explained', 'size': '9777'}