<jupyter_start><jupyter_text>Universal Sentence Encoder Baseline for NSURL-2019 Shared Task 8 In this notebook, we will walk you through the process of reproducing the Universal Sentence Encoder baseline for the NSURL-2019 Shared Task 8.  Loading Required Modules We start by loading the needed python libraries.<jupyter_code>import os
import tensorflow as tf
from tensorflow import keras
import tensorflow_hub as hub
import pandas as pd
import tensorflow_text
from tensorflow import keras
from sklearn.metrics import f1_score<jupyter_output><empty_output><jupyter_text>Loading Data Using pandas, we can load and inspect the training, validation, and testing datasets as follows:<jupyter_code>df_train = pd.read_csv("../../data/nsurl/q2q_similarity_workshop_v2.1.tsv", sep="\t")
df_test = pd.read_csv("../../private_datasets/q2q/q2q_no_labels_v1.0.tsv", sep="\t")<jupyter_output><empty_output><jupyter_text>Below we list the 5 first entries in the training data.<jupyter_code>df_train.head()<jupyter_output><empty_output><jupyter_text>And last but not least, the first 5 entries in the test data.<jupyter_code>df_test.head()<jupyter_output><empty_output><jupyter_text>Model Preparation We start by setting the randomisation seed:<jupyter_code>tf.random.set_seed(123)<jupyter_output><empty_output><jupyter_text>Next we load the Universal Sentence Encoder (WARNING: This will download and cache a huge model of around 1 GB in size)<jupyter_code>embed = hub.load("https://tfhub.dev/google/universal-sentence-encoder-multilingual/3")<jupyter_output><empty_output><jupyter_text>Then we define the input and output to the model:<jupyter_code>q1_input = keras.Input(shape=512, name='q1')
q2_input = keras.Input(shape=512, name='q2')
label = keras.Input(shape=(1,), name='label')<jupyter_output><empty_output><jupyter_text>This is followed by defining the structure of the network:<jupyter_code>feat_1 = tf.abs(q1_input - q2_input)
feat_2 = q1_input*q2_input
features = tf.concat([q1_input, q2_input, feat_1, feat_2], 1)
logits = keras.layers.Dense(512*4, activation=tf.nn.tanh)(features)
logits = keras.layers.Dense(512*4, activation=tf.nn.tanh)(tf.concat([logits, features],1))
logits = keras.layers.Dense(512*4, activation=tf.nn.tanh)(tf.concat([logits, features],1))
logits = keras.layers.Dense(1, activation=tf.nn.sigmoid)(tf.concat([logits, features],1))<jupyter_output><empty_output><jupyter_text>Then we construct and compile the model:<jupyter_code>model = keras.Model(inputs=[q1_input, q2_input], outputs=logits)
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])<jupyter_output><empty_output><jupyter_text>Model Training First we perpare the inputs and outputs to be fed to the model during training:<jupyter_code>X1_train = embed(df_train["question1"])
X2_train = embed(df_train["question2"])
Y_train = df_train["label"]<jupyter_output><empty_output><jupyter_text>Next we fit the data:<jupyter_code>model.fit([X1_train, X2_train],
          Y_train,
          epochs=10,
          batch_size=32)<jupyter_output>Epoch 1/10
375/375 [==============================] - 28s 75ms/step - loss: 0.5235 - accuracy: 0.7531
Epoch 2/10
375/375 [==============================] - 32s 85ms/step - loss: 0.3728 - accuracy: 0.8455
Epoch 3/10
375/375 [==============================] - 31s 84ms/step - loss: 0.3934 - accuracy: 0.8390
Epoch 4/10
375/375 [==============================] - 42s 113ms/step - loss: 0.3507 - accuracy: 0.8524
Epoch 5/10
375/375 [==============================] - 44s 117ms/step - loss: 0.3172 - accuracy: 0.8621
Epoch 6/10
375/375 [==============================] - 33s 88ms/step - loss: 0.3271 - accuracy: 0.8597
Epoch 7/10
375/375 [==============================] - 31s 83ms/step - loss: 0.2910 - accuracy: 0.8766
Epoch 8/10
375/375 [==============================] - 43s 116ms/step - loss: 0.3111 - accuracy: 0.8609
Epoch 9/10
375/375 [==============================] - 44s 116ms/step - loss: 0.3072 - accuracy: 0.8674
Epoch 10/10
375/375 [==============================] - 34s 90ms/step - loss: 0[...]<jupyter_text>Submission Preperation We perpare the features for each testset instance as follows:<jupyter_code>x1_test = embed(df_test["question1"])
x2_test = embed(df_test["question2"])<jupyter_output><empty_output><jupyter_text>Then we predict the labels for each:<jupyter_code>predictions = (model.predict([x1_test, x2_test])>0.5).astype(int)<jupyter_output><empty_output><jupyter_text>We perpare the predictions as a pandas dataframe.<jupyter_code>df_preds = pd.DataFrame(data=predictions, columns=["prediction"], index=df_test["QuestionPairID"])
df_preds.reset_index(inplace=True)<jupyter_output><empty_output><jupyter_text>In the final step, we save the predictions as required by the competition guidelines.<jupyter_code>if not os.path.exists("predictions"):
    os.mkdir("predictions")
df_preds.to_csv("./predictions/q2q.tsv", index=False, sep="\t")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.17898359991901194, 'fraction_numerical': 0.06276574205304718, 'mean_word_length': 6.730829420970266, 'pattern_counts': {'":': 0, '<': 60, '<?xml version=': 0, '>': 61, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '630187', 'license': 'MIT', 'n_tokens_mistral': 1922, 'n_tokens_neox': 1691, 'n_words': 374, 'path': 'notebooks/USE/NSURL-2019-Task-8.ipynb', 'repo_name': 'mawdoo3/alue_baselines', 'size': '4939'}