<jupyter_start><jupyter_text>ELMO/ARAVEC/FASTTEXT Baseline for IDAT In this notebook, we will walk you through the process of reproducing the ELMO/ARAVEC/FASTTEXT baseline for the IDAT Irony detection task.  Loading Required Modules We start by loading the needed python libraries.<jupyter_code>import os
import tensorflow as tf
from tensorflow import keras
import pandas as pd
from sklearn.metrics import f1_score
from embed_classer import embed<jupyter_output><empty_output><jupyter_text>Loading Data Using pandas, we can load and inspect the training and testing datasets as follows:<jupyter_code>df_train = pd.read_csv("../../data/idat/IDAT_training_text.csv")
df_test = pd.read_csv("../../data/idat/IDAT_test_text.csv")<jupyter_output><empty_output><jupyter_text>Below we list the 5 first entries in the training data.<jupyter_code>df_train.head()<jupyter_output><empty_output><jupyter_text>Below we list the 5 first entries in the testing data.<jupyter_code>df_test.head()<jupyter_output><empty_output><jupyter_text>Model Preparation We start by setting the randomisation seed and the maximum sentence length:<jupyter_code>tf.random.set_seed(123)
max_sentence_len = 200
model_type = "fasttext"

if model_type == "aravec":
    model_path = '../pretrained/full_uni_sg_300_twitter.mdl'
    size = 300
elif model_type == "fasttext":
    model_path = '../pretrained/cc.ar.300.bin'
    size = 300
elif model_type == "elmo":
    model_path= '../pretrained'
    size = 1024<jupyter_output><empty_output><jupyter_text>Next we load our model of choice:<jupyter_code>embedder = embed(model_type, model_path)<jupyter_output>Warning : `load_model` does not return WordVectorModel or SupervisedModel any more, but a `FastText` object which is very similar.<jupyter_text>Then we define the input and output to the model:<jupyter_code>sentence = keras.Input(shape=(max_sentence_len, size), name='sentence')
label = keras.Input(shape=(1,), name='label')<jupyter_output><empty_output><jupyter_text>This is followed by defining the structure of the network:<jupyter_code>forward_layer = tf.keras.layers.LSTM(size)
backward_layer = tf.keras.layers.LSTM(size, go_backwards=True)
masking_layer = tf.keras.layers.Masking()
rnn = tf.keras.layers.Bidirectional(forward_layer, backward_layer=backward_layer)
logits = rnn(sentence)
logits = keras.layers.Dense(1, activation=tf.nn.sigmoid)(logits)<jupyter_output><empty_output><jupyter_text>Then we construct and compile the model:<jupyter_code>model = keras.Model(inputs=sentence, outputs=logits)
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])<jupyter_output><empty_output><jupyter_text>Model Training First we perpare the inputs and outputs to be fed to the model during training:<jupyter_code>X_train = embedder.embed_batch(df_train["text"].tolist(), max_sentence_len)
Y_train = df_train["label"]<jupyter_output><empty_output><jupyter_text>Next we fit the data:<jupyter_code>model.fit(X_train, Y_train, epochs=5, batch_size=32)<jupyter_output>Epoch 1/5
126/126 [==============================] - 22s 152ms/step - loss: 0.6383 - accuracy: 0.6719
Epoch 2/5
126/126 [==============================] - 19s 153ms/step - loss: 0.5171 - accuracy: 0.7562
Epoch 3/5
126/126 [==============================] - 19s 153ms/step - loss: 0.4821 - accuracy: 0.7812
Epoch 4/5
126/126 [==============================] - 19s 152ms/step - loss: 0.4441 - accuracy: 0.8017
Epoch 5/5
126/126 [==============================] - 19s 152ms/step - loss: 0.4091 - accuracy: 0.8222<jupyter_text>Submission Preperation We perpare the features for each testset instance as follows:<jupyter_code>X_test = embedder.embed_batch(df_test["text"].tolist(), max_sentence_len)
Y_test = df_test["label"]<jupyter_output><empty_output><jupyter_text>We predict and evaluate the prediction as follows:<jupyter_code>predictions = (model.predict(X_test)>0.5).astype(int)
f1_score(Y_test, predictions, average="macro")<jupyter_output><empty_output><jupyter_text>We perpare the predictions as a pandas dataframe.<jupyter_code>df_preds = pd.DataFrame(data=predictions, columns=["prediction"], index=df_test["id"])
df_preds.reset_index(inplace=True)
if not os.path.exists("./predictions/{}".format(model_type)):
    os.makedirs("./predictions/{}".format(model_type), exist_ok=True)
df_preds.to_csv("./predictions/{}/irony.tsv".format(model_type), index=False, sep="\t")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14971687429218572, 'fraction_numerical': 0.03352208380520951, 'mean_word_length': 7.162661737523106, 'pattern_counts': {'":': 3, '<': 55, '<?xml version=': 0, '>': 56, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '280213', 'license': 'MIT', 'n_tokens_mistral': 1634, 'n_tokens_neox': 1495, 'n_words': 323, 'path': 'notebooks/elmo_aravec_fasttext/IDAT-irony-detection.ipynb', 'repo_name': 'mawdoo3/alue_baselines', 'size': '4415'}