<jupyter_start><jupyter_text>Lab assignment: detecting toxic comments with Deep Learning  In this assignment we will analyze text comments appearing in social media, trying to automatically detect those of toxic content: insults, threats, obscene comments, ... to do so we will make use of word embeddings and convolutional and recurrent neural networks.  Guidelines Throughout this notebook you will find empty cells that you will need to fill with your own code. Follow the instructions in the notebook and pay special attention to the following symbols. You will need to solve a question by writing your own code or answer in the cell immediately below, or in a different file as instructed. This is a hint or useful observation that can help you solve this assignment. This is an advanced and voluntary exercise that can help you gain a deeper knowledge into the topic. Good luck!To avoid missing packages and compatibility issues you should run this notebook under one of the [recommended Deep Learning environment files](https://github.com/albarji/teaching-environments/tree/master/deeplearning). The following code will embed any plots into the notebook instead of generating a new window:<jupyter_code>import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Lastly, if you need any help on the usage of a Python function you can place the writing cursor over its name and press Caps+Shift to produce a pop-out with related documentation. This will only work inside code cells. Let's go!  The Keras library In this lab we will make use of the [keras](http://keras.io/) Deep Learning library for Python. This library allows building several kinds of shallow and deep networks, following either a sequential or a graph architecture.  Data loading Data is provided as two separate files, one with texts for training the model and another one for testing. Both files are available in compressed form under the *data* folder.       Load the data into two pandas Dataframes, train for the training data and test for the test data.<jupyter_code>####### INSERT YOUR CODE HERE<jupyter_output><empty_output><jupyter_text>If you have loaded the data properly, you should be able to visualize the first rows of each data set as follows<jupyter_code>train.head()
test.head()<jupyter_output><empty_output><jupyter_text>As you can see, the data files include a column *comment_text* with the text we must classify, and 6 additional columns with the kinds of toxicity that might be present in a comment: *toxic*, *severe_toxic*, *obscene*, *threat*, *insult* and *identity_hate*.  Data processing We can't introduce text directly into the network, so we will have to tranform it to a vector representation. To do so, we will first **tokenize** the text into words (or tokens), and assign a unique identifier to each word found in the text. Doing this will allow us to perform the encoding. We can do this easily by making use of the **Tokenizer** class in keras:<jupyter_code>from keras.preprocessing.text import Tokenizer<jupyter_output><empty_output><jupyter_text>A Tokenizer offers convenient methods to split texts down to tokens. At construction time we need to supply the Tokenizer the maximum number of different words we are willing to represent. If our texts have greater word variety than this number, the least frequent words will be discarded. We will choose a number large enough for our purpose.<jupyter_code>maxwords = 15000
tokenizer = Tokenizer(num_words = maxwords)<jupyter_output><empty_output><jupyter_text>We now need to **fit** the Tokenizer to the training texts, which can be done as follows:<jupyter_code>tokenizer.fit_on_texts(train["comment_text"])<jupyter_output><empty_output><jupyter_text>After fitting the Tokenizer we can see a report of the number of times each word has been found in the input texts.<jupyter_code>tokenizer.word_counts<jupyter_output><empty_output><jupyter_text>Now the we have trained the tokenizer we can use it to vectorize the texts. In particular, we would like to transform the texts to sequences of word indexes. We can do this through the **text_to_sequences** method<jupyter_code>X_train = tokenizer.texts_to_sequences(train["comment_text"])<jupyter_output><empty_output><jupyter_text>We can see now how a text has been transformed to a list of word indexes.<jupyter_code>X_train[0]<jupyter_output><empty_output><jupyter_text>What do these indexes mean? Each number represents a word recognized by the trained Tokenizer. To check what word each index represents, we can take a look at the **word_index** attribute:<jupyter_code>tokenizer.word_index<jupyter_output><empty_output><jupyter_text>As usual, we will need to repeat this transformation to indexes for the test data.  Repeat the transformation to sequences of indexes for the test data. Save the result of the transformation into an X_test variable..<jupyter_code>####### INSERT YOUR CODE HERE<jupyter_output><empty_output><jupyter_text>This is enough to train a Sequential Network. However, for efficiency reasons it is recommended that all sequences in the data have the same number of elements. Since this is not the case for our data, as each text has a different number of words, we should **pad** the sequences to ensure the same length. The padding procedure adds a special *null* symbol to short sequences, and clips out parts of long sequences, thus enforcing a common size.This procedure can be performed easily by again resorting to keras tools. To choose an appropriate padding length we can have a look at a boxplot of the number of tokens in our training texts.<jupyter_code>import matplotlib.pyplot as plt

plt.boxplot([len(text) for text in X_train], whis=[15, 85], showfliers=False)
plt.ylabel("#Words in a text")
plt.show()<jupyter_output><empty_output><jupyter_text>We will pad all sequences to have a maximum of 120 indexes (words), which is enough to represent 85% of our texts without dropping tokens.<jupyter_code>from keras.preprocessing.sequence import pad_sequences 
maxsequence = 120
X_train = pad_sequences(X_train, maxlen=maxsequence)<jupyter_output><empty_output><jupyter_text>Repeat the process for the test data, saving the result into a variable X_test.<jupyter_code>####### INSERT YOUR CODE HERE<jupyter_output><empty_output><jupyter_text>The texts are now ready to serve as inputs for the model. We must also process the toxicity labels to serve as outputs. To do so, just create Dataframe or numpy arrays **y_train** and **y_test** containing the 6 columns with the toxicity indicators.<jupyter_code>####### INSERT YOUR CODE HERE<jupyter_output><empty_output><jupyter_text>Continuous Bag of Words model (CBoW)  We will start with a simple, yet effective model to deal with sequences of words: the Continuous Bag of Words (CBoW) model. This model is comprised of:* An [Embedding](https://keras.io/layers/embeddings/) layer that transforms word indexes to a vector representation that is learned with the model. * A  **GlobalAveragePooling1D** layer that obtains a representation of the whole document as the average of all its word vector representations* A final layer taking the decision of the network. Let's start by creating a keras Sequential model<jupyter_code>from keras.models import Sequential

model = Sequential()<jupyter_output><empty_output><jupyter_text>The first layer in a network that works with indexes should be the Embedding layer. We will transform each index (word) into an embedding vector of 64 elements, which will be learned by the network. This can be done as<jupyter_code>from keras.layers.embeddings import Embedding

model.add(Embedding(input_dim=maxwords, input_length=maxsequence, output_dim=64))<jupyter_output><empty_output><jupyter_text>Note how the Embedding layer requires three parameters: the number of different words in our problem (input_dim), the length of the input sequences (input_length) and the size of the embedding vectors to be created (output_dim). After the embedding we will add the averaging layer to obtain a single vector of values representing the whole document.<jupyter_code>from keras.layers import GlobalAveragePooling1D

model.add(GlobalAveragePooling1D())<jupyter_output><empty_output><jupyter_text>After this we will add the output layer. The problem we are trying to solve is a **multilabel** problem: there are 6 possible toxic classes for each text, but a text might include several of these classes, or none of them. Therefore, the output layer should contain 6 neurons followed by a sigmoid activation. This way each class might activate independently of the rest.<jupyter_code>from keras.layers.core import Dense, Activation

model.add(Dense(6))
model.add(Activation('sigmoid'))<jupyter_output><empty_output><jupyter_text>And with this the design of the CBoW is finished! Let's check what we have built<jupyter_code>model.summary()<jupyter_output><empty_output><jupyter_text>Note how most of the model parameters are located in the Embedding layer. That is, the focus of the network is on finding good word representations that allow classifying the sentiment of the sentence. We now follow the usual procedure for compiling the network. Note that since we have a multiple but independent output neurons, we select **binary_crossentropy** as the network loss.<jupyter_code>model.compile(loss='binary_crossentropy', optimizer='adam', metrics=["accuracy"])<jupyter_output><empty_output><jupyter_text>And we proceed on to train the network. Since the CBoW model is very simple, training should be reasonably fast.<jupyter_code>model.fit(X_train, y_train, batch_size=32, epochs=10)<jupyter_output><empty_output><jupyter_text>After training we can measure the performance over the test set as usual<jupyter_code>score = model.evaluate(X_test, y_test)
print("Test loss", score[0])
print("Test accuracy", score[1])<jupyter_output><empty_output><jupyter_text>Note however that accuracy might not be the best metric for this problem: fortunately most of the comments in social media are not toxic, and so it's easy to obtain good accuracy by just classifying everything as non-toxic. In this kind of unbalanced problems it is better to make use of the [ROC AUC score](https://scikit-learn.org/stable/modules/model_evaluation.htmlroc-metrics).<jupyter_code>from sklearn.metrics import roc_auc_score

y_pred = model.predict(X_test)
print("AUC score", roc_auc_score(y_test, y_pred))<jupyter_output><empty_output><jupyter_text>Convolutional mixing model Using the average of word embeddings is not optimal, since we are throwing away information about word ordering. A simple method to take neighborhoods of words into account is to add a small convolutional layer after the Embeddings layer. This way the embeddings of each word gets mixed with the embeddings of their neighbouring words, thus producing a more contextualized representation.       Create a new model that follows the same architecture as the CBoW model above, but including a 1-dimensional convolution between the Embedding and the GlobalAveragePooling1D layers. You can use 64 filters with kernel size 9: this will mix the embeddings of length 64 of each word into new embeddings again of size 64, that now contain also information from close words. Try also with a small number of epochs, in the range 1-3. Do you obtain better results with this model?<jupyter_code>####### INSERT YOUR CODE HERE<jupyter_output><empty_output><jupyter_text>LSTM model  Even if a simple convolutional mixing of neighbouring words might help, we are still missing something important. Language makes use of relationships between distant words to produce meaningful sentences, and we will need to make use of a more powerful mixing model to improve our results. To do so, we will make use of a Long-short Term Memory layer. An LSTM layer that creates a document-level representation as a vector of 32 values can be created with Keras as follows<jupyter_code>from keras.layers.recurrent import LSTM

lstm_layer = LSTM(32)<jupyter_output><empty_output><jupyter_text>If you are training your networks in an nVidia GPU, don't use the LSTM layer. Instead, use the CuDNNLSTM layer. This is a different implementation of the LSTM neurons that is highly optimized for nVidia GPUs, and can make your model train 10x times faster than using the standard LSTM implementation.  If no other parameters are given the LSTM will process the whole input sequence, and forward the computed document-level representation after processing the last element of the sequence.  Create an LSTM model by copying the design from the CBoW model, but replacing the averaging layer by an LSTM layer of 32 units. Can you obtain better test results with this model?   LSTM layers are expensive to compute, due to their recurrent nature. Don't worry if training takes much longer than with the CBoW model. Also, LSTM networks are prone to overfitting. For now you can control this by limiting the number if epochs of training in the model. Try with a very small number of epochs, in the range 1-3.<jupyter_code>####### INSERT YOUR CODE HERE<jupyter_output><empty_output><jupyter_text>GRU model Sometimes it is more effective to use a simplified version of the LSTM layer. This would be the Gated Recurrent Unit layer, which can be created in Keras as the GRU layer.<jupyter_code>from keras.layers import GRU

gru_layer = GRU(32)<jupyter_output><empty_output><jupyter_text>Similarly to the LSTM layer, if you are training your GRU network in an nVidia GPU, it is highly recommended that you use the CuDNNGRU layer instead.   Repeat the model above, replacing the LSTM layer by a GRU layer. Can you obtain better results with this simplified recurrent network?<jupyter_code>####### INSERT YOUR CODE HERE<jupyter_output><empty_output><jupyter_text>Global MaxPooling An alternative way to summarize the sequence of inputs in the LSTM/GRU layer is to use a different aggregation operation instead of returning the layer outputs at the last value of the sequence. To do this we will tell the LSTM/GRU to return a sequence, which means the layer will produce an output vector for each element in the sequence, hence generating a transformed sequence instead. For instance, in the LSTM layer this is done by defining it as follows<jupyter_code>LSTM(32, return_sequences=True)<jupyter_output><empty_output><jupyter_text>Note the **return_sequences** parameter. We still need some mechanism to summarize the sequence returned by the LSTM into a single vector. A popular method inspired by convolutional networks is to use MaxPooling. To apply this over a sequence we will use the **GlobalMaxPool1D** layer.<jupyter_code>from keras.layers import GlobalMaxPool1D
GlobalMaxPool1D()<jupyter_output><empty_output><jupyter_text>Define a new Embeddings + LSTM/GRU where the recurrent layer returns a sequence, and a GlobalMaxPool1D layer is used as a summary after that. Are you able to obtain better results in test with this architecture?<jupyter_code>####### INSERT YOUR CODE HERE<jupyter_output><empty_output><jupyter_text>Bidirectional mixing  Another improvement to the recurrent mixing architecture is to able it to read the input sequence in both directions: from left to right (standard) and from right to left. This can be implemented easily by adding a **Bidirectional** wrapper over the LSTM/GRU definition<jupyter_code>from keras.layers import Bidirectional

Bidirectional(LSTM(32, return_sequences=True))<jupyter_output><empty_output><jupyter_text>This effectively creates two LSTM layers at the same level, one for each reading direction. The outputs of these two layers are concatenated and forwarded to the next layer.  Take the best network design you created so far and make the LSTM/GRU layer bidirectional. Do you obtain better test AUC now?<jupyter_code>####### INSERT YOUR CODE HERE<jupyter_output><empty_output><jupyter_text>Stacked mixing  Much like other neural layers, LSTM/GRU layers can be stacked on top of each other to produce more complex models. To do this we should configure each layer to return sequences until the last LSTM/GRU layer in the network, which should either return only the last value in the transformed sequence or be followed by some summarizing layer (like GlobalMaxPool1D). For instance, the following would be a valid structure (note we are not adding layers to any model, this is just an example)<jupyter_code>LSTM(32, return_sequences=True)
LSTM(32, return_sequences=True)
GlobalMaxPool1D()<jupyter_output><empty_output><jupyter_text>Note also that when using several LSTM/GRU layers, it is usually not useful to the Bidirectional wrapper in the layers after the first.  Repeat the training of the previous network, but using 2 LSTM/GRU layers. Make sure to configure the first of those layers in a way that it outputs a whole sequence for the next layer. Are you able to improve the results?<jupyter_code>####### INSERT YOUR CODE HERE<jupyter_output><empty_output><jupyter_text>Fine tuning the network desing You can mix the components seen so far to improve the network further. Fine tuning their parameters can give better results:* Embeddings: size of the embedding.* LSTM/GRU layers: size of the layer, number of layers.* Bidirectional wrapper.* Max Pooling or getting the last element of the LSTM/GRU layer. Alternatively, you can also make use of [GlobalAveragePooling1D](https://keras.io/layers/pooling/), as we did in the CBoW model.Dropout can also be used to prevent overfitting. Beyond the standard Dropout layers, other forms of Dropout have been found useful in text processing:* Dropout parameters inside LSTM/GRU layers: parameters **dropout** and **recurrent_dropout** of these layers allow introducing dropout inside the blocks that process the inputs and the recurrent states.* [SpatialDropout1D](https://keras.io/layers/core/): useful for performing dropout on the embeddings layer, removes entire channels (embeddings features) for all words. This makes more sense since values within the same channel are usually highly correlated across words.  Fine-tune the network design, experimenting with the layers and parameters listed above. How much AUC can you achieve?<jupyter_code>####### INSERT YOUR CODE HERE<jupyter_output><empty_output><jupyter_text>Transfer learning The results so far can be further improved if we make use of a Language Model. This involves introducing in our model one or more pre-trained layers, that have been learned using a much larger, unsupervised dataset.To try this idea we will make use of the [FastText embeddings](https://fasttext.cc/docs/en/english-vectors.html) to initialize our Embedding layer. The following function loads from the FastText file a dictionary of embeddings for those words that have appeared in a given dictionary.<jupyter_code>import numpy as np

def load_vectors_words(fname, words):
    """Loads embeddings from a FastText file. Only loads embeddings for the given dictionary of words"""
    data = {}
    with open(fname) as fin:
        next(fin)  # Skip first line, just contains embeddings size data
        for line in fin:
            tokens = line.rstrip().split(' ')
            word = tokens[0]
            if word in words:
                data[word] = np.array(list(map(float, tokens[1:])))
    return data<jupyter_output><empty_output><jupyter_text>Download and unzip any of the embedding files for english available at the FastText website. Note that larger embedding files will likely produce better results, but will also require more hard drive space. Use the provided function to load the dictionary of the embeddings for the tokenizer words. You can access the words used by the tokenizer through the word_index attribute. Store the computed embeddings dictionary into a variable named embeddings.<jupyter_code>####### INSERT YOUR CODE HERE<jupyter_output><empty_output><jupyter_text>We can query the loaded dictionary to check what is the embedding for a given word. For instance, the embedding for  the word "stupid" would be:<jupyter_code>embeddings["stupid"]<jupyter_output><empty_output><jupyter_text>To insert these pre-trained embeddings into a Keras network, we need to create a matrix of weights that we can use to initialize the Embeddings layer. This should be done according to the indexes the Tokenizer has assigned to each word: the i-th row of the embedding matrix must correspond to the embedding learned for the i-th word in the Tokenizer. If some word found in our texts does not appear in the embeddings file we loaded, it is a good practice to initialize it following a random distribution following the mean and standard deviation of the rest of embeddings. The following function does all this work for us.<jupyter_code>def create_embedding_matrix(embeddings, tokenizer):
    """Creates a weight matrix for an Embedding layer using an embeddings dictionary and a Tokenizer"""
    
    # Compute mean and standard deviation for embeddings
    all_embs = np.stack(embeddings.values())
    emb_mean,emb_std = all_embs.mean(), all_embs.std()
    
    embedding_size = len(next(iter(embeddings.values())))
    
    embedding_matrix = np.random.normal(emb_mean, emb_std, (tokenizer.num_words, embedding_size))
    for word, i in tokenizer.word_index.items():
        if i >= tokenizer.num_words: break
        embedding_vector = embeddings.get(word)
        if embedding_vector is not None: 
            embedding_matrix[i] = embedding_vector
            
    return embedding_matrix<jupyter_output><empty_output><jupyter_text>Create the embedding weights matrix using the function above, storing it in a variable named embedding_matrix<jupyter_code>####### INSERT YOUR CODE HERE<jupyter_output><empty_output><jupyter_text>We can now check the matrix shape to ensure it has been build correctly. It must have a number of rows equal to the number of words in our tokenizer (15000), and a number of columns equal to the size of the FastText embedding (300).<jupyter_code>embedding_matrix.shape<jupyter_output><empty_output><jupyter_text>With this matrix we can create our transfer learning network! It is designed the same way as previous networks, the only difference being when creating the Embedding layer. To initialize this layer with the pre-trained embeddings, use the **embedding_matrix** you created as follows<jupyter_code>pretrained = Embedding(maxwords, embedding_matrix.shape[1], weights=[embedding_matrix], trainable=False)<jupyter_output><empty_output><jupyter_text>This creates an Embedding layer with the pre-trained FastText embeddings, provided as the *weights* parameter. Note also the pameter `trainable=False`: this tells Keras that it should not change the weights of this layer. This avoids a common problem in deep learning known as [catastrophic forgetting](https://en.wikipedia.org/wiki/Catastrophic_interference), where adjusting the network for our specific dataset might make it forget about all the pre-training. In some settings, however, it can be useful to fine-tune the weights of a pre-trained layer for our specific problem, though great care must be taken not to learn too much.  Create a network using a pretrained Embedding layer and what you learned from previous models. Has pretraining helped in improving AUC?<jupyter_code>####### INSERT YOUR CODE HERE<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.05648410045427273, 'fraction_numerical': 0.0037284649010028287, 'mean_word_length': 5.527272727272727, 'pattern_counts': {'":': 0, '<': 181, '<?xml version=': 0, '>': 182, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '45', 'dir': 'jupyter-structured-clean-dedup', 'id': '82210', 'license': 'MIT', 'n_tokens_mistral': 6208, 'n_tokens_neox': 5918, 'n_words': 3044, 'path': 'labs/lab5_toxic/toxic_student.ipynb', 'repo_name': 'albarji/curso-analisis-textos', 'size': '23334'}