<jupyter_start><jupyter_text>Using CapsNet and Keras to Identify Toxic Online Comments[<NAME>](https://twitter.com/thenickwalsh) -- Developer Evangelist, [Datmo](https://datmo.com) BackgroundThe purpose of this notebook is to showcase the use of **Capsule Layers** / **CapsNet**, particularly in solving an NLP problem, where contextuality and superposition of data is important. You can read more CapsNets here [1](https://towardsdatascience.com/capsule-neural-networks-are-here-to-finally-recognize-spatial-relationships-693b7c99b12), [2](https://hackernoon.com/what-is-a-capsnet-or-capsule-network-2bfbe48769cc), [3](https://en.wikipedia.org/wiki/Capsule_neural_network). In this case, we'll be attempting to classify online comments as toxic or nontoxic. Toxic comments will also have a further classification, delineating between 1 of a few different "styles" of toxicity mentioned below. We'll be using **Datmo**'s [open source CLI](https://github.com/datmo/datmo) to help us get our environments sorted out and to track our experiment results and repository states along the way. The specific goal of this model, as mentioned in the original [Kaggle competition](https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge) prompt, is to create a "multi-headed model that’s capable of detecting different types of of toxicity like threats, obscenity, insults, and identity-based hate". The __training and test datasets__ are comprised of comments from Wikipedia’s talk page edits.This notebook is a fork of the final [CapsNet+GRU kernel](https://www.kaggle.com/chongjiujjin/capsule-net-with-gru) submitted to the competition by [chongjiujjin](https://www.kaggle.com/chongjiujjin/).---  Training the ModelWe can easily start importing libraries without worrying about installing them on our system, since Datmo uses Docker under the hood to setup an environment for our notebook within a container.<jupyter_code>import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)

# Input data files are available in the "input/" directory.
# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory

import os
from keras.layers import Dense,Input,LSTM,Bidirectional,Activation,Conv1D,GRU
from keras.callbacks import Callback
from keras.layers import Dropout,Embedding,GlobalMaxPooling1D, MaxPooling1D, Add, Flatten
from keras.preprocessing import text, sequence
from keras.layers import GlobalAveragePooling1D, GlobalMaxPooling1D, concatenate, SpatialDropout1D
from keras import initializers, regularizers, constraints, optimizers, layers, callbacks
from keras.callbacks import EarlyStopping,ModelCheckpoint
from keras.models import Model
from keras.optimizers import Adam
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn.metrics import roc_auc_score
print(os.listdir("input"))

# Any results you write to the current directory are saved as output.<jupyter_output>['train.csv', 'crawl-300d-2M.vec', 'glove.840B.300d.txt', 'test.csv', 'test_labels.csv']<jupyter_text>In NLP, an _embedding file_ contains dense quantified relationships between words, conveying more value than sparse techniques such as 'bag of words'. We can leverage these relationships in an _embedding layer_ later on in our model training.<jupyter_code>EMBEDDING_FILE = 'input/glove.840B.300d.txt'

train= pd.read_csv('input/train.csv')
test = pd.read_csv('input/test.csv')
# Extract the comment text from the input data (which contains indexing IDs that aren't meaningful to us)
train["comment_text"].fillna("fillna")
test["comment_text"].fillna("fillna")
X_train = train["comment_text"].str.lower()
y_train = train[["toxic", "severe_toxic", "obscene", "threat", "insult", "identity_hate"]].values

X_test = test["comment_text"].str.lower()
# Define model hyperparameters
max_features=100000
maxlen=200
embed_size=300
# Define optimization metric scoring function
class RocAucEvaluation(Callback):
    def __init__(self, validation_data=(), interval=1):
        super(Callback, self).__init__()

        self.interval = interval
        self.X_val, self.y_val = validation_data

    def on_epoch_end(self, epoch, logs={}):
        if epoch % self.interval == 0:
            y_pred = self.model.predict(self.X_val, verbose=0)
            score = roc_auc_score(self.y_val, y_pred)
            print("\n ROC-AUC - epoch: {:d} - score: {:.6f}".format(epoch+1, score))
# Tokenize input for use in model (NLP technique)
tok=text.Tokenizer(num_words=max_features,lower=True)
tok.fit_on_texts(list(X_train)+list(X_test))
X_train=tok.texts_to_sequences(X_train)
X_test=tok.texts_to_sequences(X_test)
x_train=sequence.pad_sequences(X_train,maxlen=maxlen)
x_test=sequence.pad_sequences(X_test,maxlen=maxlen)
# Load the GloVe embeddings
embeddings_index = {}
with open(EMBEDDING_FILE,encoding='utf8') as f:
    for line in f:
        values = line.rstrip().rsplit(' ')
        word = values[0]
        coefs = np.asarray(values[1:], dtype='float32')
        embeddings_index[word] = coefs

word_index = tok.word_index
#prepare embedding matrix
num_words = min(max_features, len(word_index) + 1)
embedding_matrix = np.zeros((num_words, embed_size))
for word, i in word_index.items():
    if i >= max_features:
        continue
    embedding_vector = embeddings_index.get(word)
    if embedding_vector is not None:
        # words not found in embedding index will be all-zeros.
        embedding_matrix[i] = embedding_vector
# Define model architecture. This implementation is written entirely using Keras.

from keras.layers import K, Activation
from keras.engine import Layer
from keras.layers import Dense, Input, Embedding, Dropout, Bidirectional, GRU, Flatten, SpatialDropout1D
gru_len = 128
Routings = 5
Num_capsule = 10
Dim_capsule = 16
dropout_p = 0.25
rate_drop_dense = 0.28

def squash(x, axis=-1):
    # s_squared_norm is really small
    # s_squared_norm = K.sum(K.square(x), axis, keepdims=True) + K.epsilon()
    # scale = K.sqrt(s_squared_norm)/ (0.5 + s_squared_norm)
    # return scale * x
    s_squared_norm = K.sum(K.square(x), axis, keepdims=True)
    scale = K.sqrt(s_squared_norm + K.epsilon())
    return x / scale


# A Capsule Implement with Pure Keras
class Capsule(Layer):
    def __init__(self, num_capsule, dim_capsule, routings=3, kernel_size=(9, 1), share_weights=True,
                 activation='default', **kwargs):
        super(Capsule, self).__init__(**kwargs)
        self.num_capsule = num_capsule
        self.dim_capsule = dim_capsule
        self.routings = routings
        self.kernel_size = kernel_size
        self.share_weights = share_weights
        if activation == 'default':
            self.activation = squash
        else:
            self.activation = Activation(activation)

    def build(self, input_shape):
        super(Capsule, self).build(input_shape)
        input_dim_capsule = input_shape[-1]
        if self.share_weights:
            self.W = self.add_weight(name='capsule_kernel',
                                     shape=(1, input_dim_capsule,
                                            self.num_capsule * self.dim_capsule),
                                     # shape=self.kernel_size,
                                     initializer='glorot_uniform',
                                     trainable=True)
        else:
            input_num_capsule = input_shape[-2]
            self.W = self.add_weight(name='capsule_kernel',
                                     shape=(input_num_capsule,
                                            input_dim_capsule,
                                            self.num_capsule * self.dim_capsule),
                                     initializer='glorot_uniform',
                                     trainable=True)

    def call(self, u_vecs):
        if self.share_weights:
            u_hat_vecs = K.conv1d(u_vecs, self.W)
        else:
            u_hat_vecs = K.local_conv1d(u_vecs, self.W, [1], [1])

        batch_size = K.shape(u_vecs)[0]
        input_num_capsule = K.shape(u_vecs)[1]
        u_hat_vecs = K.reshape(u_hat_vecs, (batch_size, input_num_capsule,
                                            self.num_capsule, self.dim_capsule))
        u_hat_vecs = K.permute_dimensions(u_hat_vecs, (0, 2, 1, 3))
        # final u_hat_vecs.shape = [None, num_capsule, input_num_capsule, dim_capsule]

        b = K.zeros_like(u_hat_vecs[:, :, :, 0])  # shape = [None, num_capsule, input_num_capsule]
        for i in range(self.routings):
            b = K.permute_dimensions(b, (0, 2, 1))  # shape = [None, input_num_capsule, num_capsule]
            c = K.softmax(b)
            c = K.permute_dimensions(c, (0, 2, 1))
            b = K.permute_dimensions(b, (0, 2, 1))
            outputs = self.activation(K.batch_dot(c, u_hat_vecs, [2, 2]))
            if i < self.routings - 1:
                b = K.batch_dot(outputs, u_hat_vecs, [2, 3])

        return outputs

    def compute_output_shape(self, input_shape):
        return (None, self.num_capsule, self.dim_capsule)


def get_model():
    input1 = Input(shape=(maxlen,))
    embed_layer = Embedding(max_features,
                            embed_size,
                            input_length=maxlen,
                            weights=[embedding_matrix],
                            trainable=False)(input1)
    embed_layer = SpatialDropout1D(rate_drop_dense)(embed_layer)

    x = Bidirectional(
        GRU(gru_len, activation='relu', dropout=dropout_p, recurrent_dropout=dropout_p, return_sequences=True))(
        embed_layer)
    capsule = Capsule(num_capsule=Num_capsule, dim_capsule=Dim_capsule, routings=Routings,
                      share_weights=True)(x)
    # output_capsule = Lambda(lambda x: K.sqrt(K.sum(K.square(x), 2)))(capsule)
    capsule = Flatten()(capsule)
    capsule = Dropout(dropout_p)(capsule)
    output = Dense(6, activation='sigmoid')(capsule)
    model = Model(inputs=input1, outputs=output)
    model.compile(
        loss='binary_crossentropy',
        optimizer='adam',
        metrics=['accuracy'])
    model.summary()
    return model
# Instantiate architecture
model = get_model()

# Define training job parameters
batch_size = 256
epochs = 3

#Define train-test data split for scoring model
X_tra, X_val, y_tra, y_val = train_test_split(x_train, y_train, train_size=0.95, random_state=233)
RocAuc = RocAucEvaluation(validation_data=(X_val, y_val), interval=1)
# Fit model (will take some time)
hist = model.fit(X_tra, y_tra, batch_size=batch_size, epochs=1, validation_data=(X_val, y_val),
                 callbacks=[RocAuc], verbose=1)
# Save weights from the model, allows prediction without retraining and sharing model with others.
model.save_weights('best.hdf5')<jupyter_output><empty_output><jupyter_text>Let's create a snapshot of our model and training results!Great, we've been able to recreate the model on the training data! There are a few things we're going to want to do now.**Save your notebook!** * ` File --> Save and Checkpoint`**Create a snapshot** (in your root project folder)```bash$ datmo snapshot create -m "Original capsulenet + GRU classifier" --stats acc:0.9800 --config batch_size:256 --config epochs:3```**Visualize the snapshot we just took**```bash$ datmo snapshot ls+------------------------------------------+---------------------+------------------------------------------+---------------------+--------------------------------------+-------+|                    id                    |      created at     |                  config                  |        stats        |               message                | label |+------------------------------------------+---------------------+------------------------------------------+---------------------+--------------------------------------+-------+| 7a21e186c3e6e778455eb95f27b9cb94c41bd3c6 | 2018-06-08 01:59:21 | {u'epochs': u'3', u'batch_size': u'256'} | {u'acc': u'0.9800'} | Original capsulenet + GRU classifier |  None |+------------------------------------------+---------------------+------------------------------------------+---------------------+--------------------------------------+-------+```  Creating predictions for the Kaggle submission dataGiven a test set from the competition, we can use our model to create predictions for classifying the unlabeled  comments present in `test.csv`.<jupyter_code># Perform prediction on the Kaggle test set previously imported
y_pred = model.predict(x_test, batch_size=1024, verbose=1)
# Create a pandas dataframe from the model predictions (prev numpy array)
prediction_df = pd.DataFrame(y_pred, columns=["toxic", "severe_toxic", "obscene", "threat", "insult", "identity_hate"])
prediction_df.head(15)
# Let's combine the original comments with the model predictions so we can more easily make sense of the results.
combined_df = test.join(prediction_df)
combined_df.head(15)
# Saving our dataframe to file
combined_df.to_csv('kaggle_results.csv', index=False)<jupyter_output><empty_output><jupyter_text>We've finished writing out our Kaggle results, let's take a snapshot.We can create another Datmo snapshot here to save the state of our environment, notebook, and files now that we're finished with our Kaggle results. This means we could always revert back to this snapshot's state should we decide to keep tinkering, or simply have a clean checkpoint for others to revert to should they want to reproduce our work up to this point.**Save your notebook!** * ` File --> Save and Checkpoint`**Create a snapshot** (in your root project folder)```bash$ datmo snapshot create -m "CapsNet prediction on Kaggle test data" --stats time:243s --config results_file:kaggle_results.csv ```**Visualize the snapshot we just took** Notice, we can see the snapshot we just created, as well as the one we took immediately following training!```bash$ datmo snapshot ls+------------------------------------------+---------------------+------------------------------------------+---------------------+----------------------------------------+-------+|                    id                    |      created at     |                  config                  |        stats        |                message                 | label |+------------------------------------------+---------------------+------------------------------------------+---------------------+----------------------------------------+-------+| f398e5f6c50fc5a66f45bf77e81bcbcdeaafcdb8 | 2018-06-08 23:04:05 | {u'results_file': u'kaggle_results.csv'} |  {u'time': u'243s'} | CapsNet prediction on Kaggle test data |  None || 7a21e186c3e6e778455eb95f27b9cb94c41bd3c6 | 2018-06-08 01:59:21 | {u'epochs': u'3', u'batch_size': u'256'} | {u'acc': u'0.9800'} |  Original capsulenet + GRU classifier  |  None |+------------------------------------------+---------------------+------------------------------------------+---------------------+----------------------------------------+-------+``` --- Using our saved model to predict on new dataAlthough the Kaggle competition is finished, we can use our model to classify other comments and see what it thinks!Because the model weights are available (either saved by the user after training, or using the weights provided from the repository), we can perform this classification without needing to retrain the model.  1. Instantiate the model architectureThis cell is only necessary if you are running the notebook without performing the training in the same session.Note: because we're using a custom layer (Capsule), we'll need to first redefine the architecture beforeloading in the weights.<jupyter_code>from keras.layers import K, Activation
from keras.engine import Layer
from keras.layers import Dense, Input, Embedding, Dropout, Bidirectional, GRU, Flatten, SpatialDropout1D
gru_len = 128
Routings = 5
Num_capsule = 10
Dim_capsule = 16
dropout_p = 0.25
rate_drop_dense = 0.28

def squash(x, axis=-1):
    # s_squared_norm is really small
    # s_squared_norm = K.sum(K.square(x), axis, keepdims=True) + K.epsilon()
    # scale = K.sqrt(s_squared_norm)/ (0.5 + s_squared_norm)
    # return scale * x
    s_squared_norm = K.sum(K.square(x), axis, keepdims=True)
    scale = K.sqrt(s_squared_norm + K.epsilon())
    return x / scale


# A Capsule Layer implementation with pure Keras
class Capsule(Layer):
    def __init__(self, num_capsule, dim_capsule, routings=3, kernel_size=(9, 1), share_weights=True,
                 activation='default', **kwargs):
        super(Capsule, self).__init__(**kwargs)
        self.num_capsule = num_capsule
        self.dim_capsule = dim_capsule
        self.routings = routings
        self.kernel_size = kernel_size
        self.share_weights = share_weights
        if activation == 'default':
            self.activation = squash
        else:
            self.activation = Activation(activation)

    def build(self, input_shape):
        super(Capsule, self).build(input_shape)
        input_dim_capsule = input_shape[-1]
        if self.share_weights:
            self.W = self.add_weight(name='capsule_kernel',
                                     shape=(1, input_dim_capsule,
                                            self.num_capsule * self.dim_capsule),
                                     # shape=self.kernel_size,
                                     initializer='glorot_uniform',
                                     trainable=True)
        else:
            input_num_capsule = input_shape[-2]
            self.W = self.add_weight(name='capsule_kernel',
                                     shape=(input_num_capsule,
                                            input_dim_capsule,
                                            self.num_capsule * self.dim_capsule),
                                     initializer='glorot_uniform',
                                     trainable=True)

    def call(self, u_vecs):
        if self.share_weights:
            u_hat_vecs = K.conv1d(u_vecs, self.W)
        else:
            u_hat_vecs = K.local_conv1d(u_vecs, self.W, [1], [1])

        batch_size = K.shape(u_vecs)[0]
        input_num_capsule = K.shape(u_vecs)[1]
        u_hat_vecs = K.reshape(u_hat_vecs, (batch_size, input_num_capsule,
                                            self.num_capsule, self.dim_capsule))
        u_hat_vecs = K.permute_dimensions(u_hat_vecs, (0, 2, 1, 3))
        # final u_hat_vecs.shape = [None, num_capsule, input_num_capsule, dim_capsule]

        b = K.zeros_like(u_hat_vecs[:, :, :, 0])  # shape = [None, num_capsule, input_num_capsule]
        for i in range(self.routings):
            b = K.permute_dimensions(b, (0, 2, 1))  # shape = [None, input_num_capsule, num_capsule]
            c = K.softmax(b)
            c = K.permute_dimensions(c, (0, 2, 1))
            b = K.permute_dimensions(b, (0, 2, 1))
            outputs = self.activation(K.batch_dot(c, u_hat_vecs, [2, 2]))
            if i < self.routings - 1:
                b = K.batch_dot(outputs, u_hat_vecs, [2, 3])

        return outputs

    def compute_output_shape(self, input_shape):
        return (None, self.num_capsule, self.dim_capsule)


def get_model():
    input1 = Input(shape=(maxlen,))
    embed_layer = Embedding(max_features,
                            embed_size,
                            input_length=maxlen,
                            weights=[embedding_matrix],
                            trainable=False)(input1)
    embed_layer = SpatialDropout1D(rate_drop_dense)(embed_layer)

    x = Bidirectional(
        GRU(gru_len, activation='relu', dropout=dropout_p, recurrent_dropout=dropout_p, return_sequences=True))(
        embed_layer)
    capsule = Capsule(num_capsule=Num_capsule, dim_capsule=Dim_capsule, routings=Routings,
                      share_weights=True)(x)
    # output_capsule = Lambda(lambda x: K.sqrt(K.sum(K.square(x), 2)))(capsule)
    capsule = Flatten()(capsule)
    capsule = Dropout(dropout_p)(capsule)
    output = Dense(6, activation='sigmoid')(capsule)
    model = Model(inputs=input1, outputs=output)
    model.compile(
        loss='binary_crossentropy',
        optimizer='adam',
        metrics=['accuracy'])
    model.summary()
    return model
# Load the weights from disk for our model object
model.load_weights('best.hdf5')<jupyter_output><empty_output><jupyter_text>2. Predict on a handful of manually defined strings<jupyter_code>X_test = ["this is an innocent comment", "stfu baddie", "shit fuck piss"] # create list of strings to test
comment_df = pd.DataFrame(data=X_test,columns=["comment_text"]) # create dataframe from this list for later

# tokenize comments for use with the model
X_test=tok.texts_to_sequences(X_test)
x_test=sequence.pad_sequences(X_test,maxlen=maxlen)

# perform prediction
predictions = model.predict(x_test, batch_size=1024, verbose=1)[0:len(X_test)]
# Visualize results
prediction_df = pd.DataFrame(predictions, columns=["toxic", "severe_toxic", "obscene", "threat", "insult", "identity_hate"])
combined_df = comment_df.join(prediction_df) # join the comment dataframe with the results dataframe
combined_df.head(15)
# Optional: Write dataframe out to CSV
combined_df.to_csv('manual_comment_results.csv', index=False)<jupyter_output><empty_output><jupyter_text>3. Predicting on a new CSV of commentsNow that you understand how the code works on a handful of arbitrary strings, we can go a step further and perform prediction on a larger comment dataset loaded from a CSV file.<jupyter_code># Load in your own data
new_comments_df = pd.read_csv('input/test.csv') # Replace 'test.csv' with your dataset
X_test = test["comment_text"].str.lower() # Replace "comment_text" with the label of the column containing your comments
# Tokenizing your data for use with the model
X_test=tok.texts_to_sequences(X_test)
x_test=sequence.pad_sequences(X_test,maxlen=maxlen)
# Perform predictions and concatenate data into one dataframe
predictions = model.predict(x_test, batch_size=1024, verbose=1)[0:len(X_test)]
prediction_df = pd.DataFrame(predictions, columns=["toxic", "severe_toxic", "obscene", "threat", "insult", "identity_hate"])
combined_df = new_comments_df.join(prediction_df) # join the comment dataframe with the results dataframe

combined_df.head(15)
# Optional: Create an output folder to store your predictions

outdir = './output'
if not os.path.exists(outdir):
    os.mkdir(outdir)
# Optional: Create an output folder and write dataframe out to CSV

outname = 'user_dataset_results.csv' # You can rename this file to whatever you'd like. Using the same output filename multiple times could cause overwriting, be careful!
fullpath = os.path.join('./output', outname)  

combined_df.to_csv(fullpath, index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1408005617977528, 'fraction_numerical': 0.015712780898876403, 'mean_word_length': 2.7157534246575343, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 28, 'https://': 9, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 35, 'profanity_word_list.json': 1, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '130051', 'license': 'MIT', 'n_tokens_mistral': 7062, 'n_tokens_neox': 6615, 'n_words': 1812, 'path': 'toxic-comment-identification/capsnet-keras.ipynb', 'repo_name': 'datmo/datmo-tutorials', 'size': '22788'}