<jupyter_start><jupyter_text>Toxic Text Segmentation Start by importing all dependencies<jupyter_code>import csv
import chars2vec 
import re
import numpy as np
from tensorflow.keras import datasets, layers, models, losses, callbacks, Model, backend
from sklearn.model_selection import train_test_split
from utils import f1, read_text_data, read_data_span
from models import get_SegNet, get_DeconvNet, get_UNet
import statistics<jupyter_output><empty_output><jupyter_text>Read data<jupyter_code>texts = read_text_data('data/tsd_train_readable.csv')
spans = read_data_span('data/tsd_train_readable.csv')
texts.extend(read_text_data('data/tsd_trial_readable.csv'))
spans.extend(read_data_span('data/tsd_trial_readable.csv'))


processed_texts = []
processed_spans = []
print(f"Lengths equal: {len(texts)==len(spans)}" + "\n")<jupyter_output>Lengths equal: True<jupyter_text>Get the maximum comment size (in no. of chars)<jupyter_code>max_size = 0
for i in range(0, len(texts)-1):
    if len(texts[i]) > max_size:
        max_size = len(texts[i])
print(f"max size of sentence (in chars): {max_size}")<jupyter_output><empty_output><jupyter_text>Preprocess the training data, after analysis the max sentence size is 1000 characters long, also removing empty strings and split the spans in to actual lists<jupyter_code>c2v_model = chars2vec.load_model('eng_50')
word_limit = 1024
for i in range(0, len(texts)-1):
    to_use = True
    if len(texts[i]) > word_limit:
        to_use = False
    if texts[i] == "":
        to_use = False
    new_spans = [int(j) for j in spans[i][1:-1].split(", ")]
    if max(new_spans) > len(texts[i]) - 1:
        to_use = False
    if to_use:
        if spans[i] != []:
            full_span = [[0,0,1] for j in range(0, word_limit)]
            for char_offset in new_spans:
                full_span[char_offset] = [1,0,0]
            for j in range(0, len(texts[i])-1):
                if full_span[j][1] == 0 and full_span[j][2] == 1:
                    full_span[j] = [0,1,0]
        else:
            full_span = [[1,0,0] for j in range(0, len(texts[i]))]           
        processed_texts.append(texts[i])
        processed_spans.append(full_span)<jupyter_output><empty_output><jupyter_text>Define the training and testing datasets with numpy zero arrays, this is to allow us to pad the endOf the toxic span with zeros as it is a fully convolutional network<jupyter_code>train_Y = np.zeros(shape=(len(processed_spans), 1024, 3))
train_X = np.zeros(shape=(len(processed_texts), 1024, 50))<jupyter_output><empty_output><jupyter_text>Terrible Python best practise but you might wanna manually free up some memory. This is going to be a very large compuation<jupyter_code>del texts
del spans<jupyter_output><empty_output><jupyter_text>Build Train_X<jupyter_code>for x, string in enumerate(processed_texts):
    for y, char in enumerate(string):
            char_vect = c2v_model.vectorize_words([char])
            train_X[x][y] = [word_vect for word_vect in char_vect[0]]<jupyter_output><empty_output><jupyter_text>Build train_Y<jupyter_code>for x, label in enumerate(processed_spans):
    for y, output in enumerate(label):
        train_Y[x][y] = output<jupyter_output><empty_output><jupyter_text>Split train_X and train_Y into training and validation datasets.<jupyter_code>train_X, val_X, train_Y, val_Y = train_test_split(train_X, train_Y, test_size=0.1, random_state=42)<jupyter_output><empty_output><jupyter_text>Confirm the shape of the train and val datasets, should be ([sample_size], 1000, 50) and ([sample_size], 1000, 3) respectively<jupyter_code>print(train_X.shape)
print(train_Y.shape)<jupyter_output><empty_output><jupyter_text>As a global variable would be out of scope for the callback object class manually create a HighScore class<jupyter_code>class HighScore:
    def __init__(self):
        self.high_score = 0
    def get_high_score(self):
        return self.high_score
    def set_high_score(self, new_score):
        self.high_score = new_score
high_score = HighScore()<jupyter_output><empty_output><jupyter_text>Free up more memory<jupyter_code>del processed_texts
del processed_spans<jupyter_output><empty_output><jupyter_text>A prediction callback to act as a validation step, as the tensor is of a different shape to the F1 score of SemEval we must Convert it into it's proper form before checking the F1 score.<jupyter_code>class PredictionCallback(callbacks.Callback):    
    def on_epoch_end(self, epoch, logs={}):
        y_pred = self.model.predict(val_X)
        scores = []
        for x, pred in enumerate(y_pred):
            score = f1([j for j, i in enumerate(pred) if np.argmax(i) == 0], [j for j, i in enumerate(val_Y[x]) if np.argmax(i) == 0])
            scores.append(score)
        score = statistics.mean(scores)
        if score > high_score.get_high_score():
            high_score.set_high_score(score)
            model.save(f"Example")
        print(f"F1 score: {score}")<jupyter_output><empty_output><jupyter_text>Get the model and view it's structure.<jupyter_code>model = get_UNet(is_deep=False, is_sko=False)
model.summary()<jupyter_output><empty_output><jupyter_text>Train the model<jupyter_code>history = model.fit(train_X, train_Y, epochs=300, batch_size=32, callbacks=[PredictionCallback()])<jupyter_output><empty_output><jupyter_text>Save model and check final validation score<jupyter_code>model.save(f"UNet_model_example")
scores = []
pred_Y = model.predict(val_X)

for x, pred in enumerate(pred_Y):
    y_pred_f1_compatible = [j for j, i in enumerate(pred) if np.argmax(i) == 0]
    y_true_f1_compatible = [j for j, i in enumerate(val_Y[x]) if np.argmax(i) == 0]
    score = f1(y_pred_f1_compatible, y_true_f1_compatible)
    scores.append(score)

print('avg F1 %g' % statistics.mean(scores))<jupyter_output><empty_output><jupyter_text>Manually view some predictions to check validity<jupyter_code>for x, pred in enumerate(pred_Y):
    score = f1([j for j, i in enumerate(pred) if np.argmax(i) == 0], [j for j, i in enumerate(val_Y[x]) if np.argmax(i) == 0])
    print(f"F1 score: {score}")
    print(f"Predicted categorical: {[np.argmax(i) for i in pred]}")
    print(f"Predicted span: {[j for j, i in enumerate(pred) if np.argmax(i) == 0]}")
    print(f"Ground truth span: {[j for j, i in enumerate(val_Y[x]) if np.argmax(i) == 0]}" + "\n")
    if x == 100:
          break<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1150663544106167, 'fraction_numerical': 0.015456674473067917, 'mean_word_length': 4.7556154537286615, 'pattern_counts': {'":': 1, '<': 68, '<?xml version=': 0, '>': 72, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '594798', 'license': 'MIT', 'n_tokens_mistral': 2351, 'n_tokens_neox': 2240, 'n_words': 550, 'path': 'Toxic_Text_Segmentatio_model.ipynb', 'repo_name': 'TheElectricFlock/SemEval-2020-task-5', 'size': '6405'}