<jupyter_start><jupyter_text>SemEval 2021 Task 5: Toxic Spans Detection  Research & Development<jupyter_code>import csv
import os
import random
from ast import literal_eval
from itertools import groupby
from operator import itemgetter

import numpy as np
import pandas as pd
import torch
from plotnine import *
from pytorch_lightning import (
    LightningModule,
    LightningDataModule,
    Trainer,
    seed_everything,
)
from pytorch_lightning.callbacks import ModelCheckpoint, EarlyStopping
from torch.optim import AdamW
from torch.optim.lr_scheduler import ReduceLROnPlateau
from torch.utils import data
from tqdm import tqdm
from transformers import MobileBertTokenizerFast, MobileBertForTokenClassification


pd.set_option("display.max_colwidth", None)
random.seed(a=42)
def inspect_df(df: pd.DataFrame, n: int = 5) -> pd.DataFrame:
    """Helper method to easily inspect DataFrames."""

    print(f"shape: {df.shape}")

    return df.head(n)
PROJECT_ROOT = os.getcwd().split("toxic-spans-detection")[0] + "toxic-spans-detection"<jupyter_output><empty_output><jupyter_text>Table of Contents- [Exploratory Data Analysis](Exploratory-Data-Analysis)- [A First Baseline](A-First-Baseline)- [As a Downstream Task](As-a-Downstream-Task) Our [objective](https://competitions.codalab.org/competitions/25623learn_the_details-overview) here is to train a suitable model that will be able to predict the toxic spans -if any- contained in new, unseen documents.We will treat this as a binary sequence labelling task.  Exploratory Data Analysis<jupyter_code>DATA_DIR = os.path.join(PROJECT_ROOT, "data")

trainset = pd.read_csv(os.path.join(DATA_DIR, "tsd_train.csv"))

inspect_df(trainset)
def deserialize_spans(spans: pd.Series) -> pd.Series:

    try:
        return spans.apply(literal_eval)

    except ValueError as exc:  # in case cell is executed more than once
        print(str(exc))
        return spans
# we will deserialize spans from strings to proper objects (lists)
trainset["spans"] = deserialize_spans(trainset["spans"])
# how many data points contain toxic spans?
toxic = trainset["spans"].apply(len) > 0
(
    ggplot()
    + geom_bar(mapping=aes(x=toxic), colour="white")
    + labs(title="text contains toxic spans ?", x="")
    + coord_flip()
)
# some non-toxic documents
inspect_df(trainset[~toxic])<jupyter_output>shape: (485, 2)<jupyter_text>We can observe that, although these `485` texts do not have toxic spans as annotation, some of them clearly contain toxic words, phrases etc.<jupyter_code>trainset["text"].map(len).describe()<jupyter_output><empty_output><jupyter_text>The average document in the training dataset has `204` characters, the shortest one is only `4` characters while the largest one is `1000`.<jupyter_code>trainset[toxic]["spans"].map(len).describe()<jupyter_output><empty_output><jupyter_text>The average toxic document in the training dataset has `~19` toxic characters (i.e. characters belonging to toxic character sequences).<jupyter_code>(
    ggplot()
    + geom_histogram(mapping=aes(x=trainset["text"].map(len)), colour="white", bins=30)
    + xlab("toxic characters")
)<jupyter_output><empty_output><jupyter_text>We can observe that in a typical span, let's say:$$ [ 3, 4, 5, 6, 44, 45, 46 ] $$groups of consequtive integers ($ 3 \rightarrow 6, 44 \rightarrow 46$) are basically individual character sequences.Based on that observation, we can also get a peek on the number of toxic character sequences in the training dataset.<jupyter_code>def continuous_spans(iterable: list):
    """Yield groups of continuous spans from an `iterable`.

    Example:

        iterable: [10, 11, 12, 13, 14, 15, 51, 52, 53, 54, 55, 56]

        Returns: an iterator of groups. Each group is itself an iterator that materializes in a continuous span:
                     1. [10, 11, 12, 13, 14, 15]
                     2. [51, 52, 53, 54, 55, 56]

    Inspired by: https://more-itertools.readthedocs.io/en/stable/api.html#more_itertools.consecutive_groups
    """
    for _, g in groupby(enumerate(iterable), key=lambda x: x[0] - x[1]):
        yield map(itemgetter(1), g)
charsequences = trainset[toxic]["spans"].apply(
    lambda s: [list(charseq) for charseq in continuous_spans(s)]
)
charsequences.map(len).describe()
(
    ggplot()
    + geom_histogram(mapping=aes(x=charsequences.map(len)), colour="white", bins=30)
    + xlab("toxic character sequences")
)<jupyter_output><empty_output><jupyter_text>So, a typical toxic document of the training dataset is likely to have 1 or 2 character sequences that attribute to its toxicity.  A First Baseline We will establish an $ F_1 $ score, suitable for the representations of this task.<jupyter_code>def f1(predictions, gold):
    """
    F1 (a.k.a. DICE) operating on two lists of offsets (e.g., character).

        >>> assert f1([0, 1, 4, 5], [0, 1, 6]) == 0.5714285714285714

    :param predictions: a list of predicted offsets
    :param gold: a list of offsets serving as the ground truth
    :return: a score between 0 and 1

    Originally in: https://github.com/ipavlopoulos/toxic_spans/blob/master/evaluation/semeval2021.py
    """
    if len(gold) == 0:
        return 1.0 if len(predictions) == 0 else 0.0

    if len(predictions) == 0:
        return 0.0

    predictions_set = set(predictions)
    gold_set = set(gold)

    nom = 2 * len(predictions_set.intersection(gold_set))
    denom = len(predictions_set) + len(gold_set)

    return float(nom) / float(denom)<jupyter_output><empty_output><jupyter_text>To better understand the complexity of this task, we will create a basic random sequence labeller and assess its performance.<jupyter_code>class RandomSequenceLabeller(object):
    """Random sequence labeller - returns toxic offsets at random."""

    def predict(self, doc: str) -> list:

        return [i for i, ch in enumerate(doc) if random.random() > 0.5]
# fetch the validation set
validationset = pd.read_csv(os.path.join(DATA_DIR, "tsd_trial.csv"))

validationset["text"] = validationset["text"].apply(str.lower)
validationset["spans"] = deserialize_spans(validationset["spans"])<jupyter_output><empty_output><jupyter_text>Let us isolate a document of the validation set, for demonstration purposes.<jupyter_code>validationset["text"].loc[11]
random_model = RandomSequenceLabeller()

pred = random_model.predict(doc=validationset["text"].loc[11])
print(validationset["text"].loc[11])
print("-" * 100)
print(str(pred))
f1(pred, gold=validationset["spans"].loc[11])
# and now for the whole validation dataset
valset = validationset.copy()

valset["random_predictions"] = valset["text"].apply(random_model.predict)
valset["f1_score"] = valset.apply(
    lambda s: f1(s.random_predictions, gold=s.spans), axis=1
)

inspect_df(valset)
valset["f1_score"].mean()<jupyter_output><empty_output><jupyter_text>So, averaging over the validation dataset, our random classifier achieves an F1 score of approx. `17%`.We consider this a baseline performance for this task. Let's improve on this.  As a Downstream Task We will begin with some pre-trained model, that can be fine-tuned to our task,to develop a better feel for the problem.We will utilize a [transformer](https://huggingface.co/google/mobilebert-uncased) MobileBERT model and its respective tokenizer for preprocessing.<jupyter_code>device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

device
class ToxicSpansDataset(data.Dataset):
    def __init__(self, data):
        self.data = data

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):

        row = self.data.iloc[idx]

        return {
            "tokens": torch.tensor(row["pad_tokenized_text"]).long(),
            "attention_mask": torch.tensor(row["pad_attention_mask"]).long(),
            "labels": torch.tensor(row["pad_encoded_span"]).long(),
            "pad_offset_mapping": torch.tensor(row["pad_offset_mapping"]).long(),
            "pad_span": torch.tensor(row["pad_raw_spans"]).long(),
        }
class ToxicSpansDataModule(LightningDataModule):
    def __init__(self, tokenizer, batch_size: int = 32, max_len: int = 512):

        super().__init__()

        self.data_dir = os.path.join(PROJECT_ROOT, "data")
        self.tokenizer = tokenizer
        self.batch_size = batch_size
        self.max_length = max_len
        self.trainset = None
        self.validationset = None

    def prepare_data(self, *args, **kwargs) -> None:

        # load data into memory
        trainset = pd.read_csv(os.path.join(self.data_dir, "tsd_train.csv"))
        validationset = pd.read_csv(os.path.join(self.data_dir, "tsd_trial.csv"))

        # preprocessing
        self.trainset = self._preprocess(trainset)
        self.validationset = self._preprocess(validationset)

    def _preprocess(self, df: pd.DataFrame) -> pd.DataFrame:

        df["spans"] = deserialize_spans(df["spans"])

        docs, spans = df["text"].values, df["spans"]

        data = []
        for sentence, span in tqdm(zip(docs, spans), total=len(docs)):

            encoded = self.tokenizer(
                sentence,
                add_special_tokens=True,
                padding="max_length",
                return_offsets_mapping=True,
                max_length=self.max_length,
            )

            encoded_span = np.array(
                [
                    1 if any((left <= chr_pos < right for chr_pos in span)) else 0
                    for left, right in encoded["offset_mapping"]
                ]
            )

            tokens = np.array(encoded.encodings[0].tokens)

            padded_span = np.pad(
                span,
                mode="constant",
                pad_width=(0, 1024 - len(span)),
                constant_values=-1,
            )

            data.append(
                [
                    sentence,
                    span,
                    padded_span,
                    encoded["input_ids"],
                    encoded["attention_mask"],
                    encoded["offset_mapping"],
                    encoded_span,
                    tokens,
                ]
            )

        return pd.DataFrame(
            data,
            columns=[
                "raw_text",
                "raw_spans",
                "pad_raw_spans",
                "pad_tokenized_text",
                "pad_attention_mask",
                "pad_offset_mapping",
                "pad_encoded_span",
                "pad_tokens",
            ],
        )

    def train_dataloader(self) -> data.DataLoader:

        return data.DataLoader(
            ToxicSpansDataset(self.trainset), num_workers=8, batch_size=self.batch_size
        )

    def val_dataloader(self) -> data.DataLoader:

        return data.DataLoader(
            ToxicSpansDataset(self.validationset),
            num_workers=8,
            batch_size=self.batch_size,
        )
def split_sentence(tokenizer, text: str, max_sentence_length: int, spans=None) -> tuple:

    encoded = tokenizer(text, add_special_tokens=True, return_offsets_mapping=True)

    sentences_offsets = [
        offset[1]
        for input_ids, offset in zip(encoded["input_ids"], encoded["offset_mapping"])
        if input_ids == 1012
    ]

    if len(sentences_offsets) == 0 or len(text) - sentences_offsets[-1] > 0:
        sentences_offsets.append(len(text))

    new_split = [0]
    for split_offset_id in range(len(sentences_offsets) - 1):
        if sentences_offsets[split_offset_id + 1] - new_split[-1] > max_sentence_length:
            new_split.append(sentences_offsets[split_offset_id])

    if not new_split[-1] == sentences_offsets[-1]:
        new_split.append(sentences_offsets[-1])

    if spans:
        spans_vector = [i in row["spans"] for i in range(len(row["text"]))]

    texts, new_spans, offsets = [], [], []
    for offset_id in range(len(new_split) - 1):
        a, b = new_split[offset_id], new_split[offset_id + 1]
        texts.append(text[a:b])
        offsets.append(new_split[offset_id])

        if spans:
            spans_in_subtext = [
                idx for idx, is_true in enumerate(spans_vector[a:b]) if is_true
            ]
            new_spans.append(spans_in_subtext)

    if not spans:
        new_spans = [[] for _ in range(len(offsets))]

    return texts, offsets, new_spans
class ToxicSpansModule(LightningModule):
    def __init__(self, model=None, *args, **kwargs):

        super().__init__(*args, **kwargs)

        self.model = model

    def forward(self, *args, **kwargs):
        """Use for inference only."""
        pred = self.model(*args, **kwargs)

        return pred

    def training_step(self, batch, batch_nb) -> dict:
        """The full training loop."""
        outputs = self(
            batch["tokens"],
            token_type_ids=None,
            attention_mask=batch["attention_mask"],
            labels=batch["labels"],
        )

        loss = outputs.loss
        self.log("train_loss", loss.item(), logger=True, on_step=False, on_epoch=True)

        return {"loss": loss}

    def validation_step(self, batch, batch_nb) -> None:
        """The full validation loop."""
        outputs = self(
            batch["tokens"],
            token_type_ids=None,
            attention_mask=batch["attention_mask"],
            labels=batch["labels"],
        )

        loss = outputs.loss
        self.log("val_loss", loss.item(), logger=True, on_step=False, on_epoch=True)

        logits = outputs.logits.detach().cpu().numpy()

        # get predictions from logits
        y_pred = np.argmax(logits, axis=-1).astype(int)
        # also fetch ground truth
        y_true = batch["labels"].to("cpu").numpy().astype(int)

        pad_span = batch["pad_span"].to("cpu").numpy().astype(int)
        pad_offset_mapping = batch["pad_offset_mapping"].to("cpu").numpy().astype(int)

        spanwise_f1 = []

        for i in range(len(y_true)):

            # remove '-1' padding introduced in ToxicSpansDataModule#_preprocessing
            true_spans = list(set(pad_span[i]) - {-1})

            predicted_offsets = pad_offset_mapping[i][y_pred[i].astype(bool)]
            predicted_spans = [
                j for offset in predicted_offsets for j in range(offset[0], offset[1])
            ]

            f1_score = f1(predictions=predicted_spans, gold=true_spans)
            spanwise_f1.append(f1_score)

        self.log(
            "spanwise_f1",
            np.mean(np.array(spanwise_f1)),
            prog_bar=True,
            logger=True,
            on_step=False,
            on_epoch=True,
        )

    def predict_data(self, data: pd.DataFrame, tokenizer) -> pd.DataFrame:

        self.model.eval()
        self.model.cuda()

        result = pd.DataFrame({"spans": pd.Series(np.zeros(len(data))).values})

        for i, row in tqdm(data.iterrows(), total=len(data)):
            texts, offsets, _ = split_sentence(
                tokenizer, row["text"], max_sentence_length=1000
            )

            predicted_spans = list()
            for text, offset in zip(texts, offsets):
                encoded = tokenizer(
                    text,
                    add_special_tokens=True,
                    padding="max_length",
                    truncation=True,
                    eturn_offsets_mapping=True,
                    max_length=sentence_length,
                )
                item = {
                    k: torch.tensor(v).unsqueeze(0).long().cuda()
                    for k, v in encoded.items()
                }

                output = self(
                    item["input_ids"],
                    token_type_ids=None,
                    attention_mask=item["attention_mask"],
                )
                logits = output.logits.detach().cpu().numpy()
                y_pred = np.argmax(logits, axis=-1).squeeze().astype(int)
                predicted_offsets = np.array(encoded["offset_mapping"])[
                    y_pred.astype(bool)
                ]
                spans = [
                    i
                    for offset in predicted_offsets
                    for i in range(offset[0], offset[1])
                ]
                predicted_spans.extend(list(np.array(spans) + offset))

            result.loc[i, "spans"] = str(predicted_spans)
        return result

    def configure_optimizers(self) -> dict:
        """Define optimizers and LR schedulers."""
        optimizer = AdamW(self.model.parameters(), lr=2e-5, eps=1e-8)
        scheduler = ReduceLROnPlateau(
            optimizer, mode="min", factor=0.5, patience=1, verbose=True
        )

        return {
            "optimizer": optimizer,
            "lr_scheduler": scheduler,
            "monitor": "val_loss",
        }
def train(epochs: int = 10, batch_size: int = 3):

    seed_everything(42)

    # define model checkpoints
    model_checkpoint = ModelCheckpoint(
        dirpath="checkpoints/",
        save_weights_only=True,
        save_top_k=3,
        monitor="spanwise_f1",
        mode="max",
        every_n_epochs=1,
    )

    # define training process callbacks
    early_stop_callback = EarlyStopping(
        monitor="spanwise_f1", mode="max", min_delta=0.01, patience=10, verbose=True
    )

    callbacks = [model_checkpoint, early_stop_callback]

    # use tokenizer from MobileBERT pre-trained model
    tokenizer = MobileBertTokenizerFast.from_pretrained(
        "google/mobilebert-uncased", do_lower_case=True
    )

    base = MobileBertForTokenClassification.from_pretrained(
        "google/mobilebert-uncased",
        num_labels=2,
        output_attentions=False,
        output_hidden_states=False,
    )

    # instantiate a data module
    datamodule = ToxicSpansDataModule(tokenizer=tokenizer, batch_size=batch_size)
    # instantiate a trainer
    trainer = Trainer(
        max_epochs=epochs, callbacks=callbacks, gpus=1, deterministic=True
    )

    model = ToxicSpansModule(model=base)

    # train the model
    trainer.fit(model, datamodule=datamodule)

    # fetch the held-out dataset
    testset = pd.read_csv(os.path.join(DATA_DIR, "tsd_test.csv"))

    # make predictions
    predict(model, testset)
def predict(model, predictions: pd.DataFrame) -> None:
    """Special method in order to predict on test dataset."""
    model.eval()
    model.cuda()

    # use tokenizer from MobileBERT pre-trained model
    tokenizer = MobileBertTokenizerFast.from_pretrained(
        "google/mobilebert-uncased", do_lower_case=True
    )

    result = pd.DataFrame({"spans": pd.Series(np.zeros(len(predictions))).values})

    for i, row in tqdm(predictions.iterrows(), total=len(predictions)):

        texts, offsets, _ = split_sentence(
            tokenizer, row["text"], max_sentence_length=512
        )

        predicted_spans = []
        for text, offset in zip(texts, offsets):

            encoded = tokenizer(
                text,
                add_special_tokens=True,
                padding="max_length",
                truncation=True,
                return_offsets_mapping=True,
                max_length=512,
            )

            item = {
                k: torch.tensor(v).unsqueeze(0).long().cuda()
                for k, v in encoded.items()
            }

            output = model(
                item["input_ids"],
                token_type_ids=None,
                attention_mask=item["attention_mask"],
            )
            logits = output.logits.detach().cpu().numpy()
            y_pred = np.argmax(logits, axis=-1).squeeze().astype(int)
            predicted_offsets = np.array(encoded["offset_mapping"])[y_pred.astype(bool)]
            spans = [
                i for offset in predicted_offsets for i in range(offset[0], offset[1])
            ]

            predicted_spans.extend(list(np.array(spans) + offset))

        result.loc[i, "spans"] = str(predicted_spans)

    result.to_csv(
        os.path.join(DATA_DIR, "spans-pred.txt"),
        header=False,
        sep="\t",
        quoting=csv.QUOTE_NONE,
        escapechar="\n",
    )<jupyter_output><empty_output><jupyter_text>We will now fine-tune the model.<jupyter_code>train(
    epochs=10, batch_size=3
)  # batch size > 3 yielded CUDA out of memory error in my (poor) setup<jupyter_output>Global seed set to 42
Some weights of the model checkpoint at google/mobilebert-uncased were not used when initializing MobileBertForTokenClassification: ['cls.predictions.decoder.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.predictions.dense.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.bias', 'cls.seq_relationship.weight', 'cls.predictions.transform.dense.weight']
- This IS expected if you are initializing MobileBertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing MobileBertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification [...]
{'fraction_non_alphanumeric': 0.11175972130684493, 'fraction_numerical': 0.01242820826664156, 'mean_word_length': 2.7203152364273206, 'pattern_counts': {'":': 11, '<': 45, '<?xml version=': 0, '>': 64, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '291834', 'license': 'MIT', 'n_tokens_mistral': 6759, 'n_tokens_neox': 6308, 'n_words': 1481, 'path': 'notebooks/development.ipynb', 'repo_name': 'ilias-ant/toxic-spans-detection', 'size': '21242'}