<jupyter_start><jupyter_text>When pretrained model is loaded in Colab, it throws an error during predicting.<jupyter_code>FOLDER = 2
try:
  import google.colab
  IN_COLAB = True
except:
  IN_COLAB = False

if IN_COLAB:
  from google.colab import drive
  drive.mount("/gdrive", force_remount=True)


  !pip uninstall -y kaggle
  !pip install kaggle
  !mkdir /root/.kaggle
  !cp /gdrive/MyDrive/kaggle/kaggle.json /root/.kaggle

  !kaggle competitions download -c  feedback-prize-2021
  !mkdir -p ../input/feedback-prize-2021
  !unzip -n -qq 'feedback-prize-2021.zip' -d ../input/feedback-prize-2021

  !kaggle datasets download -d  nbroad/debertav2xlfasttokenizer
  !mkdir -p ../input/debertav2xlfasttokenizer
  !unzip -n -qq debertav2xlfasttokenizer.zip -d ../input/debertav2xlfasttokenizer

  !kaggle datasets download -d  bobber/10creatingfoldsproperlyhopefullyp
  !mkdir -p ../input/10creatingfoldsproperlyhopefullyp
  !unzip -n -qq 10creatingfoldsproperlyhopefullyp.zip -d ../input/10creatingfoldsproperlyhopefullyp

  # !ln -s /gdrive/MyDrive/kaggle/feedback-prize-2021/blurr-models/bigbird-roberta-base models
  !sudo apt-get install sshfs
  !mkdir output
  !mkdir -p ~/.ssh
  # !ssh-keyscan 172.16.58.3 >> ~/.ssh/known_hosts
  # !sudo sshfs -o allow_other,default_permissions,IdentityFile=/gdrive/MyDrive/kaggle/ssh-key-2021-12-29-kaggle.key \
  #   opc@172.16.58.3:/models/feedback-prize-2021/deberta-v2-xlarge-bs-8/folder{FOLDER} output
  !ssh-keyscan 172.16.17.32 >> ~/.ssh/known_hosts
  !sudo sshfs -o allow_other,default_permissions,IdentityFile=/gdrive/MyDrive/kaggle/magic.key \
    opc@172.16.17.32:/models/feedback-prize-2021/deberta-v3-large/folder{FOLDER} output

  !pip install transformers
  !pip install datasets
  !pip install tensorboardX<jupyter_output>Mounted at /gdrive
Found existing installation: kaggle 1.5.12
Uninstalling kaggle-1.5.12:
  Successfully uninstalled kaggle-1.5.12
Collecting kaggle
  Downloading kaggle-1.5.12.tar.gz (58 kB)
[K     |████████████████████████████████| 58 kB 6.8 MB/s 
[?25hRequirement already satisfied: six>=1.10 in /usr/local/lib/python3.7/dist-packages (from kaggle) (1.15.0)
Requirement already satisfied: certifi in /usr/local/lib/python3.7/dist-packages (from kaggle) (2021.10.8)
Requirement already satisfied: python-dateutil in /usr/local/lib/python3.7/dist-packages (from kaggle) (2.8.2)
Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from kaggle) (2.23.0)
Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from kaggle) (4.63.0)
Requirement already satisfied: python-slugify in /usr/local/lib/python3.7/dist-packages (from kaggle) (6.1.1)
Requirement already satisfied: urllib3 in /usr/local/lib/python3.7/dist-packages (from kaggle) (1.24.3)[...]<jupyter_text>8-bit Adam Optimization 👾 The optimizer is responsible for computing the gradient statistics for back propagation. These calculations are typically done on 32-bit values, but this notebook demonstrates how to use an 8-bit optimizer that saves memory and increases speed. The problem with reducing the number of bits is that the precision of each value decreases. <NAME> ([@timdettmers](https://www.kaggle.com/timdettmers)) did research at Facebook to figure out how to do stable optimization using 8 bits using a clever quantization trick. For a more detailed look at his research, please [read his paper](https://arxiv.org/abs/2110.02861) or [view his humorous video](https://www.youtube.com/watch?v=IxrlHAJtqKE). The GitHub repo, which contains installation instructions for your specific GPU, can be found here: https://github.com/facebookresearch/bitsandbytes  It was found that this allows for slightly faster training and for slightly larger models to be loaded into memory without sacrificing performance.  🔥 v18 Update: Thanks to @gregorlied for noticing that the embeddings were not optimized at 32-bits when using 8-bit Adam. Version 18 fixes that. 🔥🔥 v24 Update: deberta-v2-xlarge shown to train only with 8-bit Adam and not with 32-bit Adam (Version 22). This notebook also shows how to use fast version of deberta v2 tokenizer (not yet merged to master branch) 🔥🔥🔥 v25 Update: A more thorough testing of 8-bit vs 32-bit Adam was carried out. [Report available here](https://wandb.ai/nbroad/feedback-prize/reports/8-bit-Adam-vs-32-bit-Adam-tests--VmlldzoxNDgwNTc1) tldr: 8bit Adam is slower but saves memory and results in a higher CV. In this notebook, I compare the training times between the regular 32-bit Adam and the 8-bit Adam optimizer when training longformer-large for 1 epoch using a maximum sequence length of 2048. To use 8-bit Adam, you need to install the library and then change the one [line where the optimizer gets created](Optimizer). In some cases using 8-bit Adam allows for larger batch sizes. Not this time, though 😅 If you want to jump straight to the results (1 run, 1 epoch comparison), [click here](Weights-and-Biases-Report-✨) If you want to see the notebook during the actual runs, version 1 has 8-bit Adam and version 3 has 32-bit Adam. I think 8-bit Adam is mostly useful for training large language models from scratch, and less for finetuning models with < 1B parameters. Perhaps the best use-case for Kaggle would be for the users who don't have any other compute and batch size of 1 just barely doesn't fit using 32-bit Adam. In that instance, 8-bit Adam would allow people to use Kaggle GPUs to train models that wouldn't fit otherwise. (Like in v24) The speed-ups are modest as seen in this image from the paper below.![8bit speed table](https://pbs.twimg.com/media/FBLeOZnVEAkt-Ij?format=png&name=small) 8 bit optimization also enables fitting bigger models on smaller GPUs: ![8bit fit big models](https://pbs.twimg.com/media/FBLeMS_VIBEettR?format=png&name=900x900) [Tim's announcement tweet](https://twitter.com/Tim_Dettmers/status/1446472128979562499?s=20)   Install necessary libraries 📚You must install the right version of `bitsandbytes` according to the GPU's CUDA version.<jupyter_code># Take note of what cuda version you have by running either of the following commands
# !conda list | grep cudatoolkit
!nvidia-smi

# choices: {cuda92, cuda 100, cuda101, cuda102, cuda110, cuda111, cuda113}
# replace XXX with the respective number
# pip install bitsandbytes-cudaXXX
# I believe Colab is 11.2 so the command would be: pip install bitsandbytes-cuda112
if IN_COLAB:
    !pip install bitsandbytes-cuda112 -q
else:
    #Kaggle need to use 110
    !pip install bitsandbytes-cuda110 -q
# !pip install -U wandb -q
!pip install seqeval git+https://github.com/huggingface/transformers.git -q
!pip install sentencepiece -q
# This tests if the installation was successful
!wget https://gist.githubusercontent.com/TimDettmers/1f5188c6ee6ed69d211b7fe4e381e713/raw/4d17c3d09ccdb57e9ab7eca0171f2ace6e4d2858/check_bnb_install.py && python check_bnb_install.py<jupyter_output>--2022-03-14 15:01:58--  https://gist.githubusercontent.com/TimDettmers/1f5188c6ee6ed69d211b7fe4e381e713/raw/4d17c3d09ccdb57e9ab7eca0171f2ace6e4d2858/check_bnb_install.py
Resolving gist.githubusercontent.com (gist.githubusercontent.com)... 192.168.127.12, 172.16.31.10, 172.16.17.32, ...
Connecting to gist.githubusercontent.com (gist.githubusercontent.com)|192.168.127.12|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 327 [text/plain]
Saving to: ‘check_bnb_install.py’


check_bnb_install.p   0%[                    ]       0  --.-KB/s               
check_bnb_install.p 100%[===================>]     327  --.-KB/s    in 0s      

2022-03-14 15:01:58 (18.2 MB/s) - ‘check_bnb_install.py’ saved [327/327]

SUCCESS!
Installation was successful!<jupyter_text>The following cell is only necessary if you want to use the fast tokenizer for deberta-v2<jupyter_code>if True:
  from pathlib import Path

  if IN_COLAB:
    transformers_path = Path("/usr/local/lib/python3.7/dist-packages/transformers")
  else:
    transformers_path = Path("/opt/conda/lib/python3.7/site-packages/transformers")

  import shutil

  input_dir = Path("../input/debertav2xlfasttokenizer")

  convert_file = input_dir / "convert_slow_tokenizer.py"
  conversion_path = transformers_path/convert_file.name

  if conversion_path.exists():
      conversion_path.unlink()

  shutil.copy(convert_file, transformers_path)
  deberta_v2_path = transformers_path / "models" / "deberta_v2"

  for filename in ['tokenization_deberta_v2.py', 'tokenization_deberta_v2_fast.py']:
      filepath = deberta_v2_path/filename
      if filepath.exists():
          filepath.unlink()

      shutil.copy(input_dir/filename, filepath)
import os
import json
import math
from pathlib import Path
from functools import partial

import numpy as np
import pandas as pd
import torch
from tqdm.notebook import tqdm
import datasets
from transformers import (
    TrainingArguments, 
    Trainer, 
    AutoConfig,
    AutoModelForTokenClassification,
    AutoTokenizer,
    get_scheduler,
    DataCollatorForTokenClassification,
    EarlyStoppingCallback
)
import bitsandbytes as bnb<jupyter_output><empty_output><jupyter_text>Config<jupyter_code>class CFG:
    
    fold = FOLDER
    
    # model_name = "../input/debertav2xlfasttokenizer/microsoft-deberta-v2-xlarge"
    # model_name = "microsoft/deberta-v2-xlarge"
    model_name = "microsoft/deberta-v3-large"
    # model_name = "funnel-transformer/medium"
    # model_name = "./output"
    
    max_seq_length = 512*8
    text_column = "text"
    label_column = "labels"
    word_id_column = "word_ids"
    adam_bits = 8 # Choose from {8,32}

    training_args = TrainingArguments(
        output_dir="output",
        logging_dir="output/log",
        log_level='debug',
        overwrite_output_dir=True,
        # overwrite_output_dir=False,
        do_train=True,
        do_eval=True,
        evaluation_strategy="steps",
        eval_steps=800,
        # eval_steps=None, # change evaluation_strategy to steps to use this
        per_device_train_batch_size=2,
        per_device_eval_batch_size=2,
        gradient_accumulation_steps=4,
        learning_rate=2e-5,
        weight_decay=0.01,
        num_train_epochs=5,
        max_steps=-1, # set >0 to limit
        lr_scheduler_type="cosine",#"linear",
        warmup_ratio=0.1,
        logging_strategy="steps",
        logging_steps=100,
        save_strategy="epoch",
        # save_strategy="no",
        save_steps=None,
        seed=18,
        fp16=True, 
        dataloader_num_workers=4,
        run_name=f"deb-v2-xl-{adam_bits}bitAdam",
        group_by_length=True, # This can also help speed training
#         report_to="wandb",
        report_to="tensorboard",
        resume_from_checkpoint=None,
        # resume_from_checkpoint="output/checkpoint-3508",
        gradient_checkpointing=True, # save memory, but backwards pass slower
        # gradient_checkpointing=False, # save memory, but backwards pass slower
        # load_best_model_at_end=True
    )

# for convenience
args = CFG.training_args
# I use weights and biases to track training.
# The following code requires attaching a secret to the notebook.
if "wandb" in args.report_to:
    import wandb
    from kaggle_secrets import UserSecretsClient
    user_secrets = UserSecretsClient()
    key = user_secrets.get_secret("wandb")
    
    wandb.login(key=key)
    os.environ["WANDB_PROJECT"] = "feedback-prize"<jupyter_output><empty_output><jupyter_text>Create datasetThis is a little slow so it would probably be wise to create your dataset in another notebook and then load it in to the training notebook.<jupyter_code>%%time

if not Path("full_dataset.dataset").exists():
    texts, ids = [], []
    for file in tqdm(Path("../input/feedback-prize-2021/train").glob("*.txt"), total=15594, desc="Reading train texts"):
        ids.append(file.stem)

        with open(file) as fp:
            texts.append(fp.read())
        
    
def add_label_information(examples):
    
    texts = examples[CFG.text_column]
    ids = examples["id"]
    all_labels, folds, words = [], [], []
    
    for text, id_ in zip(texts, ids):
    
        df = train_df[train_df["id"]==id_]

        text = text.split()
        num_words = len(text)

        labels = ["O"]*num_words

        for discourse_type, predictionstring in df[["discourse_type", "predictionstring"]].values:

            first = True
            for word_id in map(int, predictionstring.split()):
                prefix = "I-"
                if first:
                    prefix = "B-"
                    first = False
                labels[word_id] = prefix+discourse_type
                
        all_labels.append(labels)
        folds.append(df["kfold"].values[0])
        words.append(text)

    examples[CFG.label_column] = all_labels
    examples["fold"] = folds
    examples[CFG.text_column] = words
    return examples
    

# Using fold strategy shown by Abhishek https://www.kaggle.com/abhishek/creating-folds-properly-hopefully-p/
train_df = pd.read_csv("../input/10creatingfoldsproperlyhopefullyp/train_folds.csv", usecols=["id", "discourse_type", "predictionstring", "kfold"])


# This step can take a few minutes
if not Path("full_dataset.dataset").exists(): # This was for when I was debugging so I wouldn't run this multiple times
    temp_dataset = datasets.Dataset.from_dict({"id": ids, CFG.text_column: texts})
    temp_dataset = temp_dataset.map(add_label_information, batched=True, num_proc=args.dataloader_num_workers)

    full_dataset = datasets.DatasetDict()
    full_dataset["train"] =  temp_dataset.filter(lambda x: x["fold"]!=CFG.fold)
    full_dataset["validation"] =  temp_dataset.filter(lambda x: x["fold"]==CFG.fold)
    full_dataset.save_to_disk("full_dataset.dataset")
else:
    full_dataset = datasets.DatasetDict.load_from_disk("full_dataset.dataset")
full_dataset<jupyter_output><empty_output><jupyter_text>Tokenizing dataDon't pad to max length unless you are on a TPU or you really want to extend your training<jupyter_code># DebertaV2TokenizerFast is not in the transformers library. This patches it in

def get_DebertaV2TokenizerFast(model_name):        
    from transformers.models.deberta_v2.tokenization_deberta_v2_fast import DebertaV2TokenizerFast
    return DebertaV2TokenizerFast.from_pretrained(model_name)
# https://github.com/huggingface/transformers/blob/669e3c50c98ad5b506555a551d2ecbf72ceb3c99/examples/pytorch/token-classification/run_ner.py#L371
def tokenize_and_align_labels(examples, label2id, return_word_ids=False):
    tokenized_inputs = tokenizer(
        examples[CFG.text_column],
        truncation=True,
        max_length=CFG.max_seq_length,
        # We use this argument because the texts in our dataset are lists of words (with a label for each word).
        is_split_into_words=True,
    )
    labels = []
    all_word_ids = []
    for i, label in enumerate(examples[CFG.label_column]):
        word_ids = tokenized_inputs.word_ids(batch_index=i)
        previous_word_idx = None
        label_ids = []
        for word_idx in word_ids:
            # Special tokens have a word id that is None. We set the label to -100 so they are automatically
            # ignored in the loss function.
            if word_idx is None:
                label_ids.append(-100)
            else:
                label_ids.append(label2id[label[word_idx]])
            previous_word_idx = word_idx
            
        if return_word_ids:
            all_word_ids.append(word_ids)

        labels.append(label_ids)
    
    tokenized_inputs[CFG.label_column] = labels
    
    if return_word_ids:
        tokenized_inputs[CFG.word_id_column] = all_word_ids
    
    return tokenized_inputs
%%time

# label_list = ['O', 'B-Claim', 'I-Claim', 'B-Concluding Statement', 'I-Concluding Statement', 
#               'B-Counterclaim', 'I-Counterclaim', 'B-Evidence', 'I-Evidence','B-Lead', 'I-Lead', 
#               'B-Position', 'I-Position', 'B-Rebuttal', 'I-Rebuttal']
label_list = ['B-Lead', 'I-Lead', 'B-Position', 'I-Position', 'B-Evidence', 'I-Evidence', 
              'B-Claim', 'I-Claim', 'B-Concluding Statement', 'I-Concluding Statement', 
              'B-Counterclaim', 'I-Counterclaim', 'B-Rebuttal', 'I-Rebuttal', 'O']

label2id = {label:id_ for id_, label in enumerate(label_list)}
id2label = {id_:label for id_, label in enumerate(label_list)}


if "deberta" in CFG.model_name:
    tokenizer = get_DebertaV2TokenizerFast(CFG.model_name)
else:
    tokenizer = AutoTokenizer.from_pretrained(CFG.model_name, add_prefix_space=True)
    # tokenizer = AutoTokenizer.from_pretrained(CFG.model_name)

if Path("train.dataset").exists():
    train_dataset = datasets.Dataset.load_from_disk("train.dataset")
else:
    train_dataset = full_dataset["train"].map(
            partial(
                tokenize_and_align_labels,
                label2id=label2id,
                return_word_ids=False
        ),
        batched=True,
        num_proc=args.dataloader_num_workers,
        remove_columns=["fold", "text", "id"]
    )
    
    train_dataset.save_to_disk("train.dataset")

    
if Path("validation.dataset").exists():
    validation_dataset = datasets.Dataset.load_from_disk("validation.dataset")
else:
    validation_dataset = full_dataset["validation"].map(
        partial(
            tokenize_and_align_labels,
            label2id=label2id,
            return_word_ids=True
        ),
        batched=True,
        num_proc=args.dataloader_num_workers,
            remove_columns=["fold"]
        )

    validation_dataset.save_to_disk("validation.dataset")
    
# bonus points if you can explain why it says
# Ignored unknown kwarg option direction
model_config = AutoConfig.from_pretrained(
    CFG.model_name,
    num_labels=len(label_list),
    label2id=label2id,
    id2label=id2label,
    finetuning_task="ner",
)

model = AutoModelForTokenClassification.from_pretrained(CFG.model_name, config=model_config)<jupyter_output><empty_output><jupyter_text>OptimizerHere is the key cell where the 8-bit Adam optimizer gets set. It's pretty much trivially easy...<jupyter_code>no_decay = ["bias", "LayerNorm.weight"]
optimizer_grouped_parameters = [
    {
        "params": [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)],
        "weight_decay": args.weight_decay,
    },
    {
        "params": [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)],
        "weight_decay": 0.0,
    },
]

# These are the only changes you need to make
# The first part sets the optimizer to use 8-bits
# The for loop sets embeddings to use 32-bits
if CFG.adam_bits == 32:
    optimizer = bnb.optim.Adam32bit(optimizer_grouped_parameters, lr=args.learning_rate)
if CFG.adam_bits == 8:
    optimizer = bnb.optim.Adam8bit(optimizer_grouped_parameters, lr=args.learning_rate)
    
    # Thank you @gregorlied https://www.kaggle.com/nbroad/8-bit-adam-optimization/comments#1661976
    for module in model.modules():
        if isinstance(module, torch.nn.Embedding):
            bnb.optim.GlobalOptimManager.get_instance().register_module_override(
                module, 'weight', {'optim_bits': 32}
            )            

num_update_steps_per_epoch = len(train_dataset) // args.per_device_train_batch_size // args.gradient_accumulation_steps
if args.max_steps == -1 or args.max_steps is None:
    args.max_steps = args.num_train_epochs * num_update_steps_per_epoch
else:
    args.num_train_epochs = math.ceil(args.max_steps / num_update_steps_per_epoch)
    
if args.warmup_ratio is not None:
    args.num_warmup_steps = int(args.warmup_ratio * args.max_steps)

lr_scheduler = get_scheduler(
    name=args.lr_scheduler_type,
    optimizer=optimizer,
    num_warmup_steps=args.num_warmup_steps,
    num_training_steps=args.max_steps,
)
optimizer<jupyter_output><empty_output><jupyter_text>Data Collator and Metrics 📏 This collator is really handy because I can tell it to pad to a multiple of a number. Longformer likes to have inputs in multiples of 512, so it will handle the padding for me!<jupyter_code># Data collator
# 512 for longformer
# 1024 for bigbird
# 8 for short models
if "longformer" in CFG.model_name:
    pad_to_multiple_of = 512 
elif "bigbird" in CFG.model_name:
    pad_to_multiple_of = 1024
else:
    pad_to_multiple_of = 8
    
data_collator = DataCollatorForTokenClassification(tokenizer, pad_to_multiple_of=pad_to_multiple_of)

# Metrics
metric = datasets.load_metric("seqeval")

def compute_metrics(p):
    predictions, labels = p
    predictions = np.argmax(predictions, axis=2)

    # Remove ignored index (special tokens)
    true_predictions = [
        [label_list[p] for (p, l) in zip(prediction, label) if l != -100]
        for prediction, label in zip(predictions, labels)
    ]
    true_labels = [
        [label_list[l] for (p, l) in zip(prediction, label) if l != -100]
        for prediction, label in zip(predictions, labels)
    ]

    results = metric.compute(predictions=true_predictions, references=true_labels)
    
    # This f1 is not the same as the 
    return {
        "token_precision": results["overall_precision"],
        "token_recall": results["overall_recall"],
        "token_f1": results["overall_f1"],
        "token_accuracy": results["overall_accuracy"],
    }<jupyter_output><empty_output><jupyter_text>CV calculation functions<jupyter_code># <NAME> @robikscube
# https://www.kaggle.com/robikscube/student-writing-competition-twitch
def calc_overlap(pred, ground_truth):
    """
    Calculates the overlap between prediction and
    ground truth and overlap percentages used for determining
    true positives.
    """
    set_pred = set(pred.split(' '))
    set_gt = set(ground_truth.split(' '))
    # Length of each and intersection
    len_gt = len(set_gt)
    len_pred = len(set_pred)
    inter = len(set_gt.intersection(set_pred))
    overlap_1 = inter / len_gt
    overlap_2 = inter/ len_pred
    return [overlap_1, overlap_2]


def score_feedback_comp(pred_df, gt_df):
    """
    A function that scores for the kaggle
        Student Writing Competition
        
    Uses the steps in the evaluation page here:
        https://www.kaggle.com/c/feedback-prize-2021/overview/evaluation
    """
    gt_df = gt_df[['id','discourse_type','predictionstring']] \
        .reset_index(drop=True).copy()
    pred_df = pred_df[['id','discourse_type','predictionstring']] \
        .reset_index(drop=True).copy()
    pred_df['pred_id'] = pred_df.index
    gt_df['gt_id'] = gt_df.index
    # Step 1. all ground truths and predictions for a given discourse_type are compared.
    joined = pred_df.merge(gt_df,
                           left_on=['id','discourse_type'],
                           right_on=['id','discourse_type'],
                           how='outer',
                           suffixes=('_pred','_gt')
                          )
    joined['predictionstring_gt'] = joined['predictionstring_gt'].fillna(' ')
    joined['predictionstring_pred'] = joined['predictionstring_pred'].fillna(' ')

    joined['overlaps'] = [calc_overlap(pred, gt) for pred, gt in joined[['predictionstring_pred', 'predictionstring_gt']].values]

    # 2. If the overlap between the ground truth and prediction is >= 0.5, 
    # and the overlap between the prediction and the ground truth >= 0.5,
    # the prediction is a match and considered a true positive.
    # If multiple matches exist, the match with the highest pair of overlaps is taken.
    joined['overlap1'] = joined['overlaps'].apply(lambda x: eval(str(x))[0])
    joined['overlap2'] = joined['overlaps'].apply(lambda x: eval(str(x))[1])


    joined['potential_TP'] = (joined['overlap1'] >= 0.5) & (joined['overlap2'] >= 0.5)
    joined['max_overlap'] = joined[['overlap1','overlap2']].max(axis=1)
    tp_pred_ids = joined.query('potential_TP') \
        .sort_values('max_overlap', ascending=False) \
        .groupby(['id','predictionstring_gt']).first()['pred_id'].values

    # 3. Any unmatched ground truths are false negatives
    # and any unmatched predictions are false positives.
    fp_pred_ids = [p for p in joined['pred_id'].unique() if p not in tp_pred_ids]

    matched_gt_ids = joined.query('potential_TP')['gt_id'].unique()
    unmatched_gt_ids = [c for c in joined['gt_id'].unique() if c not in matched_gt_ids]

    # Get numbers of each type
    TP = len(tp_pred_ids)
    FP = len(fp_pred_ids)
    FN = len(unmatched_gt_ids)
    #calc microf1
    denominator = (TP + 0.5*(FP+FN))
    if denominator == 0:
        return 0.0
    my_f1_score = TP / denominator
    return {
        "F1": round(my_f1_score, 4),
        "Precision": TP/(TP+FP),
        "Recall": TP/(TP+FN), # This was calculated incorrectly in the runs
    }
        

id2label={i: l for l, i in label2id.items()}
# https://www.kaggle.com/zzy990106/pytorch-ner-infer?scriptVersionId=82677278&cellId=13
def get_label_predictions(dataset, preds):

    ids = dataset["id"]
    word_ids = dataset[CFG.word_id_column]
    words = dataset[CFG.text_column]
    
    all_preds = []

    for id_, sample_preds, sample_word_ids, words in zip(ids, preds, word_ids, words):
        label_preds = [""]*len(words)

        for pred, w_id in zip(sample_preds, sample_word_ids):
            if w_id is None:
                continue
            if label_preds[w_id] == "":
                label_preds[w_id] = id2label[pred]

        j = 0
        while j < len(label_preds):
            label = label_preds[j]

            if label.startswith("B"):
                label = label.replace("B", "I")
                end = j + 1
                while end < len(label_preds) and label_preds[end] == label:
                    end += 1

                if end - j > 7:
                    all_preds.append((id_, label.lstrip("BI-"), ' '.join(map(str, list(range(j, end))))))

                j = end
            else:
                j += 1
                
    return all_preds<jupyter_output><empty_output><jupyter_text>Create custom trainer<jupyter_code>class FeedbackPrizeTrainer(Trainer):
    
    def __init__(self, *args, **kwargs):
        if "eval_dataset" in kwargs:
            # The Trainer hides some columns that are necessary for CV
            # This code makes those columns accessible
            dataset_type = kwargs["eval_dataset"].format["type"]
            dataset_columns = list(kwargs["eval_dataset"].features.keys())
            self.cv_dataset = kwargs["eval_dataset"].with_format(type=dataset_type, columns=dataset_columns)
        super().__init__(*args, **kwargs)
        
        
    def evaluation_loop(
        self, 
        dataloader,
        description,
        prediction_loss_only = None,
        ignore_keys = None,
        metric_key_prefix = "eval",
    ):
        
        eval_output =  super().evaluation_loop(
            dataloader,
            description,
            prediction_loss_only,
            ignore_keys,
            metric_key_prefix
        )
        
        # Custom CV F1 calculation
        # This same loop gets called during predict, and we can't do CV when predicting
        if metric_key_prefix == "eval":
            
            eval_id_preds = eval_output.predictions.argmax(-1)
            eval_label_preds = get_label_predictions(self.cv_dataset, eval_id_preds)
            
            eval_pred_df = pd.DataFrame(eval_label_preds, columns=["id", "discourse_type", "predictionstring"])
            
            eval_gt_df = train_df[train_df["id"].isin(self.cv_dataset["id"])].reset_index(drop=True).copy()
            
            classes = ['Lead', 'Position', 'Evidence', 'Claim', 'Concluding Statement', 'Counterclaim', 'Rebuttal']
            f1_scores = []
            for class_ in classes:
                gt_df = eval_gt_df.loc[eval_gt_df['discourse_type'] == class_].copy()
                pred_df = eval_pred_df.loc[eval_pred_df['discourse_type'] == class_].copy()
                eval_scores = score_feedback_comp(pred_df, gt_df)
                for score_name, score in eval_scores.items():
                    eval_output.metrics[f"{metric_key_prefix}_{class_}_CV_{score_name}"] = score
                f1_scores.append(eval_scores["F1"])
                
            eval_output.metrics[f"{metric_key_prefix}_Overall_CV_F1"] = np.mean(f1_scores)
        
        return eval_output

# Initialize our Trainer
trainer = FeedbackPrizeTrainer(
    model=model,
    args=args,
    train_dataset=train_dataset,
    eval_dataset=validation_dataset,
    tokenizer=tokenizer,
    data_collator=data_collator,
    compute_metrics=compute_metrics,
    optimizers=(optimizer, lr_scheduler),
    # callbacks = [EarlyStoppingCallback(early_stopping_patience=3)]
)
args, CFG.fold, CFG.model_name, CFG.max_seq_length<jupyter_output><empty_output><jupyter_text>Train! 🚆<jupyter_code>%env TOKENIZERS_PARALLELISM=true
from torch.utils.checkpoint import checkpoint

if args.do_train:
  train_result = trainer.train()
  metrics = train_result.metrics
  trainer.save_model()  # Saves the tokenizer too

  metrics["train_samples"] = len(train_dataset)

  trainer.log_metrics("train", metrics)
  trainer.save_metrics("train", metrics)
  trainer.save_state()
else:
  train_result = trainer.evaluate()
  print(train_result)<jupyter_output>env: TOKENIZERS_PARALLELISM=true<jupyter_text>Weights and Biases Report ✨ Update v25: A more thorough testing of 8-bit vs 32-bit Adam was carried out. [Report available here](https://wandb.ai/nbroad/feedback-prize/reports/8-bit-Adam-vs-32-bit-Adam-tests--VmlldzoxNDgwNTc1) tldr: 8bit Adam is slower but saves memory and results in a higher CV. (v1 and v3): You can see the lower memory usage and marginally faster training time. The loss curves are nearly identical and CV F1 scores are pretty much the same as well.<jupyter_code># model.save_pretrained('./output/hf_model')
# tokenizer.save_pretrained('./output/hf_model')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10528604884576781, 'fraction_numerical': 0.02392104382736701, 'mean_word_length': 3.4022091310751104, 'pattern_counts': {'":': 14, '<': 51, '<?xml version=': 0, '>': 60, 'https://': 18, 'lorem ipsum': 0, 'www.': 7, 'xml': 0}, 'pii_count': 8, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '18743', 'license': 'MIT', 'n_tokens_mistral': 10287, 'n_tokens_neox': 9644, 'n_words': 2312, 'path': '8_bit_adam_optimization_deberta_large_fold_2_10.ipynb', 'repo_name': 'bobbercheng/Advanced-Machine-Learning-Specialization', 'size': '29919'}