<jupyter_start><jupyter_text>RoBERTA Fine-Tuning References:      - https://arxiv.org/abs/1907.11692      - https://huggingface.co/transformers/model_doc/roberta.html      - https://github.com/huggingface/transformers/pull/1275/files      - https://github.com/huggingface/transformers/tree/master/examples/token-classification      - https://www.kaggle.com/debanga/huggingface-tokenizers-cheat-sheet      - https://github.com/billpku/NLP_In_Action      - https://androidkt.com/name-entity-recognition-with-bert-in-tensorflow/      - https://github.com/smart-patrol/sagemaker-bert<jupyter_code>import pandas as pd
import math
import numpy as np
import sagemaker
import torch
import torch.nn.functional as F
import os
import json
from sagemaker.pytorch import PyTorch
sagemaker_session = sagemaker.Session()
role = sagemaker.get_execution_role()
bucket = sagemaker_session.default_bucket()
data_directories = json.load(open("utils/objects/data_directories_roberta.json", "r"))
display(data_directories)<jupyter_output><empty_output><jupyter_text>Training script Uncomment cell below to display training script<jupyter_code># ! pygmentize source_roberta/train_roberta.py<jupyter_output><empty_output><jupyter_text>Define hyperparameters Based on RoBERTa paper and Hugging Face documentation (both mentioned in references) I choosed following set of hyperparameters to fine-tune RoBERTa. BERT-family publications suggests to try larger batch-sizes, however, computation time was too big on batch-size 64 on current machine and I had to terminate it. Anyway, I finally extended training time for one more epoch and that resulted in improvement of F1 Score by almost 1 point (comparing to recommended 3 epochs).<jupyter_code>hyperparameters = {'epochs': 4,
                   'n_tags': 20,
                   'max_len': 45,
                   'batch-size': 32
                  }<jupyter_output><empty_output><jupyter_text>I used Adam optimizer with weight decay with parameters specified in fine-tuning script for token classification from Hugging Face.  Initialization of AdamW (snippet from training script):>```python    no_decay = ["bias", "LayerNorm.weight"]    optimizer_grouped_parameters = [        {"params": [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)],         "weight_decay": 0.01},        {"params": [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)],         "weight_decay": 0.0}    ]    optimizer = AdamW(optimizer_grouped_parameters, lr=5e-5, eps=1e-8)```   I also prevent exploding gradients by gradient clipping:>```python    torch.nn.utils.clip_grad_norm_(parameters=model.parameters(),                                   max_norm=1.0)```  Define estimator<jupyter_code>estimator = PyTorch(entry_point="train_roberta.py",
                    source_dir="source_roberta",
                    role=role,
                    framework_version='1.1.0',
                    train_instance_count=1,
                    train_instance_type='ml.p2.xlarge',
                    hyperparameters = hyperparameters
                   )<jupyter_output><empty_output><jupyter_text>Training on K80 instance:<jupyter_code>estimator.fit({'training': data_directories['train_data_directory'],
               'validation': data_directories['test_data_directory']})
print(estimator.model_data)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09893364928909952, 'fraction_numerical': 0.01273696682464455, 'mean_word_length': 3.6837725381414703, 'pattern_counts': {'":': 4, '<': 21, '<?xml version=': 0, '>': 23, 'https://': 8, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '630693', 'license': 'MIT', 'n_tokens_mistral': 1055, 'n_tokens_neox': 990, 'n_words': 235, 'path': 'src/3_train_roberta.ipynb', 'repo_name': 'blawok/Named_Entity_Recognition', 'size': '3376'}