<jupyter_start><jupyter_text>Finetune a DistilBERT model on the AG News DatasetIn this notebook, we show how to finetune a pretrained DistilBERT model from the recent v4.x release of the Hugging Face 🤗 transformers library.Modified from the [example finetuning notebook in the Hugging Face docs.](https://github.com/huggingface/notebooks/blob/master/examples/text_classification.ipynb)<jupyter_code>!pip install transformers==4.5.1
!pip install datasets==1.6.0
!pip install pandas==1.1.5
from sklearn.model_selection import train_test_split
from sklearn.metrics import (
    accuracy_score,
    precision_recall_fscore_support,
)
import numpy as np
import pandas as pd
from datasets import load_dataset
data = load_dataset(
    'ag_news',
    split={
        'train': 'train[:90%]',
        'valid': 'train[90%:]',
        'test': 'test[:100%]',
    },
)
print(
    f"Size of training set: {len(data['train'])}\n",
    f"Size of validation set: {len(data['valid'])}\n",
    f"Size of test set: {len(data['test'])}\n",
)
unique_labels = set(data['test']['label'])
num_labels = len(unique_labels)
print(f"Found {num_labels} unique labels in the test set.")<jupyter_output>Found 4 unique labels in the test set.<jupyter_text>Compute metrics<jupyter_code>def compute_metrics(pred):
    labels = pred.label_ids
    preds = pred.predictions.argmax(-1)
    precision, recall, f1, _ = precision_recall_fscore_support(labels, preds, average='macro')
    acc = accuracy_score(labels, preds)
    return {
        'accuracy': acc,
        'f1': f1,
        'precision': precision,
        'recall': recall
    }<jupyter_output><empty_output><jupyter_text>Train model via 🤗 transformers APIWe define the model and training loop as per the recent v4.x Hugging Face API.<jupyter_code>model_checkpoint = "distilbert-base-uncased"
batch_size = 32
data['train'][1345]['text']
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    model_checkpoint,
    use_fast=True,
    max_length=512,
    padding=True,
    truncation=True,
    return_tensors="pt",
)

# Because we are using the "uncased" model checkpoint, we don't care about capitalization
# Below, we check that the input IDs generated are the same regardless of capitalization of tokens
# Note that in the BERT tokenizer, 101 represents the class [CLS] token and 102 represents the separator [SEP] token
tokenizer("Korea korea Korea China china CHINA")<jupyter_output><empty_output><jupyter_text>We pass the `use_fast=True` argument to the tokenizer instance to use the fast tokenizer (implemented in Rust) from the 🤗 Tokenizers library.The `truncation=True` argument ensures that an input longer that what the model selected can handle will be truncated to the maximum length accepted by the model (for BERT/DistilBERT this is 512 tokens by default).  Load pretrained modelBelow, we load a pretrained DistilBERT model from the checkpoint.<jupyter_code>from transformers import AutoModelForSequenceClassification, TrainingArguments, Trainer

model = AutoModelForSequenceClassification.from_pretrained(model_checkpoint, num_labels=num_labels)
args = TrainingArguments(
    "distilbert-finetuning",
    evaluation_strategy="epoch",
    learning_rate=3e-5,
    per_device_train_batch_size=32,
    per_device_eval_batch_size=32,
    num_train_epochs=3,
    warmup_ratio=0.25,  # Slanted triangular learning rate ramp up/down (0.2-0.3 gave best results on multiple runs)
    weight_decay=0.01,  # L2 regularization
    load_best_model_at_end=True,
    metric_for_best_model="accuracy",
)
def get_encoded_data(dataset, textcol="text"):
    """Obtain preprocessed data 
    """
    def tokenize(batch):
        tokens = tokenizer(batch[textcol], truncation=True, padding=True)
        return tokens

    encoded_dataset = (dataset
    .map(
        tokenize,
        batched=True,
        # load_from_cache_file=False
        )
    )
    encoded_dataset.set_format('torch', columns=['input_ids', 'attention_mask', 'label'])
    return encoded_dataset
encoded_dataset = get_encoded_data(data, textcol="text")
encoded_dataset
trainer = Trainer(
    model,
    args=args,
    train_dataset=encoded_dataset['train'],
    eval_dataset=encoded_dataset['valid'],
    tokenizer=tokenizer,
    compute_metrics=compute_metrics,
)
trainer.train()
trainer.evaluate()
from google.colab import drive
drive.mount('/content/drive/')
%cd '/content/drive/My Drive/Colab Notebooks/'
# Save model
model_name = "model_agnews"
trainer.save_model(model_name)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09891808346213292, 'fraction_numerical': 0.013027158313093399, 'mean_word_length': 4.860284605433376, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '526077', 'license': 'MIT', 'n_tokens_mistral': 1466, 'n_tokens_neox': 1406, 'n_words': 385, 'path': 'classification/DistilBERT_AGNews_classifier.ipynb', 'repo_name': 'prrao87/transformers-finetuning', 'size': '4529'}