<jupyter_start><jupyter_text>NLP with 🤗 Hugging Face Text Classification: Transfer Learning vs Zero-Shot Classifier🤗 [Hugging Face](https://huggingface.co/) is, in my opinion, one of the best things that has happened to Data Science over the past few years. From generalizing access to state-of-the-art NLP models with the [`transformers`](https://huggingface.co/transformers/) library to [distillation [1]](https://arxiv.org/abs/1910.01108), they are having a huge impact on the field. I recently found out about "Zero-Shot Classification". These models are classifiers that do not need any fine-tuning, apart from being told which classes it should predict. They are built on top of Natural Language Inference models, whose task is determining if sentence *A* implies, contradicts or has nothing to do with sentence *B*. This excellent [blog post](https://joeddav.github.io/blog/2020/05/29/ZSL.html) written by 🤗 Hugging Face researcher <NAME> provides more in-depth explanations.    Here is an example:<jupyter_code># transformers 3.5.1 in this notebook
from transformers import pipeline

# By default, the pipeline runs on the CPU (device=-1). Set device to 0 to use the GPU (and to 1 for the second GPU, etc...)
classifier = pipeline("zero-shot-classification", device=0)
classifier(
    "Parallel processing with GPUs is the savior of Deep Learning",
    candidate_labels=["education", "politics", "technology"],
)<jupyter_output>Some weights of the model checkpoint at facebook/bart-large-mnli were not used when initializing BartModel: ['model.encoder.version', 'model.decoder.version']
- This IS expected if you are initializing BartModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BartModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of the model checkpoint at facebook/bart-large-mnli were not used when initializing BartForSequenceClassification: ['model.encoder.version', 'model.decoder.version']
- This IS expected if you are initializing BartForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification m[...]<jupyter_text>The classifier guessed that the sentence is about tech with a probability over 99%. **But how does Zero-Shot classification compare with plain "old" fine-tuned text classification?** I) BBC News datasetLet's build a classifier of news articles labeled *business*, *entertainment*, *politics*, *sport* and *tech*. Available [here](http://mlg.ucd.ie/datasets/bbc.html), the dataset consists of 2225 documents from the BBC news website from the years 2004/2005. It was originally built for a Machine Learning paper about clustering [[2]](http://mlg.ucd.ie/files/publications/greene06icml.pdf).    Articles are individual .txt files spread into 5 folders, one for each folder. The listing below puts articles/labels into a `pandas.DataFrame()`.<jupyter_code># Utilities to handle directories, files, paths, etc...
from os import listdir
from os.path import isdir, isfile, join
from pathlib import Path

# Most original import ever
import pandas as pd

path_to_bbc_articles="bbc"
labels=[] # labels for the text classification
label_dataframes=[] # for each label, get the articles into a dataframe

for label in [dir for dir in listdir(path_to_bbc_articles) if isdir(join(path_to_bbc_articles, dir)) and dir!=".ipynb_checkpoints"]:
    labels.append(label)
    label_path=join(path_to_bbc_articles, label)
    articles_list=[]
    for article_file in [file for file in listdir(label_path) if isfile(join(label_path, file))]:
        article_path=join(label_path, article_file)
        article=Path(article_path).read_text(encoding="ISO-8859-1") # Tried utf-8 (of course) but encountered error
        # Stackoverflow said "try ISO-8859-1", it worked (dataset is 11 years old)
        articles_list.append(article)
    label_dataframes.append(pd.DataFrame({'label': label, 'article': articles_list}))
    
df=pd.concat(label_dataframes, ignore_index=True) # Concatenate all the dataframes
# Number of articles per label
df.value_counts('label')<jupyter_output><empty_output><jupyter_text>We will need integer labels to feed the transformer model:<jupyter_code>df['label_int']=df['label'].apply(lambda x:labels.index(x))<jupyter_output><empty_output><jupyter_text>Here are 5 random rows from the final dataframe:<jupyter_code>df.sample(5, random_state=42)<jupyter_output><empty_output><jupyter_text>II) Fine-tuning a pretrained text classifierAfter building the train/validation/test sets, we will go straight the point by using the [`DistilBERT`](https://huggingface.co/transformers/model_doc/distilbert.html) pre-trained transformer model (and its tokenizer).> *It is a small, fast, cheap and light Transformer model trained by distilling BERT base. It has 40% less parameters than bert-base-uncased, runs 60% faster while preserving over 95% of BERT’s performances.*<jupyter_code># Train set, validation set and test set
from sklearn.model_selection import train_test_split

train_val, test = train_test_split(df, test_size=0.1, random_state=42, shuffle=True)
train, val = train_test_split(train_val, test_size=0.2, random_state=42, shuffle=True)

# Reset the indexes of the 3 pandas.DataFrame()
train, val, test = map(lambda x:x.reset_index(drop=True), [train, val, test])<jupyter_output><empty_output><jupyter_text>TokenizeLoading DistilBERT's tokenizer, we can see that this transformer model takes input sequences composed of up to 512 tokens:<jupyter_code># Load Distilbert's tokenizer
from transformers import DistilBertTokenizerFast
tokenizer = DistilBertTokenizerFast.from_pretrained('distilbert-base-uncased')
tokenizer.max_model_input_sizes<jupyter_output><empty_output><jupyter_text>How does this compare with the lengths of the tokenized BBC articles?<jupyter_code>tokenized_articles_lengths=pd.DataFrame({'length': list(map(len, tokenizer(df['article'].to_list(), truncation=False, padding=False)['input_ids']))})
tokenized_articles_lengths.describe()<jupyter_output><empty_output><jupyter_text>The articles are, on average, 488-token-long. The longest news is composed of 5303 tokens. This means that an important part of the articles will be truncated before being fed to the transformer model. Here is the distribution of the lengths:<jupyter_code>import matplotlib.pyplot as plt
import seaborn as sns

ax=sns.histplot(tokenized_articles_lengths)
ax.set(xlabel='Length of tokenized articles', ylabel='Count', xlim=(0, 1200), title='Distribution of the tokenized articles lengths')
plt.show()

# As shown on datatrigger.org/nlp_hugging_face:

# fig, ax = plt.subplots(figsize=(16, 16))
# ax=sns.histplot(tokenized_articles_lengths, palette='dark')
# ax.set(xlim=(0, 1200))
# ax.set_xticks(range(0, 1200, 100))
# ax.set_title('Distribution of the tokenized articles lengths', fontsize=24, pad=20)
# ax.set_xlabel('Length of tokenized articles', fontsize = 18, labelpad = 10)
# ax.set_ylabel('Count', fontsize = 18, labelpad = 10)
# ax.tick_params(labelsize=14)
# plt.savefig('tokenized_articles_length_distribution.png', bbox_inches='tight');
from scipy.stats import percentileofscore
print(f'Percentile of length=512: {int(percentileofscore(tokenized_articles_lengths["length"],512))}th')<jupyter_output>Percentile of length=512: 64th<jupyter_text>About 36% of the articles will be truncated to fit the 512-token limit of DistilBERT. The truncation is mandatory, otherwise the model crashes. We will use fixed padding for the sake of simplicity here.  Fine-tune DistilBERTThe train/validation/test sets must be procesμsed to work with either PyTorch or TensorFlow.<jupyter_code># Format the train/validation/test sets
train_encodings = tokenizer(train['article'].to_list(), truncation=True, padding=True)
val_encodings = tokenizer(val['article'].to_list(), truncation=True, padding=True)
test_encodings = tokenizer(test['article'].to_list(), truncation=True, padding=True)

import torch

class BBC_Dataset(torch.utils.data.Dataset):
    def __init__(self, encodings, labels):
        self.encodings = encodings
        self.labels = labels

    def __getitem__(self, idx):
        item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
        item['labels'] = torch.tensor(self.labels[idx])
        return item

    def __len__(self):
        return len(self.labels)

train_dataset = BBC_Dataset(train_encodings, train['label_int'].to_list())
val_dataset = BBC_Dataset(val_encodings, val['label_int'].to_list())
test_dataset = BBC_Dataset(test_encodings, test['label_int'].to_list())
# Fine-tuning
from transformers import DistilBertForSequenceClassification, Trainer, TrainingArguments

training_args = TrainingArguments(
    output_dir='./results',
    num_train_epochs=3,
    per_device_train_batch_size=8,
    per_device_eval_batch_size=4,
    weight_decay=0.01,
)

# The number of predicted labels must be specified with num_labels
# .to('cuda') to do the training on the GPU
model = DistilBertForSequenceClassification.from_pretrained("distilbert-base-uncased", num_labels=len(labels)).to('cuda')

trainer = Trainer(
    model=model,                         # the instantiated 🤗 Transformers model to be trained
    args=training_args,                  # training arguments, defined above
    train_dataset=train_dataset,         # training dataset
    eval_dataset=val_dataset             # evaluation dataset
)
trainer.train()
trainer.save_model("bbc_news_model")
# Generate predictions for the test set
predictions=trainer.predict(test_dataset)<jupyter_output><empty_output><jupyter_text>Accuracy<jupyter_code>test_results=test.copy(deep=True)
test_results["label_int_pred_transfer_learning"]=predictions.label_ids
test_results['label_pred_transfer_learning']=test_results['label_int_pred_transfer_learning'].apply(lambda x:labels[x])
test_results[test_results["label"]!=test_results["label_pred_transfer_learning"]].head()<jupyter_output><empty_output><jupyter_text>The accuracy of the fine-tuned DistilBERT transformer model on the test set is **100%**. III) Zero-Shot ClassificationWe'll use the appropriate [`transformers.pipeline`](https://huggingface.co/transformers/main_classes/pipelines.html) to compute the predicted class for each article.<jupyter_code>from transformers import pipeline
classifier = pipeline("zero-shot-classification", device=0) # device=0 means GPU
# Compute the predicted label for each article
test_results['label_pred_zero_shot']=test_results['article'].apply(lambda x:classifier(x, candidate_labels=labels)['labels'][0])
# Reorder columns and save results
test_results=test_results[['article', 'label', 'label_pred_transfer_learning', 'label_pred_zero_shot']]
test_results.to_parquet("test_results.parquet")
test_results=pd.read_parquet("test_results.parquet")
error_rate=len(test_results[test_results["label"]!=test_results["label_pred_zero_shot"]])/len(test_results)
print(f'Accuracy of the Zero-Shot classifier: {round(100*(1-error_rate), 2)} %')<jupyter_output>Accuracy of the Zero-Shot classifier: 55.16 %<jupyter_text>The Zero-Shot classifier does a really bad job compared with the fine-tuned model. However, given the number of labels &mdash; 5 &mdash; this result is not that catastrophic. It is well above the 20% a random classifier would achieve (assuming balanced classes). Let's have a look at a few random articles uncorrectly labeled by the Zero-Shot classifier:<jupyter_code>test_results[test_results["label"]!=test_results["label_pred_zero_shot"]].sample(15, random_state=42)<jupyter_output><empty_output><jupyter_text>There does not seem to be a particularly problematic class, although such a assertion would require further investigation. But the length of the news could lead to poor performance. We can read about this on the [🤗 Hugging Face forum](https://discuss.huggingface.co/t/new-pipeline-for-zero-shot-text-classification/681/85). <NAME>, 🤗 Hugging Face developer and creator of the Zero-Shot pipeline, says the following:> *For long documents, I don’t think there’s an ideal solution right now. If truncation isn’t satisfactory, then the best thing you can do is probably split the document into smaller segments and ensemble the scores somehow.*We'll try another solution: summarizing the article first, then Zero-Shot classifying it. IV) Summarization + Zero-Shot Classification The easiest way to do this would have been to line up the `SummarizationPipeline` with the `ZeroShotClassificationPipeline`. This is not possible, at least with my version of the `transformers` library (3.5.1). The reason for this is that the `SummarizationPipeline` uses Facebook's BART model, whose maximal input length is 1024 tokens. However, `transformers`'s tokenizers, including `BartTokenizer`, do not automatically truncate sequences to the max input length of the corresponding model. As a consequence, the `SummarizationPipeline` crashes whenever sequences longer than 1024 tokens are given as inputs. Since there are quite a few long articles in the BBC dataset, we will have to make a custom summarization pipeline that truncates news longers than 1024 tokens.<jupyter_code># Import the tokenizer and model for summarization (the same that are used by default in Hugging Face's summarization pipeline)
from transformers import BartTokenizer, BartForConditionalGeneration, BartConfig

model_bart = BartForConditionalGeneration.from_pretrained('facebook/bart-large-cnn').to('cuda') # Run on the GPU
tokenizer_bart = BartTokenizer.from_pretrained('facebook/bart-large-cnn')
# Custom summarization pipeline (to handle long articles)
def summarize(text):
    # Tokenize and truncate
    inputs = tokenizer_bart([text], truncation=True, max_length=1024, return_tensors='pt').to('cuda')
    # Generate summary between 10 (by default) and 50 characters
    summary_ids = model_bart.generate(inputs['input_ids'], num_beams=4, max_length=50, early_stopping=True)
    # Untokenize
    return([tokenizer_bart.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=False) for g in summary_ids][0])
# Apply summarization then zero-shot classification to the test set
test_results['label_pred_sum_zs']=test_results['article'].apply(lambda x:classifier(summarize(x), candidate_labels=labels)['labels'][0])
test_results.to_parquet("test_results.parquet")
error_rate_sum_zs=len(test_results[test_results["label"]!=test_results["label_pred_sum_zs"]])/len(test_results)
print(f'Accuracy of the Summmarization+Zero-Shot classifier pipeline: {round(100*(1-error_rate_sum_zs), 2)} %')<jupyter_output>Accuracy of the Summmarization+Zero-Shot classifier pipeline: 78.03 %
{'fraction_non_alphanumeric': 0.09421310266210689, 'fraction_numerical': 0.01388050694025347, 'mean_word_length': 6.339566929133858, 'pattern_counts': {'":': 0, '<': 51, '<?xml version=': 0, '>': 53, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '94647', 'license': 'MIT', 'n_tokens_mistral': 4573, 'n_tokens_neox': 4306, 'n_words': 1394, 'path': 'bbc_news.ipynb', 'repo_name': 'datatrigger/nlp_hugging_face', 'size': '14923'}