<jupyter_start><jupyter_text>🏷️ 🔫 Faster data annotation with a zero-shot text classifier  TL;DR1. A simple example for data annotation with Rubrix is shown: **using a zero-shot classification model to pre-annotate and hand-label data more efficiently**.2. We use the new **SELECTRA zero-shot classifier** and the Spanish part of the **MLSum**, a multilingual dataset for text summarization.3. Two data annotation rounds are performed: (1) **labeling random examples**, and (2) **bulk labeling high score examples**.4. Besides boosting the labeling process, this workflow lets you **evaluate the performance of zero-shot classification for a specific use case**. In this example use case, we observe the pre-trained zero-shot classifier provides pretty decent results, which might be enough for general news categorization.   Why- The availability of pre-trained language models with zero-shot capabilities means you can, sometimes, accelerate your data annotation tasks by pre-annotating your corpus with a pre-trained zeroshot model.- The same workflow can be applied if there is a pre-trained "supervised" model that fits your categories but needs fine-tuning for your own use case. For example, fine-tuning a sentiment classifier for a very specific type of message.  Setup RubrixRubrix, is a free and open-source tool to explore, annotate, and monitor data for NLP projects.If you are new to Rubrix, check out the ⭐ [Github repository](https://github.com/recognai/rubrix).If you have not installed and launched Rubrix, check the [Setup and Installation guide](../getting_started/setup&installation.rst).Once installed, you only need to import Rubrix:<jupyter_code>import rubrix as rb<jupyter_output><empty_output><jupyter_text>Install dependenciesFor this tutorial we only need to install a few additional dependencies:<jupyter_code>%pip install transformers datasets torch -qqq<jupyter_output><empty_output><jupyter_text>1. Load the Spanish zero-shot classifier: `Selectra`We will use the recently released [**SELECTRA zero-shot classifier model**](https://huggingface.co/Recognai/zeroshot_selectra_medium), a zero-shot classifier for Spanish language.<jupyter_code>from transformers import pipeline

classifier = pipeline("zero-shot-classification", 
                       model="Recognai/zeroshot_selectra_medium")<jupyter_output><empty_output><jupyter_text>2. Loading the `MLSum` dataset[MLSUM](https://huggingface.co/datasets/mlsum), is a large scale multilingual text summarization dataset. Obtained from online newspapers, it contains 1.5M+ article/summary pairs in five different languages -- namely, French, German, Spanish, Russian and Turkish.To illustrate the labeling process, in this tutorial we will only use the first 500 examples of its Spanish test set.<jupyter_code>from datasets import load_dataset

mlsum = load_dataset("mlsum", "es", split="test[0:500]")<jupyter_output><empty_output><jupyter_text>3. Making zero-shot predictionsThe zero-shot classifier allows you to provide arbitrary candidate labels, which it will use for its predictions.Since under the hood, this zero-shot classifier is based on [natural language inference (NLI)](https://joeddav.github.io/blog/2020/05/29/ZSL.htmlClassification-as-Natural-Language-Inference), we need to convert the candidate labels into a "hypothesis".For this we use a *hypothesis_template*, in which the `{}` will be replaced by each one of our candidate label.This template can have a big effect on the scores of your predictions and should be adopted to your use case.<jupyter_code># We adopted the hypothesis to our use case of predicting the topic of news articles
hypothesis_template = "Esta noticia habla de {}."
# The candidate labels for our zero-shot classifier
candidate_labels = ["política", "cultura", "sociedad", "economia", "deportes", "ciencia y tecnología"]

# Make predictions batch-wise
def make_prediction(rows):
    predictions = classifier(
        rows["summary"], 
        candidate_labels=candidate_labels, 
        hypothesis_template=hypothesis_template
    )
    return {key: [pred[key] for pred in predictions] for key in predictions[0]}

mlsum_with_predictions = mlsum.map(make_prediction, batched=True, batch_size=8)<jupyter_output><empty_output><jupyter_text>4. Logging predictions in RubrixLet us log the examples to Rubrix and start our hand-labeling session, which will hopefully become more efficient with the zero-shot predictions.<jupyter_code>records = []

for row in mlsum_with_predictions:
    records.append(
        rb.TextClassificationRecord(
            inputs=row["summary"],
            prediction=list(zip(row['labels'], row['scores'])),
            prediction_agent="zeroshot_selectra_medium",
            metadata={"topic": row["topic"]}
        )
    )
rb.log(records, name="zeroshot_noticias", metadata={"tags": "data-annotation"})<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08311156601842375, 'fraction_numerical': 0.00593654042988741, 'mean_word_length': 5.280205655526992, 'pattern_counts': {'":': 2, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '72094', 'license': 'Apache-2.0', 'n_tokens_mistral': 1356, 'n_tokens_neox': 1294, 'n_words': 515, 'path': 'docs/tutorials/zeroshot_data_annotation.ipynb', 'repo_name': 'sugatoray/rubrix', 'size': '4885'}