<jupyter_start><jupyter_text>Evaluation of conditional generators This notebooks serves as a tool to evaluate the conditioning quality of a text generator using a text classifier.It assumes the generator is able to use the first word of an input sequence as a label that conditions the characteristics of the output text.We construct the prompts fed to the generator in two different ways, which results in four different metrics, as we calculate the validation loss and accuracy for each procedure.**Procedure A**1. Generate text using just the labels plus a line break as prompt (one label per input sequence, as many sequences as labels).2. Delete the labels from the generated text.3. Feed the generated text to the corresponding classifier to evaluate the metrics.**Procedure A for encoder-decoder** If the generator follows an encoder-decoder architecture, the procedure is a bit different because we feed the labels to the encoder:1. Generate text using just the labels plus a line break as inputs to the encoder (one label per input sequence, as many sequences as labels) and no prompt for the decoder (start with an empty sequence, only bos token).2. Feed the generated text to the corresponding classifier to evaluate the metrics.**Procedure B**1. Generate text using as prompt an initial substring of each sequence in the validation set, with its label and a line break prepended. We are limiting the length of prompt to the minimum between 100 characters and 1/4 of the length of the text.2. Delete the labels from the generated text.3. Feed the generated text to the corresponding classifier to evaluate the metrics.The dataset/split used in this case is the same dataset that we use as validation set to train the text classifier (see [train_poems_classifier.ipynb](train_poems_classifier.ipynb))**Procedure B for encoder-decoder** If the generator follows an encoder-decoder architecture, the procedure is a bit different because we feed the labels to the encoder and the prompts to the decoder:1. Generate text using as prompts (inputs of the decoder) an initial substring of each sequence in the validation set, with its label and a line break as inputs of the encoder. We are limiting the length of prompt to the minimum between 100 characters and 1/4 of the length of the text.2. Feed the generated text to the corresponding classifier to evaluate the metrics. Set `run_as_standalone_nb = True` if you are running this notebook outside of a clone of its repository (https://github.com/Poems-AI/AI.git). For example, in a Colab or Kaggle notebook.<jupyter_code>run_as_standalone_nb = False


from pathlib import Path


if run_as_standalone_nb:
    import sys    
    root_lib_path = Path('AI').resolve()
    if not root_lib_path.exists():
        !git clone https://github.com/Poems-AI/AI.git
    if str(root_lib_path) not in sys.path:
        sys.path.insert(0, str(root_lib_path))
        
    !pip install -r {root_lib_path/'requirements.txt'}
    !apt-get install git-lfs
    !git lfs install
else:
    import local_lib_import
import os
import pandas as pd
from transformers import AutoModelForCausalLM, AutoModelForSequenceClassification, AutoTokenizer, EncoderDecoderModel
import torch
import torch.nn.functional as F
from poemsai.config import set_config_value
from poemsai.data import (build_labeled_dfs_from_splits, label_type_to_str, LabelsType, LabelsWriterStd, 
                          LabelsWriterExplained, PoemsFileConfig)
from poemsai.metrics import ConditionalGenEvaluator
from poemsai.nb_utils import download_checkpoint_from_hf_hub<jupyter_output><empty_output><jupyter_text>Clone our datasets repo:<jupyter_code>!git clone https://github.com/Poems-AI/dataset.git<jupyter_output><empty_output><jupyter_text>Choose the type of labels the generator must be conditioned on:<jupyter_code>labels_type = LabelsType.Topics<jupyter_output><empty_output><jupyter_text>Set `gen_model_name` to the checkpoint of the conditional generator you wish to evaluate:<jupyter_code>HF_USER = 'YOUR_HF_USER'
gen_model_name = 'gpt2-poems-endtags.en'
cat_name = label_type_to_str(labels_type)
clf_model_name = f'distilbert-poems-clf-by-{cat_name}'
hf_pwd = '<PASSWORD>'
download_checkpoint_from_hf_hub(gen_model_name, HF_USER, hf_pwd)
download_checkpoint_from_hf_hub(clf_model_name, HF_USER, hf_pwd)
hf_pwd = None
gen_model = AutoModelForCausalLM.from_pretrained(gen_model_name)
#gen_model = EncoderDecoderModel.from_pretrained(gen_model_name)
clf_model = AutoModelForSequenceClassification.from_pretrained(clf_model_name)
# When `gen_model` is an encoder-decoder, `gen_tokenizer` is the tokenizer of the decoder
gen_tokenizer = AutoTokenizer.from_pretrained(gen_model_name)
clf_tokenizer = AutoTokenizer.from_pretrained(clf_model_name)<jupyter_output><empty_output><jupyter_text>If the text generator `gen_model` is an encoder-decoder and the encoder and decoder have different tokenizers, execute the cell below to load tokenizer of the encoder. Don't forget to set `encoder_checkpoint` to the checkpoint of the pretrained model from which you loaded the encoder.<jupyter_code>decoder_tokenizer = None
if gen_model.config.is_encoder_decoder:
    decoder_tokenizer = gen_tokenizer
    encoder_checkpoint = "distilbert-base-uncased"
    gen_tokenizer = AutoTokenizer.from_pretrained(encoder_checkpoint)<jupyter_output><empty_output><jupyter_text>You must use the same file config and the same `BaseLabelsWriter` subclass that you used to generate the text file that was used to train `gen_model_name`.This way, the prompts fed to the generator will have the same format as the training dataset.<jupyter_code>file_conf = PoemsFileConfig.from_json('dataset/all.txt/en.txt/only_end_tags/all_poems.en.conf.json')
all_cats_ordered = [label_type_to_str(cat) for cat in LabelsType if cat != LabelsType.All]
evaluator = ConditionalGenEvaluator(gen_model, gen_tokenizer, clf_model, clf_tokenizer, file_conf,
                                    cat_name, all_cats_ordered, labels_writer=LabelsWriterStd(),
                                    gen_decoder_tokenizer=decoder_tokenizer)<jupyter_output><empty_output><jupyter_text>Metrics AThese metrics are referred to as "*[label_type] conditional loss A*" and "*[label_type] conditional accuracy A*" in the [results doc](../docs/results.md), with "[label_type]" being one of {"topic", "form"}<jupyter_code>evaluator.eval_with_labels_as_prompt()<jupyter_output><empty_output><jupyter_text>Metrics BThese metrics are referred to as "*[label_type] conditional loss B*" and "*[label_type] conditional accuracy B*" in the [results doc](../docs/results.md), with "[label_type]" being one of {"topic", "form"} If outside of Kaggle, you should point `KAGGLE_DS_ROOT` to the root folder that contains the poems datasetby Kaggle user michaelarman (https://www.kaggle.com/michaelarman/poemsdataset)<jupyter_code>set_config_value('KAGGLE_DS_ROOT', '/kaggle/input')
splits_df_path = 'dataset/all.txt/en.txt/only_end_tags/all_poems.en.splits.csv'
splits_df = pd.read_csv(splits_df_path, index_col=0)
_, valid_df = build_labeled_dfs_from_splits(splits_df, labels_type)
evaluator.eval_with_seq_fragment_as_prompt(valid_df)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06717663421418638, 'fraction_numerical': 0.003198887343532684, 'mean_word_length': 5.658333333333333, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 34, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '638110', 'license': 'Apache-2.0', 'n_tokens_mistral': 2158, 'n_tokens_neox': 2059, 'n_words': 738, 'path': 'notebooks/evaluate_conditional_generation.ipynb', 'repo_name': 'Poems-AI/AI', 'size': '7196'}