Add blanc score module

README.md

@@ -1,12 +1,183 @@
----
-title: Blanc
-emoji: 🔥
-colorFrom: red
-colorTo: red
+from transformers.models.paligemma.convert_paligemma_weights_to_hf import device---
+title: BLANC
+datasets:
+-  
+tags:
+- evaluate
+- metric
+description: "BLANC is a reference-free metric that evaluates the quality of document summaries by measuring how much they improve a pre-trained language model's performance on the document's text. It estimates summary quality without needing human-written references, using two variations: BLANC-help and BLANC-tune."
 sdk: gradio
-sdk_version: 4.44.1
+sdk_version: 3.19.1
 app_file: app.py
 pinned: false
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# Metric Card for BLANC
+
+## Metric Description
+BLANC is a reference-free evaluation metric designed to estimate the quality of document summaries. It assesses how much a summary helps a pre-trained language model (such as BERT) perform language understanding tasks on the original document's text.
+
+There are two variations of BLANC:
+
+1. BLANC-help: The summary is concatenated with document sentences during the inference task. The BLANC-help score is defined as the difference in accuracy between unmasking tokens with the summary and with filler text (same length as the summary but consisting of period symbols). It measures how much the summary boosts the model's performance on masked tokens.
+2. BLANC-tune: The model is fine-tuned on the summary text before it processes the entire document. The BLANC-tune score is calculated by comparing the performance of the fine-tuned model with that of the original model, both tasked with unmasking tokens in the document text. This method reflects how much the model's ability to understand the document improves after learning from the summary.
+
+Unlike traditional metrics such as ROUGE, BLANC does not require human-written reference summaries, making it fully human-free.
+
+## How to Use
+BLANC takes 2 mandatory arguments: `documents` (a list of documents) and `summaries` (a list of predicted summaries).
+You can specify which BLANC variation to use via the `blanc_score` parameter: `help` or `tune`.
+```python
+from evaluate import load
+
+blanc = load("phucdev/blanc")
+documents = ["Jack drove his minivan to the bazaar to purchase milk and honey for his large family."]
+summaries = ["Jack bought milk and honey."]
+results = blanc.compute(documents=documents, summaries=summaries, blanc_score="help")
+```
+
+### Inputs
+
+Args:
+- `documents` (_list of str_): Documents.
+- `summaries` (_list of str_): Predicted summaries.
+- `model_name` (_str, optional_): BERT model type to use for evaluation. Default is `bert-base-uncased`.
+- `measure` (_str, optional_): Measure type, either `improve` or `relative`, as defined in the BLANC paper. Default is `relative`.
+- `blanc_score` (_str, optional_): BLANC score type, either `help` or `tune`. Default is `help`.
+- `gap` (_int, optional_): Distance between words to mask during inference. Default is `2`.
+- `gap_mask` (_int, optional_): Number of tokens to mask at each designated position during inference. Default is `1`.
+- `gap_tune` (_int, optional_): Distance between words to mask during fine-tuning. Default is `2`.
+- `gap_mask_tune` (_int, optional_): Number of tokens to mask at each designated position during fine-tuning. Default is `1`.
+- `min_token_length_normal` (_int, optional_): Minimum number of characters in normal tokens to mask (whole words) during inference. Default is `4`.
+- `min_token_length_lead` (_int, optional_): Minimum number of characters in lead tokens (first part of words) to mask during inference. Default is `2`.
+- `min_token_length_followup` (_int, optional_): Minimum number of characters in follow-up tokens (continuations of words) to mask during inference. Default is `100`.
+- `min_token_length_normal_tune` (_int, optional_): Minimum number of characters in normal tokens to mask during fine-tuning. Default is `-1`.
+- `min_token_length_lead_tune` (_int, optional_): Minimum number of characters in lead tokens to mask during fine-tuning. Default is `-1`.
+- `min_token_length_followup_tune` (_int, optional_): Minimum number of characters in follow-up tokens to mask during fine-tuning. Default is `-1`.
+- `device` (_str, optional_): Device to run the model on, either `cpu` or `cuda`. BLANC is run on `cpu` per default.
+- `random_seed` (_int, optional_): Random seed for Python and PyTorch. Default is `0`.
+- `inference_batch_size` (_int, optional_): Batch size for inference. Default is `1`.
+- `inference_mask_evenly` (_bool, optional_): Whether to mask every `gap` tokens during inference (`True`) or mask randomly with a probability of 0.15 (`False`). Default is `True`.
+
+BLANC-help specific arguments:
+- `filler_token` (_str, optional_): Token to use as filler in lieu of the summary. Default is `.`.
+- `help_sep` (_str, optional_): Token used to separate the summary (or filler) from the sentence, or '' for no separator. Default is "".
+
+BLANC-tune specific arguments:
+- `finetune_batch_size` (_int, optional_): Batch size to use when fine-tuning on the summary. Default is `1`.
+- `finetune_epochs` (_int, optional_): Number of epochs for fine-tuning on the summary. Default is `10`.
+- `finetune_mask_evenly` (_bool, optional_): Whether to mask every `gap` tokens during fine-tuning (`True`) or mask randomly with a probability of 0.15 (`False`). Default is `True`.
+- `finetune_chunk_size` (_int, optional_): Number of summary tokens to use at a time during fine-tuning. Default is `64`.
+- `finetune_chunk_stride` (_int, optional_): Number of tokens between summary chunks for fine-tuning. Default is `32`.
+- `learning_rate` (_float, optional_): Learning rate for fine-tuning on the summary. Default is `5e-05`.
+- `warmup_steps` (_int, optional_): Number of warmup steps for fine-tuning. Default is `0`.
+
+### Output Values
+
+The metric outputs a dictionary with the following key ("blanc_tune" or "blanc_help" depending on the chosen score) and value:
+
+- blanc_{tune,help}: A floating-point score representing the quality of the summary.
+
+The BLANC score typically ranges from 0 (summary is not helpful) to 0.3 (summary provides a 30% improvement in performance), although it can theoretically range between -1 and 1. Higher scores indicate better quality summaries.
+
+#### Values from Popular Papers
+Goyal et al. (2022) compare the performance of different summarization systems using reference-free automatic metrics in [News Summarization and Evaluation in the Era of GPT-3](https://arxiv.org/abs/2209.12356).
+For the DailyMail dataset they report the following BLANC scores:
+
+- PEGASUS: 0.1137
+- BRIO: 0.1217
+- T0: 0.0889
+- GPT3-D2: 0.0983
+
+### Examples
+BLANC-help:
+```python
+from evaluate import load
+
+blanc = load("phucdev/blanc")
+documents = ["Jack drove his minivan to the bazaar to purchase milk and honey for his large family."]
+summaries = ["Jack bought milk and honey."]
+results = blanc.compute(documents=documents, summaries=summaries, blanc_score="help")
+```
+BLANC-tune:
+```python
+from evaluate import load
+
+blanc = load("phucdev/blanc")
+documents = ["Jack drove his minivan to the bazaar to purchase milk and honey for his large family."]
+summaries = ["Jack bought milk and honey."]
+results = blanc.compute(
+    documents=documents, 
+    summaries=summaries, 
+    blanc_score="tune", 
+    finetune_mask_evenly=False, 
+    show_progress_bar=False
+)
+```
+By default, BLANC is run on the CPU. Using CUDA with batching is much faster:
+
+BLANC-help:
+```python
+from evaluate import load
+
+blanc = load("phucdev/blanc")
+documents = ["Jack drove his minivan to the bazaar to purchase milk and honey for his large family."]
+summaries = ["Jack bought milk and honey."]
+results = blanc.compute(
+    documents=documents, 
+    summaries=summaries, 
+    blanc_score="help", 
+    device="cuda",
+    inference_batch_size=128
+)
+```
+BLANC-tune:
+```python
+from evaluate import load
+
+blanc = load("phucdev/blanc")
+documents = ["Jack drove his minivan to the bazaar to purchase milk and honey for his large family."]
+summaries = ["Jack bought milk and honey."]
+results = blanc.compute(
+    documents=documents, 
+    summaries=summaries, 
+    blanc_score="tune", 
+    device="cuda",
+    inference_batch_size=24, 
+    finetune_mask_evenly=False, 
+    finetune_batch_size=24
+)
+```
+
+## Limitations and Bias
+- Summary Length: BLANC tends to favor longer summaries as they generally provide more context and help the model better understand the document.
+- No Reference Summaries: BLANC operates without human-written reference summaries, which may be advantageous in certain cases but could lack the nuance that human judgment provides.
+- Limited by Language Model: The quality of BLANC scores is influenced by the choice of the underlying pre-trained language model (e.g., BERT), which may introduce biases inherent to the model itself.
+
+## Citation
+```tex
+@inproceedings{vasilyev-etal-2020-fill,
+    title = "Fill in the {BLANC}: Human-free quality estimation of document summaries",
+    author = "Vasilyev, Oleg  and
+      Dharnidharka, Vedant  and
+      Bohannon, John",
+    editor = "Eger, Steffen  and
+      Gao, Yang  and
+      Peyrard, Maxime  and
+      Zhao, Wei  and
+      Hovy, Eduard",
+    booktitle = "Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems",
+    month = nov,
+    year = "2020",
+    address = "Online",
+    publisher = "Association for Computational Linguistics",
+    url = "https://aclanthology.org/2020.eval4nlp-1.2",
+    doi = "10.18653/v1/2020.eval4nlp-1.2",
+    pages = "11--20",
+    abstract = "We present BLANC, a new approach to the automatic estimation of document summary quality. Our goal is to measure the functional performance of a summary with an objective, reproducible, and fully automated method. Our approach achieves this by measuring the performance boost gained by a pre-trained language model with access to a document summary while carrying out its language understanding task on the document{'}s text. We present evidence that BLANC scores have as good correlation with human evaluations as do the ROUGE family of summary quality measurements. And unlike ROUGE, the BLANC method does not require human-written reference summaries, allowing for fully human-free summary quality estimation.",
+}
+```
+
+## Further References
+- [BLANC paper: Fill in the BLANC: Human-free quality estimation of document summaries](https://aclanthology.org/2020.eval4nlp-1.2/)
+- [BLANC GitHub Repository](https://github.com/PrimerAI/blanc)

app.py

@@ -0,0 +1,6 @@
+import evaluate
+from evaluate.utils import launch_gradio_widget
+
+
+module = evaluate.load("phucdev/blanc_score")
+launch_gradio_widget(module)

blanc_score.py

@@ -0,0 +1,190 @@
+import datasets
+import evaluate
+
+from blanc import BlancHelp, BlancTune
+
+
+
+
+_DESCRIPTION = """
+BLANC is an automatic method for estimating the quality of document summaries without the need for human-written reference summaries. It works by measuring how much a summary helps a pre-trained language model (like BERT) perform a language understanding task, such as filling in masked words in a document. The two main variations are:
+
+1. BLANC-help: The summary is concatenated with document sentences during the inference task. The BLANC-help score is defined as the difference in accuracy between unmasking tokens with the summary and with filler text (same length as the summary but consisting of period symbols). It measures how much the summary boosts the model's performance on masked tokens.
+2. BLANC-tune: The model is fine-tuned on the summary text before it processes the entire document. The BLANC-tune score is calculated by comparing the performance of the fine-tuned model with that of the original model, both tasked with unmasking tokens in the document text. This method reflects how much the model's ability to understand the document improves after learning from the summary.
+
+These BLANC measures show good correlation with human evaluations, similar to ROUGE scores, but do not require reference summaries.
+See the BLANC paper for more details: https://aclanthology.org/2020.eval4nlp-1.2/
+"""
+
+_KWARGS_DESCRIPTION = """
+Args:
+    documents (list of str): Documents.
+    summaries (list of str): Predicted summaries.
+    model_name (str, optional): BERT model type to use for evaluation. Default is "bert-base-uncased".
+    measure (str, optional): Measure type, either "improve" or "relative", as defined in the BLANC paper. Default is "relative".
+    blanc_score (str, optional): BLANC score type, either "help" or "tune". Default is "help".
+    gap (int, optional): Distance between words to mask during inference. Default is 2.
+    gap_mask (int, optional): Number of tokens to mask at each designated position during inference. Default is 1.
+    gap_tune (int, optional): Distance between words to mask during fine-tuning. Default is 2.
+    gap_mask_tune (int, optional): Number of tokens to mask at each designated position during fine-tuning. Default is 1.
+    min_token_length_normal (int, optional): Minimum number of characters in normal tokens to mask (whole words) during inference. Default is 4.
+    min_token_length_lead (int, optional): Minimum number of characters in lead tokens (first part of words) to mask during inference. Default is 2.
+    min_token_length_followup (int, optional): Minimum number of characters in follow-up tokens (continuations of words) to mask during inference. Default is 100.
+    min_token_length_normal_tune (int, optional): Minimum number of characters in normal tokens to mask during fine-tuning. Default is -1.
+    min_token_length_lead_tune (int, optional): Minimum number of characters in lead tokens to mask during fine-tuning. Default is -1.
+    min_token_length_followup_tune (int, optional): Minimum number of characters in follow-up tokens to mask during fine-tuning. Default is -1.
+    device (str, optional): Device to run the model on, either "cpu" or "cuda". Defaults to "cpu".
+    random_seed (int, optional): Random seed for Python and PyTorch. Default is 0.
+    inference_batch_size (int, optional): Batch size for inference. Default is 1.
+    inference_mask_evenly (bool, optional): Whether to mask every `gap` tokens during inference (True) or mask randomly with a probability of 0.15 (False). Default is True.
+
+    BLANC-help specific arguments:
+        filler_token (str, optional): Token to use as filler in lieu of the summary. Default is ".".
+        help_sep (str, optional): Token used to separate the summary (or filler) from the sentence, or '' for no separator. Default is "".
+
+    BLANC-tune specific arguments:
+        finetune_batch_size (int, optional): Batch size to use when fine-tuning on the summary. Default is 1.
+        finetune_epochs (int, optional): Number of epochs for fine-tuning on the summary. Default is 10.
+        finetune_mask_evenly (bool, optional): Whether to mask every `gap` tokens during fine-tuning (True) or mask randomly with a probability of 0.15 (False). Default is True.
+        finetune_chunk_size (int, optional): Number of summary tokens to use at a time during fine-tuning. Default is 64.
+        finetune_chunk_stride (int, optional): Number of tokens between summary chunks for fine-tuning. Default is 32.
+        learning_rate (float, optional): Learning rate for fine-tuning on the summary. Default is 5e-05.
+        warmup_steps (int, optional): Number of warmup steps for fine-tuning. Default is 0.
+
+Returns:
+    score (float): The calculated BLANC score based on the selected method (BLANC-help or BLANC-tune).
+"""
+
+_CITATION = """
+@inproceedings{vasilyev-etal-2020-fill,
+    title = "Fill in the {BLANC}: Human-free quality estimation of document summaries",
+    author = "Vasilyev, Oleg  and
+      Dharnidharka, Vedant  and
+      Bohannon, John",
+    editor = "Eger, Steffen  and
+      Gao, Yang  and
+      Peyrard, Maxime  and
+      Zhao, Wei  and
+      Hovy, Eduard",
+    booktitle = "Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems",
+    month = nov,
+    year = "2020",
+    address = "Online",
+    publisher = "Association for Computational Linguistics",
+    url = "https://aclanthology.org/2020.eval4nlp-1.2",
+    doi = "10.18653/v1/2020.eval4nlp-1.2",
+    pages = "11--20",
+    abstract = "We present BLANC, a new approach to the automatic estimation of document summary quality. Our goal is to measure the functional performance of a summary with an objective, reproducible, and fully automated method. Our approach achieves this by measuring the performance boost gained by a pre-trained language model with access to a document summary while carrying out its language understanding task on the document{'}s text. We present evidence that BLANC scores have as good correlation with human evaluations as do the ROUGE family of summary quality measurements. And unlike ROUGE, the BLANC method does not require human-written reference summaries, allowing for fully human-free summary quality estimation.",
+}
+"""
+
+
+@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+class BLANC(evaluate.Metric):
+    def _info(self):
+        return evaluate.MetricInfo(
+            description=_DESCRIPTION,
+            citation=_CITATION,
+            homepage="https://github.com/PrimerAI/blanc",
+            inputs_description=_KWARGS_DESCRIPTION,
+            features=[
+                datasets.Features(
+                    {
+                        "documents": datasets.Value("string", id="sequence"),
+                        "summaries": datasets.Value("string", id="sequence"),
+                    }
+                ),
+            ],
+            codebase_urls=["https://github.com/PrimerAI/blanc"],
+            reference_urls=[
+                "https://github.com/PrimerAI/blanc",
+                "https://aclanthology.org/2020.eval4nlp-1.2/",
+            ],
+        )
+
+    def _compute(
+            self,
+            documents,
+            summaries,
+            model_name="bert-base-uncased",
+            blanc_score="help",
+            measure="relative",
+            gap=2,
+            gap_mask=1,
+            gap_tune=2,
+            gap_mask_tune=1,
+            min_token_length_normal=4,
+            min_token_length_lead=2,
+            min_token_length_followup=100,
+            min_token_length_normal_tune=-1,
+            min_token_length_lead_tune=-1,
+            min_token_length_followup_tune=-1,
+            device="cpu",
+            random_seed=0,
+            inference_batch_size=1,
+            inference_mask_evenly=True,
+            filler_token=".",
+            help_sep="",
+            finetune_batch_size=1,
+            finetune_epochs=10,
+            finetune_mask_evenly=True,
+            finetune_chunk_size=64,
+            finetune_chunk_stride=32,
+            learning_rate=5e-05,
+            warmup_steps=0,
+    ):
+        # Choose between BLANC-help and BLANC-tune methods based on measure argument.
+        if blanc_score == "help":
+            blanc_instance = BlancHelp(
+                model_name=model_name,
+                measure=measure,
+                gap=gap,
+                gap_mask=gap_mask,
+                gap_tune=gap_tune,
+                gap_mask_tune=gap_mask_tune,
+                min_token_length_normal=min_token_length_normal,
+                min_token_length_lead=min_token_length_lead,
+                min_token_length_followup=min_token_length_followup,
+                min_token_length_normal_tune=min_token_length_normal_tune,
+                min_token_length_lead_tune=min_token_length_lead_tune,
+                device=device,
+                inference_batch_size=inference_batch_size,
+                inference_mask_evenly=inference_mask_evenly,
+                filler_token=filler_token,
+                help_sep=help_sep,
+            )
+        elif blanc_score == "tune":
+            blanc_instance = BlancTune(
+                model_name=model_name,
+                measure=measure,
+                gap=gap,
+                gap_mask=gap_mask,
+                gap_tune=gap_tune,
+                gap_mask_tune=gap_mask_tune,
+                min_token_length_normal=min_token_length_normal,
+                min_token_length_lead=min_token_length_lead,
+                min_token_length_followup_tune=min_token_length_followup_tune,
+                min_token_length_normal_tune=min_token_length_normal_tune,
+                min_token_length_lead_tune=min_token_length_lead_tune,
+                device=device,
+                random_seed=random_seed,
+                inference_batch_size=inference_batch_size,
+                inference_mask_evenly=inference_mask_evenly,
+                finetune_batch_size=finetune_batch_size,
+                finetune_epochs=finetune_epochs,
+                finetune_mask_evenly=finetune_mask_evenly,
+                finetune_chunk_size=finetune_chunk_size,
+                finetune_chunk_stride=finetune_chunk_stride,
+                learning_rate=learning_rate,
+                warmup_steps=warmup_steps,
+            )
+        else:
+            raise ValueError(f"Invalid measure argument: {measure}. Choose 'help' or 'tune'.")
+
+        score = blanc_instance.eval_pairs(documents, summaries)
+
+        output_dict = {
+            f"blanc_{blanc_score}": score  # Replace with actual computed score
+        }
+
+        return output_dict

requirements.txt

@@ -0,0 +1,2 @@
+blanc~=0.3.4
+evaluate~=0.4.3