# AlignScore
This is the repository for AlignScore and its checkpoints, a metric for automatic factual consistency evaluation of text pairs. The metric is introduced in 

[AlignScore: Evaluating Factual Consistency with a Unified Alignment Function](https://arxiv.org/abs/2305.16739)

Yuheng Zha, Yichi Yang, Ruichen Li and Zhiting Hu

ACL 2023

Code is at https://github.com/yuh-zha/AlignScore

What is factual consistency and its evaluation?
* **Facutual Consistency:** For a given text pair (**a**, **b**), they are considered factual consistent if 1) all the information in **b** is also present in **a**; 2) **b** does not contradict **a**.
* **Evaluation:**  Show the degree of factual consistency between the context (text **a**) and the claim (text **b**).

Where is factual consistency evaluation applicable?
* **Summarization**: document and summary
* **Paraphrase**: sentence A and sentence B
* **Dialog**: context and response
* ...

# Leaderboard
We list the performance of AlignScore as well as other metrics here. 

| Rank | Metrics          | SummaC* | TRUE** | Other-Spearman | Average | Paper | Code |
| ---- | :--------------- | :-----: | :----: | :------------: | :-----: | :---: | :--: |
| 1    | AlignScore-large |  88.6   |  83.8  |      49.3      |  73.9   |   -   |   -  |
| 2    | AlignScore-base  |  87.4   |  82.5  |      44.9      |  71.6   |   -   |   -  |
| 3    | QAFactEval       |  83.8   |  79.4  |      42.4      |  68.5   | [:page\_facing\_up:(Fabbri et al. 2022)](https://arxiv.org/abs/2112.08542) | [:octocat:](https://github.com/salesforce/QAFactEval) |
| 4    | UniEval          |  84.6   |  78.0  |      41.5      |  68.0   | [:page\_facing\_up:(Zhong et al. 2022)](https://arxiv.org/abs/2210.07197) | [:octocat:](https://github.com/maszhongming/UniEval) |
| 5    | SummaC-CONV      |  81.0   |  78.7  |      34.2      |  64.6   | [:page\_facing\_up:(Laban et al. 2022)](https://arxiv.org/abs/2111.09525) | [:octocat:](https://github.com/tingofurro/summac) |
| 6    | BARTScore        |  80.9   |  73.4  |      34.8      |  63.0   | [:page\_facing\_up:(Yuan et al. 2022)](https://arxiv.org/abs/2106.11520) | [:octocat:](https://github.com/neulab/BARTScore) |
| 7    | CTC              |  81.2   |  72.4  |      35.3      |  63.0   | [:page\_facing\_up:(Deng et al. 2022)](https://arxiv.org/abs/2109.06379) | [:octocat:](https://github.com/tanyuqian/ctc-gen-eval) |
| 8    | SummaC-ZS        |  79.0   |  78.2  |      30.4      |  62.5   | [:page\_facing\_up:(Laban et al. 2022)](https://arxiv.org/abs/2111.09525) | [:octocat:](https://github.com/tingofurro/summac) |
| 9    | ROUGE-2          |  78.1   |  72.4  |      27.9      |  59.5   | [:page\_facing\_up:(Lin 2004)](https://aclanthology.org/W04-1013/) | [:octocat:](https://github.com/pltrdy/rouge) |
| 10   | ROUGE-1          |  77.4   |  72.0  |      28.6      |  59.3   | [:page\_facing\_up:(Lin 2004)](https://aclanthology.org/W04-1013/) | [:octocat:](https://github.com/pltrdy/rouge) |
| 11   | ROUGE-L          |  77.3   |  71.8  |      28.3      |  59.1   | [:page\_facing\_up:(Lin 2004)](https://aclanthology.org/W04-1013/) | [:octocat:](https://github.com/pltrdy/rouge) |
| 12   | QuestEval        |  72.5   |  71.4  |      25.0      |  56.3   | [:page\_facing\_up:(Scialom et al. 2021)](https://arxiv.org/abs/2103.12693) | [:octocat:](https://github.com/ThomasScialom/QuestEval) |
| 13   | BLEU             |  76.3   |  67.3  |      24.6      |  56.1   | [:page\_facing\_up:(Papineni et al. 2002)](https://aclanthology.org/P02-1040/) | [:octocat:](https://www.nltk.org/_modules/nltk/translate/bleu_score.html) |
| 14   | DAE              |  66.8   |  65.7  |      35.1      |  55.8   | [:page\_facing\_up:(Goyal and Durrett 2020)](https://aclanthology.org/2020.findings-emnlp.322/) | [:octocat:](https://github.com/tagoyal/dae-factuality) |
| 15   | BLEURT           |  69.2   |  71.9  |      24.9      |  55.4   | [:page\_facing\_up:(Sellam et al. 2020)](https://arxiv.org/abs/2004.04696) | [:octocat:](https://github.com/google-research/bleurt) |
| 16   | BERTScore        |  72.1   |  68.6  |      21.9      |  54.2   | [:page\_facing\_up:(Zhang et al. 2020)](https://arxiv.org/abs/1904.09675) | [:octocat:](https://github.com/Tiiiger/bert_score) |
| 17   | SimCSE           |  67.4   |  70.3  |      23.8      |  53.8   | [:page\_facing\_up:(Gao et al. 2021)](https://arxiv.org/abs/2104.08821) | [:octocat:](https://github.com/princeton-nlp/SimCSE) |
| 18   | FactCC           |  68.8   |  62.7  |      21.2      |  50.9   | [:page\_facing\_up:(Kryscinski et al. 2020)](https://arxiv.org/abs/1910.12840) | [:octocat:](https://github.com/salesforce/factCC) |
| 19   | BLANC            |  65.1   |  64.0  |      14.4      |  47.8   | [:page\_facing\_up:(Vasilyev et al. 2020)](https://arxiv.org/abs/2002.09836) | [:octocat:](https://github.com/PrimerAI/blanc) |
| 20   | NER-Overlap      |  60.4   |  59.3  |      18.9      |  46.2   | [:page\_facing\_up:(Laban et al. 2022)](https://arxiv.org/abs/2111.09525) | [:octocat:](https://github.com/tingofurro/summac) |
| 21   | MNLI             |  47.9   |  60.4  |      3.1       |  37.2   | [:page\_facing\_up:(Williams et al. 2018)](https://arxiv.org/abs/1704.05426) | [:octocat:](https://github.com/nyu-mll/multiNLI) |
| 22   | FEQA             |  48.3   |  52.2  |      -1.9      |  32.9   | [:page\_facing\_up:(Durmus et al. 2020)](https://arxiv.org/abs/2005.03754) | [:octocat:](https://github.com/esdurmus/feqa) |

\*  SummaC: [\[Paper\]](https://arxiv.org/abs/2111.09525) \| [\[Github\]](https://github.com/tingofurro/summac)

** TRUE: [\[Paper\]](https://arxiv.org/abs/2204.04991) \| [\[Github\]](https://github.com/google-research/true)


# Installation

Our models are trained and evaluated using PyTorch 1.12.1. We recommend using this version to reproduce the results.

1. Please first install the right version of PyTorch before installing `alignscore`.
2. You can install `alignscore` by cloning this repository and `pip install .`.
3. After installing `alignscore`, please use `python -m spacy download en_core_web_sm` to install the required spaCy model (we use `spaCy` for sentenization).

# Evaluating Factual Consistency
To evaluate the factual consistency of the `claim` w.r.t. the `context`, simply use the score method of `AlignScore`.
```python
from alignscore import AlignScore

scorer = AlignScore(model='roberta-base', batch_size=32, device='cuda:0', ckpt_path='/path/to/checkpoint', evaluation_mode='nli_sp')
score = scorer.score(contexts=['hello world'], claims=['hello world'])
```
`model`: the backbone model of the metric. Now, we only provide the metric trained on RoBERTa

`batch_size`: the batch size of the inference

`device`: which device to run the metric

`ckpt_path`: the path to the checkpoint

`evaluation_mode`: choose from `'nli_sp', 'nli', 'bin_sp', 'bin'`. `nli` and `bin` refer to the 3-way and binary classficiation head, respectively. `sp` indicates if the chunk-sentence splitting method is used. `nli_sp` is the default setting of AlignScore


# Checkpoints
We provide two versions of the AlignScore checkpoints: `AlignScore-base` and `AlignScore-large`. The `-base` model is based on RoBERTa-base and has 125M parameters. The `-large` model is based on RoBERTa-large and has 355M parameters. 

**AlignScore-base**: 
https://huggingface.co/yzha/AlignScore/resolve/main/AlignScore-base.ckpt

**AlignScore-large**:
https://huggingface.co/yzha/AlignScore/resolve/main/AlignScore-large.ckpt

# Training  
You can use the above checkpoints directly for factual consistency evaluation. However, if you wish to train an alignment model from scratch / on your own data, use `train.py`.
```python
python train.py --seed 2022 --batch-size 32 \
--num-epoch 3 --devices 0 1 2 3 \
--model-name roberta-large -- ckpt-save-path ./ckpt/ \
--data-path ./data/training_sets/ \
--max-samples-per-dataset 500000
```

`--seed`: the random seed for initialization

`--batch-size`: the batch size for training

`--num-epoch`: training epochs

`--devices`: which devices to train the metric, a list of GPU ids

`--model-name`: the backbone model name of the metric, default RoBERTa-large

`--ckpt-save-path`: the path to save the checkpoint

`--training-datasets`: the names of the training datasets

`--data-path`: the path to the training datasets

`--max-samples-per-dataset`: the maximum number of samples from a dataset

# Benchmarking
Our benchmark includes the TRUE and SummaC benchmark as well as several popular factual consistency evaluation datasets.

To run the benchmark, a few additional dependencies are required and can be installed with `pip install -r requirements.txt`.
Additionally, some depedencies are not available as packages and need to be downloaded manually (please see `python benchmark.py --help` for instructions).

Note installing `summac` may cause dependency conflicts with `alignscore`. Please reinstall `alignscore` to force the correct dependency versions.

The relevant arguments for evaluating AlignScore are:

`--alignscore`: evaluation the AlignScore metric

`--alignscore-model`: the name of the backbone model (either 'roberta-base' or 'roberta-large')

`--alignscore-ckpt`: the path to the saved checkpoint

`--alignscore-eval-mode`: the evaluation mode, defaults to `nli_sp`

`--device`: which device to run the metric, defaults to `cuda:0`

`--tasks`: which tasks to benchmark, e.g., SummEval, QAGS-CNNDM, ...

For the baselines, please see `python benchmark.py --help` for details.

{"license": "mit"}
arxiv:2305.16739 arxiv:2112.08542 arxiv:2210.07197 arxiv:2111.09525 arxiv:2106.11520 arxiv:2109.06379 arxiv:2103.12693 arxiv:2004.04696 arxiv:1904.09675 arxiv:2104.08821 arxiv:1910.12840 arxiv:2002.09836 arxiv:1704.05426 arxiv:2005.03754 arxiv:2204.04991 license:mit region:us
TAGS
#arxiv-2305.16739 #arxiv-2112.08542 #arxiv-2210.07197 #arxiv-2111.09525 #arxiv-2106.11520 #arxiv-2109.06379 #arxiv-2103.12693 #arxiv-2004.04696 #arxiv-1904.09675 #arxiv-2104.08821 #arxiv-1910.12840 #arxiv-2002.09836 #arxiv-1704.05426 #arxiv-2005.03754 #arxiv-2204.04991 #license-mit #region-us 

AlignScore
==========


This is the repository for AlignScore and its checkpoints, a metric for automatic factual consistency evaluation of text pairs. The metric is introduced in


AlignScore: Evaluating Factual Consistency with a Unified Alignment Function


Yuheng Zha, Yichi Yang, Ruichen Li and Zhiting Hu


ACL 2023


Code is at URL


What is factual consistency and its evaluation?


* Facutual Consistency: For a given text pair (a, b), they are considered factual consistent if 1) all the information in b is also present in a; 2) b does not contradict a.
* Evaluation: Show the degree of factual consistency between the context (text a) and the claim (text b).


Where is factual consistency evaluation applicable?


* Summarization: document and summary
* Paraphrase: sentence A and sentence B
* Dialog: context and response
* ...


Leaderboard
===========


We list the performance of AlignScore as well as other metrics here.



\* SummaC: [[Paper]](URL | [[Github]](URL


TRUE: [[Paper]](URL | [[Github]](URL


Installation
============


Our models are trained and evaluated using PyTorch 1.12.1. We recommend using this version to reproduce the results.


1. Please first install the right version of PyTorch before installing 'alignscore'.
2. You can install 'alignscore' by cloning this repository and 'pip install .'.
3. After installing 'alignscore', please use 'python -m spacy download en\_core\_web\_sm' to install the required spaCy model (we use 'spaCy' for sentenization).


Evaluating Factual Consistency
==============================


To evaluate the factual consistency of the 'claim' w.r.t. the 'context', simply use the score method of 'AlignScore'.


'model': the backbone model of the metric. Now, we only provide the metric trained on RoBERTa


'batch\_size': the batch size of the inference


'device': which device to run the metric


'ckpt\_path': the path to the checkpoint


'evaluation\_mode': choose from ''nli\_sp', 'nli', 'bin\_sp', 'bin''. 'nli' and 'bin' refer to the 3-way and binary classficiation head, respectively. 'sp' indicates if the chunk-sentence splitting method is used. 'nli\_sp' is the default setting of AlignScore


Checkpoints
===========


We provide two versions of the AlignScore checkpoints: 'AlignScore-base' and 'AlignScore-large'. The '-base' model is based on RoBERTa-base and has 125M parameters. The '-large' model is based on RoBERTa-large and has 355M parameters.


AlignScore-base:
URL


AlignScore-large:
URL


Training
========


You can use the above checkpoints directly for factual consistency evaluation. However, if you wish to train an alignment model from scratch / on your own data, use 'URL'.


'--seed': the random seed for initialization


'--batch-size': the batch size for training


'--num-epoch': training epochs


'--devices': which devices to train the metric, a list of GPU ids


'--model-name': the backbone model name of the metric, default RoBERTa-large


'--ckpt-save-path': the path to save the checkpoint


'--training-datasets': the names of the training datasets


'--data-path': the path to the training datasets


'--max-samples-per-dataset': the maximum number of samples from a dataset


Benchmarking
============


Our benchmark includes the TRUE and SummaC benchmark as well as several popular factual consistency evaluation datasets.


To run the benchmark, a few additional dependencies are required and can be installed with 'pip install -r URL'.
Additionally, some depedencies are not available as packages and need to be downloaded manually (please see 'python URL --help' for instructions).


Note installing 'summac' may cause dependency conflicts with 'alignscore'. Please reinstall 'alignscore' to force the correct dependency versions.


The relevant arguments for evaluating AlignScore are:


'--alignscore': evaluation the AlignScore metric


'--alignscore-model': the name of the backbone model (either 'roberta-base' or 'roberta-large')


'--alignscore-ckpt': the path to the saved checkpoint


'--alignscore-eval-mode': the evaluation mode, defaults to 'nli\_sp'


'--device': which device to run the metric, defaults to 'cuda:0'


'--tasks': which tasks to benchmark, e.g., SummEval, QAGS-CNNDM, ...


For the baselines, please see 'python URL --help' for details.



TAGS
#arxiv-2305.16739 #arxiv-2112.08542 #arxiv-2210.07197 #arxiv-2111.09525 #arxiv-2106.11520 #arxiv-2109.06379 #arxiv-2103.12693 #arxiv-2004.04696 #arxiv-1904.09675 #arxiv-2104.08821 #arxiv-1910.12840 #arxiv-2002.09836 #arxiv-1704.05426 #arxiv-2005.03754 #arxiv-2204.04991 #license-mit #region-us 
