README.md

---
license: cc-by-nc-4.0
---

This is a **Factual Consistency Evaluation** model, introduced in the [TrueTeacher paper (Gekhman et al, 2023)](https://arxiv.org/pdf/2305.11171.pdf).

The model is optimized for evaluating factual consistency in **summarization**.

It is the main model from the paper (see **"T5-11B w. ANLI + TrueTeacher full"** in **Table 1**) which is based on a **T5-11B** fine-tuned with a mixture of the following datasets:
- TrueTeacher ([Gekhman et al., 2023](https://arxiv.org/pdf/2305.11171.pdf))
- ANLI ([Nie et al., 2020](https://aclanthology.org/2020.acl-main.441.pdf))


The input format for the model is: "premise: GROUNDING_DOCUMENT hypothesis: HYPOTHESIS_SUMMARY".

The model predicts a binary label ('1' - Factualy Consistent, '0' - Factualy Inconsistent).


## Usage example:
```python
from transformers import T5ForConditionalGeneration
from transformers import T5Tokenizer

model_path = 'google/t5_11b_trueteacher_and_anli'
tokenizer = T5Tokenizer.from_pretrained(model_path)
model = T5ForConditionalGeneration.from_pretrained(model_path)

premise = 'the sun is shining'
for hypothesis, expected in [('the sun is out in the sky', '1'), 
                             ('the cat is shiny', '0')]:
  input_ids = tokenizer(f'premise: {premise} hypothesis: {hypothesis}', return_tensors='pt').input_ids
  outputs = model.generate(input_ids)
  result = tokenizer.decode(outputs[0], skip_special_tokens=True)
  print(f'premise: {premise}')
  print(f'hypothesis: {hypothesis}')
  print(f'result: {result} (expected: {expected})\n')
```

## Citation

If you use this model for a research publication, please cite the TrueTeacher paper (using the bibtex entry below) and the dataset papers mentioned above.

```
@misc{gekhman2023trueteacher,
      title={TrueTeacher: Learning Factual Consistency Evaluation with Large Language Models}, 
      author={Zorik Gekhman and Jonathan Herzig and Roee Aharoni and Chen Elkind and Idan Szpektor},
      year={2023},
      eprint={2305.11171},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```