metadata
license: cc-by-nc-4.0
This is a Factual Consistency Evaluation model, introduced in the TrueTeacher paper (Gekhman et al, 2023).
The model is optimized for evaluating factual consistency in summarization.
It is the main model from the paper (see "T5-11B w. ANLI + TrueTeacher full" in Table 1) which is based on a T5-11B fine-tuned with a mixture of the following datasets:
- TrueTeacher (Gekhman et al., 2023)
- ANLI (Nie et al., 2020)
The input format for the model is: "premise: GROUNDING_DOCUMENT hypothesis: HYPOTHESIS_SUMMARY".
The model predicts a binary label ('1' - Factualy Consistent, '0' - Factualy Inconsistent).
Usage example:
from transformers import T5ForConditionalGeneration
from transformers import T5Tokenizer
model_path = 'google/t5_11b_trueteacher_and_anli'
tokenizer = T5Tokenizer.from_pretrained(model_path)
model = T5ForConditionalGeneration.from_pretrained(model_path)
premise = 'the sun is shining'
for hypothesis, expected in [('the sun is out in the sky', '1'),
('the cat is shiny', '0')]:
input_ids = tokenizer(f'premise: {premise} hypothesis: {hypothesis}', return_tensors='pt').input_ids
outputs = model.generate(input_ids)
result = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(f'premise: {premise}')
print(f'hypothesis: {hypothesis}')
print(f'result: {result} (expected: {expected})\n')
Citation
If you use this model for a research publication, please cite the TrueTeacher paper (using the bibtex entry below) and the dataset papers mentioned above.
@misc{gekhman2023trueteacher,
title={TrueTeacher: Learning Factual Consistency Evaluation with Large Language Models},
author={Zorik Gekhman and Jonathan Herzig and Roee Aharoni and Chen Elkind and Idan Szpektor},
year={2023},
eprint={2305.11171},
archivePrefix={arXiv},
primaryClass={cs.CL}
}