Hugging Face's logo

liujch1998
/
vera

Text Classification
Transformers
PyTorch
English
t5
text-generation-inference
Inference Endpoints
Model card Files Community
3
vera / README.md
liujch1998's picture
liujch1998
Update README.md
504b4a4
preview code
|
raw
history blame
4.56 kB
metadata
license: mit
language:
  - en
pipeline_tag: text-classification
arxiv: 2305.03695

Model Card for Vera

Vera is a commonsense statement verification model. See our paper at: https://arxiv.org/abs/2305.03695.

Model Details

Model Description

Given a commonsense statement as input, Vera predicts the plausibility of this statement. Vera outputs a real-valued score in the range [0, 1]. A score of 1 means the statement is correct according to commonsense, and a score of 0 means the statement is incorrect. This score is calibrated, so a score between 0 and 1 can be interpreted as Vera's confidence that the statement is correct.

  • Developed by: Jiacheng Liu, Wenya Wang, Dianzhuo Wang, Noah A. Smith, Yejin Choi, Hannaneh Hajishirzi
  • Shared by [optional]: Jiacheng Liu
  • Model type: Transformers
  • Language(s) (NLP): English
  • License: MIT
  • Finetuned from model [optional]: T5-v1.1-XXL

Model Sources [optional]

Uses

Direct Use

Vera is intended to predict the correctness of commonsense statements.

Downstream Use [optional]

Vera can be used to detect commonsense errors made by generative LMs (e.g., ChatGPT), or filter noisy commonsense knowledge generated by other LMs (e.g., Rainier).

Out-of-Scope Use

Vera is a research prototype and may make mistakes. Do not use for making critical decisions. It is intended to predict the correctness of commonsense statements, and may be unreliable when taking input out of this scope.

Bias, Risks, and Limitations

See the Limitations and Ethics Statement section of our paper.

Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.

How to Get Started with the Model

Use the code below to get started with the model.

tokenizer = transformers.AutoTokenizer.from_pretrained('liujch1998/vera')
model = transformers.T5EncoderModel.from_pretrained('liujch1998/vera')
model.D = model.shared.embedding_dim
linear = torch.nn.Linear(model.D, 1, dtype=model.dtype)
linear.weight = torch.nn.Parameter(model.shared.weight[32099, :].unsqueeze(0))
linear.bias = torch.nn.Parameter(model.shared.weight[32098, 0].unsqueeze(0))
model.eval()
t = model.shared.weight[32097, 0].item() # temperature for calibration

statement = 'Please enter a commonsense statement here.'
input_ids = tokenizer.batch_encode_plus([statement], return_tensors='pt', padding='longest', truncation='longest_first', max_length=128).input_ids
with torch.no_grad():
    output = model(input_ids)
    last_hidden_state = output.last_hidden_state
    hidden = last_hidden_state[0, -1, :]
    logit = linear(hidden).squeeze(-1)
    logit_calibrated = logit / t
    score_calibrated = logit_calibrated.sigmoid()
    # score_calibrated is Vera's final output plausibility score

You may also refer to https://huggingface.co/spaces/liujch1998/vera/blob/main/app.py#L27-L98 for implementation.

Citation [optional]

BibTeX:

@article{Liu2023VeraAG,
  title={Vera: A General-Purpose Plausibility Estimation Model for Commonsense Statements},
  author={Jiacheng Liu and Wenya Wang and Dianzhuo Wang and Noah A. Smith and Yejin Choi and Hanna Hajishirzi},
  journal={ArXiv},
  year={2023},
  volume={abs/2305.03695}
}

Model Card Contact

Jiacheng Liu