from dataclasses import dataclass
from enum import Enum
@dataclass
class Task:
benchmark: str
metric: str
col_name: str
class Tasks(Enum):
# task_key in the json file, metric_key in the json file, name to display in the leaderboard
hallucination_rate = Task("hallucination_rate",
"hallucination_rate", "Humanlike Score (%)")
# factual_consistency_rate = Task("factual_consistency_rate", "factual_consistency_rate", "Factual Consistency Rate (%)")
# answer_rate = Task("answer_rate", "answer_rate", "Answer Rate (%)")
# average_summary_length = Task("average_summary_length",
# "average_summary_length", "Average Summary Length")
# Your leaderboard name
TITLE = """Humanlike Evaluation Model (HEM) leaderboard
"""
# What does your leaderboard evaluate?
INTRODUCTION_TEXT = """
This leaderboard (by [Vectara](https://vectara.com)) evaluates how often an LLM introduces hallucinations when summarizing a document.
The leaderboard utilizes [HHEM](https://huggingface.co/vectara/hallucination_evaluation_model), an open source hallucination detection model.
An improved version (HHEM v2) is integrated into the [Vectara platform](https://console.vectara.com/signup/?utm_source=huggingface&utm_medium=space&utm_term=integration&utm_content=console&utm_campaign=huggingface-space-integration-console).
"""
# Which evaluations are you running? how can people reproduce what you have?
LLM_BENCHMARKS_TEXT = """
## Introduction
The Hughes Hallucination Evaluation Model (HHEM) Leaderboard is dedicated to assessing the frequency of hallucinations in document summaries generated by Large Language Models (LLMs).
Hallucinations refer to instances where a model introduces factually incorrect or unrelated content in its summaries.
## How it works
Using [Vectara](https://vectara.com)'s HHEM, we measure the occurrence of hallucinations in generated summaries.
Given a source document and a summary generated by an LLM, HHEM outputs a hallucination score between 0 and 1, with 0 indicating complete hallucination and 1 representing perfect factual consistency.
The model card for HHEM can be found [here](https://huggingface.co/vectara/hallucination_evaluation_model).
## Evaluation Dataset
Our evaluation dataset consists of 1006 documents from multiple public datasets, primarily [CNN/Daily Mail Corpus](https://huggingface.co/datasets/cnn_dailymail/viewer/1.0.0/test).
We generate summaries for each of these documents using submitted LLMs and compute hallucination scores for each pair of document and generated summary. (Check the prompt we used [here](https://huggingface.co/spaces/vectara/Hallucination-evaluation-leaderboard))
## Metrics Explained
- Hallucination Rate: Percentage of summaries with a hallucination score below 0.5
- Factual Consistency Rate: The complement of the hallucination rate, expressed as a percentage.
- Answer Rate: Percentage of summaries that are non-empty. This is either the model refuses to generate a response or throws an error due to various reasons. (e.g. the model believes that the document includes inappropriate content)
- Average Summary Length: The average word count of generated summaries
## Note on non-Hugging Face models
On HHEM leaderboard, There are currently models such as GPT variants that are not available on the Hugging Face model hub. We ran the evaluations for these models on our own and uploaded the results to the leaderboard.
If you would like to submit your model that is not available on the Hugging Face model hub, please contact us at minseok@vectara.com.
## Model Submissions and Reproducibility
You can submit your model for evaluation, whether it's hosted on the Hugging Face model hub or not. (Though it is recommended to host your model on the Hugging Face)
### For models not available on the Hugging Face model hub:
1) Access generated summaries used for evaluation [here](https://github.com/vectara/hallucination-leaderboard) in "leaderboard_summaries.csv".
2) The text generation prompt is available under "Prompt Used" section in the repository's README.
3) Details on API Integration for evaluations are under "API Integration Details".
### For models available on the Hugging Face model hub:
To replicate the evaluation result for a Hugging Face model:
1) Clone the Repository
```python
git lfs install
git clone https://huggingface.co/spaces/vectara/leaderboard
```
2) Install the Requirements
```python
pip install -r requirements.txt
```
3) Set Up Your Hugging Face Token
```python
export HF_TOKEN=your_token
```
4) Run the Evaluation Script
```python
python main_backend.py --model your_model_id --precision float16
```
5) Check Results
After the evaluation, results are saved in "eval-results-bk/your_model_id/results.json".
## Results Format
The results are structured in JSON as follows:
```python
{
"config": {
"model_dtype": "float16",
"model_name": "your_model_id",
"model_sha": "main"
},
"results": {
"hallucination_rate": {
"hallucination_rate": ...
},
"factual_consistency_rate": {
"factual_consistency_rate": ...
},
"answer_rate": {
"answer_rate": ...
},
"average_summary_length": {
"average_summary_length": ...
}
}
}
```
For additional queries or model submissions, please contact minseok@vectara.com.
"""
EVALUATION_QUEUE_TEXT = """
## Some good practices before submitting a model
### 1) Make sure you can load your model and tokenizer using AutoClasses:
```python
from transformers import AutoConfig, AutoModel, AutoTokenizer
config = AutoConfig.from_pretrained("your model name", revision=revision)
model = AutoModel.from_pretrained("your model name", revision=revision)
tokenizer = AutoTokenizer.from_pretrained("your model name", revision=revision)
```
If this step fails, follow the error messages to debug your model before submitting it. It's likely your model has been improperly uploaded.
Note: make sure your model is public!
Note: if your model needs `use_remote_code=True`, we do not support this option yet but we are working on adding it, stay posted!
### 2) Convert your model weights to [safetensors](https://huggingface.co/docs/safetensors/index)
It's a new format for storing weights which is safer and faster to load and use. It will also allow us to add the number of parameters of your model to the `Extended Viewer`!
### 3) Make sure your model has an open license!
This is a leaderboard for Open LLMs, and we'd love for as many people as possible to know they can use your model 🤗
### 4) Fill up your model card
When we add extra information about models to the leaderboard, it will be automatically taken from the model card
## In case of model failure
If your model is displayed in the `FAILED` category, its execution stopped.
Make sure you have followed the above steps first.
"""
CITATION_BUTTON_LABEL = "Copy the following snippet to cite these results"
CITATION_BUTTON_TEXT = r"""
@dataset{HughesBae2023,
author = {Simon Hughes and Minseok Bae},
title = {Vectara Hallucination Leaderboard},
year = {2023},
month = {11},
publisher = {Vectara, Inc},
doi = {},
url = {https://github.com/vectara/hallucination-leaderboard},
abstract = {A leaderboard comparing LLM performance at maintaining factual consistency when summarizing a set of facts.},
keywords = {nlp, llm, hallucination, nli, machine learning},
license = {Apache-2.0},
}"""