arxiv:2310.00752 region:us
---
configs:
- config_name: train
  data_files:
  - split: train_real_world
    path:
    - "data/new_real_world.json"
  - split: train_synthetic
    path:
    - "data/new_synthetic.json"
  - split: train_mix
    path:
    - "data/new_mix.json"
---


## MetricInstruct

We present TIGERScore, a **T**rained metric that follows **I**nstruction **G**uidance to perform **E**xplainable, and **R**eference-free evaluation over a wide spectrum of text generation tasks. TIGERScore is guided by the natural language instruction to provide error analysis to pinpoint the mistakes in the generated text. Our metric is based on LLaMA-2, trained on our meticulously curated instruction-tuning dataset MetricInstruct which covers 6 text generation tasks and 23 text generation datasets. The dataset consists of 48K quadruple in the form of (instruction, input, system output, error analysis). We collected the `system outputs' through diverse channels to cover different types of errors.


Project Page: [https://tiger-ai-lab.github.io/TIGERScore/](https://tiger-ai-lab.github.io/TIGERScore/)

Paper: [https://arxiv.org/abs/2310.00752](https://arxiv.org/abs/2310.00752)

Code: [https://github.com/TIGER-AI-Lab/TIGERScore](https://github.com/TIGER-AI-Lab/TIGERScore)

Demo: [https://huggingface.co/spaces/TIGER-Lab/TIGERScore](https://huggingface.co/spaces/TIGER-Lab/TIGERScore)

TIGERScore-7B-V1.0: [https://huggingface.co/TIGER-Lab/TIGERScore-7B-V1.0](https://huggingface.co/TIGER-Lab/TIGERScore-7B-V1.0)

TIGERScore-13B-V1.0: [https://huggingface.co/TIGER-Lab/TIGERScore-13B-V1.0](https://huggingface.co/TIGER-Lab/TIGERScore-13B-V1.0)


We present the MetricInstruct dataset, which is employed to fine-tune TIGERScore. The three underlying criteria for dataset construction are:

1. Dataset diversity: we choose 23 distinctive datasets as the source context to cover enough generation tasks.
2. Error coverage: we take system outputs generated from 50+ text generation systems to cover all types of errors and guarantee a balanced distribution.
3. Quality ensurance: to ensure MetricInstruct is tailored to gather in-depth error analysis, we sourced it by prompting OpenAI GPT models and then filtered through different heuristics to eliminate low-quality error analysis.

## Data Source

Our system outputs come from two channels, namely real-world system outputs and synthetic outputs. The real-world system outputs are obtained from real systems, which ensures the error distribution is aligned with real-world ones.
Check out our paper for more details.

| Task     |  Real-World Dataset                       |  Output Source |  Synthetic Dataset                  |  Output Source |
|:--------:|:-----------------------------------------:|:--------------:|:-----------------------------------:|:--------------:|
| Summarization     | SummEval, XSum,Newsroom,SAMSum            | 27 Systems     | CNN/DM, XSum,Gigaword,SAMSum        | GPT-4          |
| Translation    | WMT                                       | 18 Systems     | WMT                                 | GPT-4          |
| Data-to-Text      | WebNLG-2020,WikiTableText,ToTTo           | 17 Systems     | WikiTableText,Dart,ToTTo            | GPT-4          |
| Long-Form QA    | ASQA,FeTaQA,CosmosQA,ELI5                 | 5 Systems      | ASQA,FeTaQACosmos QA,ELI5           | GPT-4          |
| MathQA   | GSM8K                                     | 5 Systems      | GSM8K,MathQA                        | GPT-4          |
| Instruct | MixInstruct                               | 11 Systems     | LIMA,AlpacaFarmOASST1,Guanaco,Dolly | GPT-4          |


## Data Format
The dataset consists of 48K quadruple in the form of (instruction, input, system output, error analysis).
For each item in the dataset, `task` represents its corresponding text generation task, `instruction` is its task instruction, `input_context` is its input source, and `hypo_output` is the generated output, and `errors` is the error analysis given by ChatGPT or GPT-4.

## Formatting
To format the data fields into a single prompt for finetuning or testing, We provide the following code for users to refer:
```python
FINETUNE_INST = "You are evaluating errors in a model-generated output for a(an) ${task} task."
FINETUNE_INPUT = """\
Task instruction: ${generation_instruction}
Source: ${input_context}
Model-generated Output: ${hypothesis_output}

Based on the given task instruction and source, identify errors in this model-generated output.
For each error you give in the response, please also elaborate the following information:
- error location (the words that are wrong in the output)
- error aspect it belongs to.
- explanation why it's an error, and the correction suggestions.
- severity of the error ("Major" or "Minor"). 
- reduction of score (between 0.5 and 5 given the severity of the error)

Your evaluation output:
"""
inst_part = Template(FINETUNE_INST)
inst_part = inst_part.substitute(task=task)
input_part = Template(FINETUNE_INPUT)
input_part = input_part.substitute(
    generation_instruction=instruction,
    input_context=input_context,
    hypothesis_output=hypo_output
)
prompt = (inst_part + "\n" + input_part).strip("\n ") + "\n"
encodings = tigerscore_tokenizer(prompt, return_tensors="pt")
input_ids = encodings["input_ids"].to(tigerscore_model.device)
attention_mask = encodings["attention_mask"].to(tigerscore_model.device)
```

Example of formatted prompt:
```txt
You are evaluating errors in a model-generated output for a(an) translation task.
Task instruction: Translate the following text from German to English.
Source: Der künftige EM-Cheforganisator Philipp Lahm soll laut Grindel im DFB-Präsidium mitarbeiten.
Model-generated Output: According to Grindel, the future head of the European Championships, Philipp Lahm, is to participate in the DFB Presidency.

Based on the given task instruction and source, identify errors in this model-generated output.
For each error you give in the response, please also elaborate the following information:
- error location (the words that are wrong in the output)
- error aspect it belongs to.
- explanation why it's an error, and the correction suggestions.
- severity of the error ("Major" or "Minor"). 
- reduction of score (between 0.5 and 5 given the severity of the error)

Your evaluation output:
```
## Citation

```
@article{jiang2023TIGERScore,
  title={TIGERScore: Towards Building Explainable Metric for All Text Generation Tasks},
  author={Dongfu Jiang, Yishan Li, Ge Zhang, Wenhao Huang, Bill Yuchen Lin, Wenhu Chen},
  journal={arXiv preprint arXiv:2310.00752},
  year={2023}
}
```