size_categories:n<1K license:apache-2.0 region:us
---
license: apache-2.0
size_categories:
- n<1K
ArXiv: 2306.04757
---

# Project Links


# Dataset Description
The IMPACT dataset contains 50 human created prompts for each category, 200 in total, to test LLMs general writing ability.


Instructed LLMs demonstrate promising ability in writing-based tasks, such as composing letters or ethical debates. This dataset consists prompts across 4 diverse usage scenarios:
- **Informative Writing**: User queries such as self-help advice or explanations for various concept
- **Professional Writing**: Format involves suggestions presentations or emails in a business setting
- **Argumentative Writing**: Debate positions on ethical and societal question
- **Creative Writing**: Diverse writing formats such as stories, poems, and songs.

The IMPACT dataset is included in our [InstructEval Benchmark Suite](https://github.com/declare-lab/instruct-eval).

# Evaluation Results
We leverage ChatGPT to judge the quality of the generated answers by LLMs. In terms of:
- Relevance: how well the answer engages with the given prompt
- Coherence: general text quality such as organization and logical flow

Each answer is scored on a Likert scale from 1 to 5. We evaluate the models in the zero-shot
setting based on the given prompt and perform sampling-based decoding with a temperature of 1.0

| **Model** | **Size** | **Informative** |  | **Professional** |  | **Argumentative** |  | **Creative** |  | **Avg.** |  |
| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|  |  | Rel. | Coh. | Rel. | Coh. | Rel. | Coh. | Rel. | Coh. | Rel. | Coh. |
| **ChatGPT** | - | 3.34 | 3.98 | 3.88 | 3.96 | 3.96 | 3.82 | 3.92 | 3.94 | 3.78 | 3.93 |
| [**Flan-Alpaca**](https://huggingface.co/declare-lab/flan-alpaca-xxl) | 11B | 3.56 | 3.46 | 3.54 | 3.70 | 3.22 | 3.28 | 3.70 | 3.40 | 3.51 | 3.46 |
| [**Dolly-V2**](https://huggingface.co/databricks/dolly-v2-12b) | 12 B | 3.54 | 3.64 | 2.96 | 3.74 | 3.66 | 3.20 | 3.02 | 3.18 | 3.30 | 3.44 |
| [**StableVicuna**](https://huggingface.co/TheBloke/stable-vicuna-13B-HF) | 13B | 3.54 | 3.64 | 2.96 | 3.74 | 3.30 | 3.20 | 3.02 | 3.18 | 3.21 | 3.44 |
| [**Flan-T5**](https://huggingface.co/google/flan-t5-xxl) | 11B | 2.64 | 3.24 | 2.62 | 3.22 | 2.54 | 3.40 | 2.50 | 2.72 | 2.58 | 3.15 |

# Citation
Please consider citing the following article if you found our work useful:

```
bibtex
@article{chia2023instructeval,
      title={INSTRUCTEVAL: Towards Holistic Evaluation of Instruction-Tuned Large Language Models}, 
      author={Yew Ken Chia and Pengfei Hong and Lidong Bing and Soujanya Poria},
      journal={arXiv preprint arXiv:2306.04757},
      year={2023}
}
```

