task_categories:question-answering annotations_creators:expert-generated size_categories:1K<n<10K source_datasets:original language:en license:mit arxiv:2309.07852 region:us
---
configs:
- config_name: main
  data_files: r2_compiled_anon_fixed.jsonl
- config_name: lfqa_random
  data_files:
  - split: train
    path: rand_lfqa_train.json
  - split: test
    path: rand_lfqa_test.json
  - split: validation
    path: rand_lfqa_val.json
- config_name: lfqa_domain
  data_files:
  - split: train
    path: domain_lfqa_train.json
  - split: test
    path: domain_lfqa_test.json
  - split: validation
    path: domain_lfqa_val.json
license: mit
task_categories:
- question-answering
language:
- en
source_datasets:
- original
pretty_name: ExpertQA
annotations_creators:
- expert-generated
size_categories:
- 1K<n<10K
---

# Dataset Card for ExpertQA

## Dataset Description

- **Repository: https://github.com/chaitanyamalaviya/ExpertQA** 
- **Paper: https://arxiv.org/pdf/2309.07852** 
- **Point of Contact: chaitanyamalaviya@gmail.com** 

### Dataset Summary

We provide here the data accompanying the paper: [ExpertQA: Expert-Curated Questions and Attributed Answers](https://arxiv.org/pdf/2309.07852). The ExpertQA dataset contains 2177 examples from 32 different fields.

### Supported Tasks

The `main` data contains 2177 examples that can be used to evaluate new methods for estimating factuality and attribution, while the `lfqa_domain` and `lfqa_rand` data can be used to evaluate long-form question answering systems.

## Dataset Creation 

### Curation Rationale

ExpertQA was created to evaluate factuality & attribution in language model responses to domain-specific questions, as well as evaluate long-form question answering in domain-specific settings.

### Annotation Process

Questions in ExpertQA were formulated by experts spanning 32 fields. The answers to these questions are expert-verified, model-generated answers to these questions. Each claim-evidence pair in an answer is judged by experts for various properties such as the claim’s informativeness, factuality, citeworthiness, whether the claim is supported by the evidence, and reliability of the evidence source. Further, experts revise the original claims to ensure they are factual and supported by trustworthy sources.

## Dataset Structure

### Data Instances

We provide the main data, with judgements of factuality and attribution, under the `default` subset. 
The long-form QA data splits are provided at `lfqa_domain` (domain split) and `lfqa_rand` (random split).
Additional files are provided in our [GitHub repo](https://github.com/chaitanyamalaviya/ExpertQA).

### Data Fields

The main data file contains newline-separated json dictionaries with the following fields:
* `question` - Question written by an expert.
* `annotator_id` - Anonymized annotator ID of the author of the question.
* `answers` - Dict mapping model names to an Answer object. The model names can be one of `{gpt4, bing_chat, rr_sphere_gpt4, rr_gs_gpt4, post_hoc_sphere_gpt4, post_hoc_gs_gpt4}`. 
* `metadata` - A dictionary with the following fields:
    * `question_type` - The question type(s) separated by "|".
    * `field` - The field to which the annotator belonged.
    * `specific_field` - More specific field name within the broader field.

Each Answer object contains the following fields:
* `answer_string`: The answer string.
* `attribution`: List of evidences for the answer (not linked to specific claims). Note that these are only URLs, the evidence passages are stored in the Claim object -- see below.
* `claims`: List of Claim objects for the answer.
* `revised_answer_string`: Revised answer by annotator.
* `usefulness`: Usefulness of original answer marked by annotator.
* `annotation_time`: Time taken for annotating this answer.
* `annotator_id`: Anonymized annotator ID of the person who validated this answer.

Each Claim object contains the following fields:
* `claim_string`: Original claim string.
* `evidence`: List of evidences for the claim (URL+passage or URL).
* `support`: Attribution marked by annotator.
* `reason_missing_support`: Reason for missing support specified by annotator.
* `informativeness`: Informativeness of claim for the question, marked by annotator.
* `worthiness`: Worthiness of citing claim marked by annotator.
* `correctness`: Factual correctness of claim marked by annotator.
* `reliability`: Reliability of source evidence marked by annotator.
* `revised_claim`: Revised claim by annotator.
* `revised_evidence`: Revised evidence by annotator.


### Citation Information

```
@inproceedings{malaviya23expertqa,
    title = {ExpertQA: Expert-Curated Questions and Attributed Answers},
    author = {Chaitanya Malaviya and Subin Lee and Sihao Chen and Elizabeth Sieber and Mark Yatskar and Dan Roth},
    booktitle = {arXiv},
    month = {September},
    year = {2023},
    url = "https://arxiv.org/abs/2309.07852"
}
```
