task_categories:question-answering size_categories:n<1K language:en license:cc-by-4.0 legal region:us
---
language:
- en
license: cc-by-4.0
size_categories:
- n<1K
task_categories:
- question-answering
pretty_name: craqan_v1
tags:
- legal
dataset_info:
  features:
  - name: title
    dtype: string
  - name: article
    dtype: string
  - name: article_titles
    sequence: string
  - name: article_sections
    sequence: string
  - name: section
    dtype: string
  - name: section_index
    dtype: int64
  - name: section_sentences
    dtype: string
  - name: question
    dtype: string
  - name: answer
    dtype: string
  - name: sentences_required
    sequence: int64
  - name: url
    dtype: string
  - name: time_downloaded
    dtype: string
  splits:
  - name: train
    num_bytes: 17788270
    num_examples: 263
  download_size: 0
  dataset_size: 17788270
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
---

# Coreference Resolution in Question Answering (CRaQAn)

250+ question-answer pairs that require coreference resolution across sentences from selected Wikipedia passages.

## Generation Process

Given the relative complexity of our task (coreference resolution across passages for question-answering), we aimed 
to avoid crowd-sourcing this dataset and instead focused on using LLMs to automate our process. Every question-answer
pair in the CRaQAn dataset was automatically generated using a Recursive Criticism and Improvement (RCI) loop. To 
accomplish our RCI loop, we wrote a GENERATOR prompt and several REVIEWER prompts, which can be found [here](https://huggingface.co/datasets/Edge-Pyxos/CRaQAn_v1/tree/main/generation_demo/prompts).

## Review Process

Every question-answer pair in the CRaQAn v1 dataset was reviewed by at least two human reviewers. We intend for this to be a
high-trust and high-quality dataset that can be used for various applications. Every human reviewer was given the
following criteria. For each QA pair:

1. The question is clear and not ambiguous with regards to the text.
2. The question is a single question, and not two separate or related questions joined by the word "and".
3. The question does not contain or assume any information outside of the required sentences.
4. The answer is correct and reasonably terse.
5. The question-answer pair must not rely on any information from outside the required sentences.
6. The question-answer pair relies on information from each of the required sentences.
7. The number of required sentences is 2 or 3.
8. The Markdown is correctly formatted.

## CRaQAn Usage

```python
from datasets import load_dataset
import pandas as pd
from IPython.display import display, Markdown

# Load dataset.
craqan = load_dataset("Edge-Pyxos/CRaQAn_v1", split = "train")
df = pd.DataFrame(craqan)

# Fix issue with section_sentences that happens during Huggingface conversion.
df["section_sentences"] = df["section_sentences"].apply(json.loads)

# Visualize a sample from the dataset.
row = df.sample(1).squeeze()
sentences = ""
for idx, s in enumerate(row.section_sentences):
    sentences += (" <mark> " + s["sentence"] + " </mark> ") if idx in row.sentences_required else " " + s["sentence"]

display(Markdown(f"# Article: {row.title}"))
display(Markdown(row.article_titles[row.section_index]))
display(Markdown(f"*Required Sentences: {row.sentences_required}*"))
display(Markdown(sentences))
display(Markdown(f"**Question**: " + row.question))
display(Markdown("**Answer**: " + row.answer))
display(Markdown("-------------------"))
```

## Demo Usage

We provide all prompts, code, and processes used to generate the CRaQAn-v1 dataset in our [demo notebook](https://huggingface.co/datasets/Edge-Pyxos/CRaQAn_v1/blob/main/generation_demo/create_dataset.ipynb).
