task_categories:question-answering task_categories:text-retrieval task_ids:open-domain-qa task_ids:document-retrieval annotations_creators:found annotations_creators:machine-generated size_categories:1M<n<10M language:pl license:cc-by-sa-4.0 arxiv:2305.05486 arxiv:2309.08469 region:us
MAUPQA is a collection of datasets for Polish Open-domain Question Answering.
@inproceedings{rybak-2023-maupqa,
    title = "{MAUPQA}: Massive Automatically-created {P}olish Question Answering Dataset",
    author = "Rybak, Piotr",
    booktitle = "Proceedings of the 9th Workshop on Slavic Natural Language Processing 2023 (SlavicNLP 2023)",
    month = may,
    year = "2023",
    address = "Dubrovnik, Croatia",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.bsnlp-1.2",
    pages = "11--16",
    abstract = "Recently, open-domain question answering systems have begun to rely heavily on annotated datasets to train neural passage retrievers. However, manually annotating such datasets is both difficult and time-consuming, which limits their availability for less popular languages. In this work, we experiment with several methods for automatically collecting weakly labeled datasets and show how they affect the performance of the neural passage retrieval models. As a result of our work, we publish the MAUPQA dataset, consisting of nearly 400,000 question-passage pairs for Polish, as well as the HerBERT-QA neural retriever.",
}
---
task_categories:
- question-answering
- text-retrieval
task_ids:
- open-domain-qa
- document-retrieval
language:
- pl
pretty_name: MAUPQA
size_categories:
- 1M<n<10M
annotations_creators:
- found
- machine-generated
license: cc-by-sa-4.0
---

# Dataset Card for MAUPQA Dataset

## Dataset Description

- **Paper:** [MAUPQA: Massive Automatically-created Polish Question Answering Dataset](https://arxiv.org/abs/2305.05486), [SilverRetriever: Advancing Neural Passage Retrieval for Polish Question Answering](https://arxiv.org/abs/2309.08469)
- **Point of Contact:** [Piotr Rybak](mailto:piotr.cezary.rybak@gmail.com)


### Dataset Summary

MAUPQA is a collection of 14 datasets for Polish document retrieval. Most of the datasets are either machine-generated or machine-translated from English. Across all datasets, it consists of over 1M questions, 1M positive, and 7M hard-negative question-passage pairs.

### Supported Tasks and Leaderboards

- `document-retrieval`: The dataset can be used to train a model for document retrieval. Success on this task is typically measured by [top-k retrieval accuracy](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.top_k_accuracy_score.html) or [NDCG](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.ndcg_score.html).

### Languages

The text is in Polish, as spoken by the [Internet users](https://github.com/facebookresearch/cc_net), [Polish Wikipedia](https://pl.wikipedia.org/) editors, or is an output of generative or translation models. The BCP-47 code for Polish is pl-PL.

## Dataset Structure

### Data Instances

The dataset consists of over 8 million question-passage pairs. For each instance, there is a `question`, a passage (`passage_title`, `passage_text`), and a boolean indicator if the passage is `relevant` for the given question (i.e. does it contain the answers).

For a small subset of `question` there is also a list of possible `answers` formulated in a natural language, in a way a Polish
speaker would answer the questions.

```
{
  'question_id': 1,
  'question': 'Na którym kontynencie leży państwo Gujana, panie Krzysztofie?',
  'answers': "['W Ameryce Południowej']",
  'passage_title': 'Gujana (ujednoznacznienie)',
  'passage_text': 'Gujana (region) – region Ameryki Południowej Gujana – państwo w Ameryce Południowej Gujana Brytyjska – dawna kolonia brytyjska; obecnie państwo Gujana Gujana Francuska – departament zamorski Francji; dawniej kolonia francuska Gujana Holenderska – dawna kolonia holenderska; obecnie państwo Surinam',
  'relevant': True,
  'passage_source': 'crawling',
  'subset': '1z10'
}
```

### Data Fields

Question-passage pairs:

- `question_id`: an integer id of the question
- `question`: a string containing the question
- `passage_title`: a string containing the title of the Wikipedia article
- `passage_text`: a string containing the passage text as extracted by the human annotator
- `relevant`: a boolean flag representing whether a passage is relevant to the question (i.e. does it contain the answers)
- `annotated_by`: a string containing the name of the annotator who verified the relevance of the pair
- `answers`: a string containing a list of possible short answers to the question
- `passage_source`: a string containing the method of obtaining the passage. One of the following:
  - `manual-annotation`: the question-passage pair was manually annotated
  - `crawling`: the question-passage pairs were created by taking advantage of the specific structure of crawled website
  - `dataset-translation`: the dataset was created by machine-translating the English dataset
  - `generative-model`: the question was created by the generative model based on the given passage
  - `bm25-negatives`: the passage was found by the BM25 retriever and scored using a multilingual cross-encoder to ensure it is not relevant
  - `bm25-positives`: the passage was found by the BM25 retriever and scored using a multilingual cross-encoder to ensure it is relevant
- `subset`: a string containing the name of the dataset

### Data Splits

MAUPQA is a collection of 14 datasets and most of them are weakly labeled. Therefore, the intended use of MAUPQA is for training only. As such, all examples belong to a single `train` split. We recommend using the [PolQA](https://huggingface.co/datasets/ipipan/polqa) dataset for evaluation.

Basic statistics of all 14 datasets:

| dataset           | # questions | # answers | # positive passages | # negative passages |
|-------------------|------------:|----------:|--------------------:|--------------------:|
| 1z10              |      22,835 |    21,415 |              22,014 |             139,471 |
| czy-wiesz-v2      |      29,078 |         - |              29,078 |             143,306 |
| gpt3-cc           |      10,146 |    10,146 |              10,177 |              89,203 |
| gpt3.5-cc         |      29,591 |    29,583 |              29,720 |             251,959 |
| gpt3.5-wiki       |      29,674 |    29,636 |              29,748 |             115,564 |
| mkqa              |       4,036 |     4,036 |               3,968 |              19,814 |
| mqa               |     172,768 |         - |             178,131 |           1,249,659 |
| msmarco           |     389,987 |         - |             416,763 |           3,006,996 |
| multilingual-NLI  |     100,752 |    64,900 |              68,096 |             743,857 |
| nq                |     135,781 |         - |             139,976 |             797,436 |
| poleval2021-pairs |       1,977 |         - |               2,088 |              17,608 |
| poquad            |      56,588 |    46,157 |              46,187 |             299,865 |
| templates         |      15,993 |    14,504 |              15,993 |              45,228 |
| wiki-def          |      18,093 |    18,092 |              18,093 |              84,956 |
| Total             |   1,017,299 |   238,469 |           1,010,032 |           7,004,922 |


## Dataset Creation

### Curation Rationale

Open-domain question answering systems rely heavily on annotated datasets to train neural document retrievers. However, manually annotating such datasets is both difficult and time-consuming. To overcome these difficulties, we experimented with several methods for automatically collecting weakly labeled datasets. As a result, MAUPQA enables the development of robust document retrieval systems for Polish.

### Source Data

#### Initial Data Collection and Normalization

Below, we briefly describe each dataset. For a detailed description please refer to the [paper](https://arxiv.org/abs/2305.05486).

* `1z10`: We transcribe 333 recordings of the [Jeden z Dziesięciu](https://pl.wikipedia.org/wiki/Jeden_z_dziesi%C4%99ciu) TV show using the Whisper model and extract the question-answer pairs using GPT-3.5 model. We use the BM25 retriever and the GPT-3.5-based cross-encoder to match questions with Wikipedia passages.
* `czy-wiesz-v2`:  We first crawl all questions from the [Did you know?](https://pl.wikipedia.org/wiki/Wikiprojekt:Czy_wiesz/archiwum) section on Polish Wikipedia together with a link to the relevant Wikipedia article. Then, we use the [multilingual cross-encoder](https://huggingface.co/unicamp-dl/mMiniLM-L6-v2-mmarco-v2) to choose the most relevant passage.
* `gpt3-cc`: We sample random passages from [CCNet](https://github.com/facebookresearch/cc_net) corpus and prompt GPT-3 to generate a relevant question. 
* `gpt3.5-cc`: We sample random passages from [CCNet](https://github.com/facebookresearch/cc_net) corpus and prompt GPT-3.5 to generate a relevant question. 
* `gpt3.5-wiki`: We sample random passages from Polish Wikipedia and prompt GPT-3.5 to generate a relevant question. 
* `mkqa`: We clean the Polish subset of the [MKQA](https://huggingface.co/datasets/mkqa) dataset by removing questions without answers, requiring long answers (*Why?* and *How?* questions), and ambiguous ones ("Who is the *current* president?*). We use the BM25 retriever and the [multilingual cross-encoder](https://huggingface.co/unicamp-dl/mMiniLM-L6-v2-mmarco-v2) to choose the most relevant passage.
* `mqa`: We clean the Polish subset of the [MQA](https://huggingface.co/datasets/clips/mqa) dataset by removing artificially created questions like "What is the best hotel in *{city}*?" for hundreds of different *cities*. To clean the dataset, we cluster lexically similar questions/passages and remove clusters with over 5 questions.
* `msmarco`: We translate the [MS MARCO](https://huggingface.co/datasets/ms_marco) dataset into Polish using the machine translation model.
* `multilingual-NLI`: We extract question-answer pairs from the Polish subset of the [multilingual-NLI](https://huggingface.co/datasets/MoritzLaurer/multilingual-NLI-26lang-2mil7) dataset. We create questions using the following template: "Czy *{premise}*?" (Eng. "Does *{premise}*?") and use hypotheses as passages. We consider `entailment` and `contradiction` labels as relevant and `neutral` as negative.
* `nq`: We translate the [NQ](https://huggingface.co/datasets/natural_questions) dataset into Polish using the machine translation model.
* `poleval2021-pairs`: We take [allegro/polish-question-passage-pairs](https://huggingface.co/datasets/allegro/polish-question-passage-pairs) without any changes.
* `poquad`: We extract question-passages pairs from the training split of the [PoQuAD](https://huggingface.co/datasets/clarin-pl/poquad) dataset.
* `templates`: We take advantage of the Wikipedia structure to generate questions using predefined templates. For example, list pages group together similar entities (e.g. "Writers born in Poland") which allow generating questions like "Where was *{writer name}* born?". In total, we use 33 templates to generate questions. We use the [multilingual cross-encoder](https://huggingface.co/unicamp-dl/mMiniLM-L6-v2-mmarco-v2) to choose the most relevant passage from the linked article.
* `wiki-def`: We use [Wiktionary](https://www.wiktionary.org/) to generate questions based on word definitions. We use definitions that have links to Wikipedia articles to create the question-passage pairs. For example, the definition of "Monday" is "the first day of the week". Based on it, we generate the question "What is the name of *the first day of the week*?".

Additionally, we extend each dataset by sampling the hard negative passages using a BM25 retriever and score using a [multilingual cross-encoder](https://huggingface.co/unicamp-dl/mMiniLM-L6-v2-mmarco-v2) to ensure that passages are not relevant.

#### Who are the source language producers?

The text is in Polish, as spoken by the [Internet users](https://github.com/facebookresearch/cc_net), [Polish Wikipedia](https://pl.wikipedia.org/) editors, or is an output of generative or translation models.

### Annotations

#### Annotation process

The MAUPQA dataset doesn't provide additional annotation except for the annotation present in the source datasets.

#### Who are the annotators?

Please refer to the description of the source datasets.

### Personal and Sensitive Information

The dataset should not contain any personal or sensitive information. However, we use the [CCNet](https://github.com/facebookresearch/cc_net) dataset as a source of passages that we didn't manually inspect for personal and sensitive information.

## Considerations for Using the Data

### Social Impact of Dataset

This dataset was created to promote the research in the open-domain question answering for Polish and allow developing question answering systems.

### Discussion of Biases

The machine-translated datasets might not represent the natural language as used by native Polish speakers. Similarly, the questions generated by the generative models might not be representative or correct.

Most of the question-passage pairs are created automatically using the BM25 retriever and as such it is biased to lexically similar pairs.

### Other Known Limitations

The MAUPQA dataset is mostly automatically generated and can therefore contain a high proportion of noise and incorrectly labeled question-passage pairs.

## Additional Information

### Dataset Curators

The MAUPQA dataset was collected by Piotr Rybak and Maciej Ogrodniczuk from the [Institute of Computer Science, Polish Academy of Sciences](http://zil.ipipan.waw.pl/) but the source datasets were created by many more researchers. Please refer to the original dataset descriptions for the full authorship.

This work was supported by the European Regional Development Fund as a part of 2014–2020 Smart Growth Operational Programme, CLARIN — Common Language Resources and Technology Infrastructure, project no. POIR.04.02.00-00C002/19.

### Licensing Information

CC BY-SA 4.0

### Citation Information

```
@inproceedings{rybak-2023-maupqa,
    title = "{MAUPQA}: Massive Automatically-created {P}olish Question Answering Dataset",
    author = "Rybak, Piotr",
    booktitle = "Proceedings of the 9th Workshop on Slavic Natural Language Processing 2023 (SlavicNLP 2023)",
    month = may,
    year = "2023",
    address = "Dubrovnik, Croatia",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.bsnlp-1.2",
    pages = "11--16",
    abstract = "Recently, open-domain question answering systems have begun to rely heavily on annotated datasets to train neural passage retrievers. However, manually annotating such datasets is both difficult and time-consuming, which limits their availability for less popular languages. In this work, we experiment with several methods for automatically collecting weakly labeled datasets and show how they affect the performance of the neural passage retrieval models. As a result of our work, we publish the MAUPQA dataset, consisting of nearly 400,000 question-passage pairs for Polish, as well as the HerBERT-QA neural retriever.",
}
```

```
@misc{rybak2023silverretriever,
    title={SilverRetriever: Advancing Neural Passage Retrieval for Polish Question Answering}, 
    author={Piotr Rybak and Maciej Ogrodniczuk},
    year={2023},
    eprint={2309.08469},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}
```