arxiv:2302.07738 arxiv:2109.12870 arxiv:2304.07327 region:us
# Usage
```
from datasets import load_dataset
dataset = load_dataset("lbourdois/Aya", sep=";")
```

# Overall statistics (47,023 questions)

Tasks covered:
- faq: 16668 (35.45%) (if faq is considered open_qa, then 48.30% of data is open_qa)
- open_qa: 6044 (12.85%)
- mrc: 900 (1.91%)
- qam: 5000 (10.63%)
- mcqa: 2483 (5.28%)

- text generation: 8668 (18,43%)  
Details:  
  - context_generation_with_answer: 528 (1.12%)
  - context_generation_with_question: 528 (1.12%)
  - context_generation_with_answer_and_question: 528 (1.12%)
  - question_generation_with_answer: 484 (1.03%)
  - question_generation_with_context: 528 (1.12%)
  - question_generation_with_answer_and_context: 462 (0.98%)
  - paraphrase_generation: 528 (1.12%)
  - text_generation_from_an_article: 528 (1.12%)
  - text_generation_from_title_of_an_article: 594 (1.26%)
  - text_generation_from_title_of_a_review: 792 (1.68%)
  - binary_text_generation_from_title_review: 792 (1.68%)
  - title_generation_from_an_article: 418 (0.89%)
  - title_generation_from_a_review: 396 (0.84%)
  - data_to_text: 660 (1.40%)
  - textual_merging: 462 (0.98%)
  - textual_simplification: 440 (0.94%)

- classification: 3190 (6.78%)  
Details:    
  - intent_classification: 660 (1.40%)
  - sentiment_analysis: 616 (1.31%)
  - star_classification: 484 (1.03%)
  - textual_entailment: 484 (1.03%)
  - paraphrase_detection: 484 (1.03%)
  - classes_classification: 462 (0.98%)
- token classification: 1386 (2.95%)  
Details:    
  - ner: 462 (0.98%)
  - pos: 462 (0.98%)
  - keywords_extraction: 462 (0.98%)
- coreference: 220 (0.47%)
- fill_mask: 528 (1.12%)
- summarization: 616 (1.31%)
- sentence_similarity: 396 (0.84%)
- question answering: 924 (1.96%)  
Details:  
  - qa: 264 (0.56%)    (SQuAD 1.0 format)
  - qa_v2: 660 (1.40%) (SQuAD 2.0 format)

Subjects covered:
- unknow: 24872 (52.89%)
- movie: 12346 (26.26%)]
- weather_information: 2845 (6.05%)
- pharmacy: 2483 (5.28%)
- medical: 928 (1.97%)
- math: 921 (1.96%)
- french: 794 (1.69%)
- science: 633 (1.35%)
- socio-demographic: 519 (1.10%)
- history: 289 (0.61%)
- chemistry: 152 (0.32%)
- physics: 115 (0.24%)
- geography: 51 (0.11%) 
- contemporary_world: 26 (0.06%)
- home_tips: 23 (0.05%)
- financial_ed: 19 (0.04%)
- animal: 7 (0.04%)

### Note to the Aya team
Possibility of sub-sampling (movies may be over-represented, for example) to balance classes within French, but also in relation to other languages if 47,023 data (+ those already in Aya) are too large.  
Details by dataset are available below.



# Statistics by dataset

## antoinelb7/alloprof 

### Description
3000 questions from the [antoinelb7/alloprof](https://huggingface.co/datasets/antoinelb7/alloprof) dataset by Lefebvre-Brossard et al. (2023), licensed under mit.

This dataset consists of questions posed by students aged 6 to 16, the answers to which have been verified on Allo Prof, a Quebec homework help website. The questions cover nine different educational subjects: mathematics, french, science, history, chemistry, physics, geography, contemporary_world, financial_ed.

The distribution of questions in the dataset is as follows:
- mathematics: 921 (30.70%)
- french: 794 (26.46%)
- science: 633 (21.10%)
- history: 289 (9.63%)
- chemistry: 152 (5.06%)
- physics: 115 (3.83%)
- geography: 51 (1.70%)
- contemporary_world: 26 (0.86%)
- financial_ed: 19 (0.63%)

The 3,000 lines were obtained after cleaning (deletion of questions requiring an image to be understood, deletion of URL links to prevent the model from generating false URLs, deletion of questions requiring a previous question not available to be understood, deletion of user pseudonyms, etc.).

As the questions are taken from a Quebec site, the data is consequently oriented. That is, the French used is [québécois]( https://en.wikipedia.org/wiki/Quebec_French) and the questions are based on the Quebec educational program (for example, the history questions focus mainly on the history of Quebec).

### Note to the Aya team

I've reread all 3,000 lines (both questions and answers), and they seem to be of high quality (the questions may contain typos due to the fact that they were asked by students, and the answers by adults who are teachers or members of the Allo Prof team). However, with so many lines, it's possible that there may have been some omissions in the margins.

### Citation

> @misc{lef23,
  doi = {10.48550/ARXIV.2302.07738},
  url = {https://arxiv.org/abs/2302.07738},
  author = {Lefebvre-Brossard, Antoine and Gazaille, Stephane and Desmarais, Michel C.},
  keywords = {Computation and Language (cs.CL), Information Retrieval (cs.IR), Machine Learning (cs.LG), FOS: Computer and information sciences, FOS: Computer and information sciences},
  title = {Alloprof: a new French question-answer education dataset and its use in an information retrieval case study},
  publisher = {arXiv},
  year = {2023},
  copyright = {Creative Commons Attribution Non Commercial Share Alike 4.0 International}
}



## facebook/belebele

### Description
900 questions from the [facebook/belebele](https://huggingface.co/datasets/facebook/belebele) dataset by Bandarkar et al. (2023) which is licensed under cc-by-sa-4.0.

Belebele is a multiple-choice machine reading comprehension (MRC) dataset.
Each question has four multiple-choice answers, only one of which is correct, and is linked to a short passage from the dataset [FLORES-200](https://github.com/facebookresearch/flores/tree/main/flores200).
The 900 questions cover 488 distinct passages and there are 1-2 associated questions for each.

The French text from FLORES-200 is a manual translation from English by native translators in each language. The Belebele questions were also created manually (and the answers double-checked).

It was not possible to specify the themes addressed in these questions. 

To adapt this dataset to Aya's expectations, a prompt was created including the passage plus a question taking up the four propositions in the form of bullet points.

### Note to the Aya team

As the dataset is multilingual (122 languages), it can be reused for languages other than French. 

### Citation

> @article{bandarkar2023belebele,
      title={The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants}, 
      author={Lucas Bandarkar and Davis Liang and Benjamin Muller and Mikel Artetxe and Satya Narayan Shukla and Donald Husa and Naman Goyal and Abhinandan Krishnan and Luke Zettlemoyer and Madian Khabsa},
      year={2023},
      journal={arXiv preprint arXiv:2308.16884}
}



## xglue/qam
### Description

5000 questions from the [xglue](https://huggingface.co/datasets/xglue) dataset (specifically the QAM split) by Liang et al. (2023) which is under a license not explicitly specified other than "the XGLUE datasets are intended for non-commercial research purposes only to promote advancement in the field of artificial intelligence and related areas, and is made available free of charge without extending any license or other intellectual property rights". 

xglue/qam is a Question Answering Matching dataset. The authors propose questions with answers that may or may not be relevant. Only relevant pairs of questions and answers have been kept.
The data is in native French, taken from a search engine (unspecified, but we can assume it's Bing, as it is thanked in the acknowledgements and the paper comes from a Microsoft team).

### Note to the Aya team
This dataset is also available in English and German.

### Citation

> @article{Liang2020XGLUEAN,
  title={XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation},
  author={Yaobo Liang and Nan Duan and Yeyun Gong and Ning Wu and Fenfei Guo and Weizhen Qi and Ming Gong and Linjun Shou and Daxin Jiang and Guihong Cao and Xiaodong Fan and Ruofei Zhang and Rahul Agrawal and Edward Cui and Sining Wei and Taroon Bharti and Ying Qiao and Jiun-Hung Chen and Winnie Wu and Shuguang Liu and Fan Yang and Daniel Campos and Rangan Majumder and Ming Zhou},
  journal={arXiv},
  year={2020},
  volume={abs/2004.01401}
}



## qanastek/frenchmedmcqa
### Description

2483 questions from the [qanastek/frenchmedmcqa](https://huggingface.co/datasets/qanastek/frenchmedmcqa) dataset by Labrak et al. (2023) which is licensed under cc-by-sa-4.0.

Frenchmedmcqa is a multiple-choice QA (MCQA) taken from real exams of the French medical specialization diploma in pharmacy. There are 5 possible answers each time, and there may be one or more correct answers.
To adapt this dataset to Aya's expectations, a prompt was created including the passage plus a question with the five propositions in the form of bullet points.

The dataset is split into 2171 train samples, 312 validation samples and 622 test samples. We have kept the train and validation data, but not the test, to avoid data leaks.

### Note to the Aya team

In the dataset, the training set is indicated by qanastek/frenchmedmcqa_train and the validation set by qanastek/frenchmedmcqa_validation.

### Citation

> @inproceedings{labrak-etal-2022-frenchmedmcqa,
    title = "{F}rench{M}ed{MCQA}: A {F}rench Multiple-Choice Question Answering Dataset for Medical domain",
    author = "Labrak, Yanis and Bazoge, Adrien and Dufour, Richard and       Daille, Beatrice and Gourraud, Pierre-Antoine and Morin, Emmanuel and Rouvier, Mickael",
    booktitle = "Proceedings of the 13th International Workshop on Health Text Mining and Information Analysis (LOUHI)",
    month = dec,
    year = "2022",
    address = "Abu Dhabi, United Arab Emirates (Hybrid)",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.louhi-1.5",
    pages = "41--46",
}


## Wikipedia:Oracle
### Description

2500 questions from [Wikipedia:Oracle](https://fr.wikipedia.org/wiki/Wikip%C3%A9dia:Oracle) which is a Wikipedia page for encyclopedic questions that have not been answered in Wikipedia or on the Internet, and where Wikipedia users try to provide an answer. As the data comes from Wikipedia, the license is CC-BY-SA 3.0.

The 2,500 pieces of data were obtained after cleaning (deletion of questions requiring an image to be understood, deletion of URL links to prevent the model from generating false URLs, deletion of questions requiring a previous question not available to be understood, deletion of user pseudonyms, etc.).

It was not possible to specify the themes covered by these questions. 

### Note to the Aya team

I've reread all 2,500 questions, but not the answers. It is therefore possible that some lines contain irrelevant answers (jokes by Wikipedia users rather than a serious response to the answer, for example).
A re-reading would therefore be relevant for this data.



## clips/mfaq
### Description

16668 questions from the [clips/mfaq](https://huggingface.co/datasets/clips/mfaq) dataset (more precisely the 'fr-faq' split) by De Bruyn et al. (2021) which is licensed under cc0-1.0.

MFAQ is a faq-like question/answer extraction from the [OSCAR](https://huggingface.co/datasets/oscar) dataset (2019 version by Ortiz Suarez et al.).

The French part originally contains more than 344391 questions/answers from 1395 different ones.
We have kept only 16668 of them, as we felt they were the most qualitative and relevant (the price of a plane to a destination from Paris in 2019, for example, is not very relevant).  

The questions selected come from 10 of the 1395 sites: pix-geeks.com, lemonde.fr, ou-et-quand.net, merckmanuals.com, jardinaddict.com, mariekrtonne.com, dansmaculotte.com, lesentimentparfait.fr and remedes-animaux.org.

The questions cover six different topics: movies, weather, medical, socio-demographic, home_tips and animal.

A few details:
- about the films: year of release, original name of the film title, director's name, main actors, type of film, whether there was a sequel, etc.
- on socio-demographic information: the number of inhabitants in France's main cities in 2020, the number of schools in the city, the distance between the city and the nearest emergency service.
- on weather_information: on average, in city X in April Y, the temperature is Z °C
- on medical: general medical information from the Merck company (no advertising for the company or any of its products, just questions like "X's symptoms are as follows", "Y's known causes are smoking, alcohol, etc.", invitations to consult a doctor for professional advice).
- on home tips: how to grow X, how to clean Y.
- on animals: any health problems they may have.

The distribution of questions in the dataset is as follows:
- movie: 12345 (74.064%)
- weather_information: 2846 (17.075%)
- medical: 928 (5.568%)
- socio-demographic: 519 (3.110%)
- home_tips: 23 (0.138%)
- animal: 7 (0.042%)

### Note to the Aya team
The mfaq dataset exists in 21 languages, it can be reused for languages other than French. 
However, not as it stands, as it would probably have to be sorted (for French, for example, only 10 sites out of 1395 have been kept).

### Citation

> @misc{debruyn2021mfaq,
      title={MFAQ: a Multilingual FAQ Dataset}, 
      author={Maxime {De Bruyn} and Ehsan Lotfi and Jeska Buhmann and Walter Daelemans},
      year={2021},
      eprint={2109.12870},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}


## OpenAssistant/oasst1
### Description

502 questions from the dataset [OpenAssistant/oasst1](https://huggingface.co/datasets/OpenAssistant/oasst) by Köpf et al (2023) under apache-2.0 license.

I have reread all 502 lines (both questions and answers) and they seem to me to be of good quality. However, with so many lines, there may have been some omissions at the margin.

It wasn't possible to specify the topics covered in the questions. 

### Note to the Aya team

Oasst1 is available in 34 languages other than French. However, some cleaning up is needed. On the one hand, the information provided is not necessarily reliable: for example, it is stated that there are 4251 messages for French, when in fact there are only 2474 (some data have been incorrectly tagged in French). In addition, not all messages have been proofread (for French, some answers to questions are "Idk bro" which are not relevant for training a model).

### Citation
> @misc{köpf2023openassistant,
      title={OpenAssistant Conversations -- Democratizing Large Language Model Alignment}, 
      author={Andreas Köpf and Yannic Kilcher and Dimitri von Rütte and Sotiris Anagnostidis and Zhi-Rui Tam and Keith Stevens and Abdullah Barhoum and Nguyen Minh Duc and Oliver Stanley and Richárd Nagyfi and Shahul ES and Sameer Suri and David Glushkov and Arnav Dantuluri and Andrew Maguire and Christoph Schuhmann and Huu Nguyen and Alexander Mattick},
      year={2023},
      eprint={2304.07327},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}



## DFP

### Description

15928 questions from the dataset [DFP](https://huggingface.co/datasets/CATIE-AQ/DFP) by Bourdois et al. (2023) under cc-by-sa-4.0 license.

The distribution of questions in the dataset is as follows:
- text generation: 8668 (54,42%)
Details:  
  - context_generation_with_answer: 528 (3.31%)
  - context_generation_with_question: 528 (3.31%)
  - context_generation_with_answer_and_question: 528 (3.31%)
  - question_generation_with_answer: 484 (3.04%)
  - question_generation_with_context: 528 (3.31%)
  - question_generation_with_answer_and_context: 462 (2.90%)
  - paraphrase_generation: 528 (3.31%)
  - text_generation_from_an_article: 528 (3.31%)
  - text_generation_from_title_of_an_article: 594 (3.73%)
  - text_generation_from_title_of_a_review: 792 (4.97%)
  - binary_text_generation_from_title_review: 792 (4.97%)
  - title_generation_from_an_article: 418 (2.62%)
  - title_generation_from_a_review: 396 (2.49%)
  - data_to_text: 660 (4.14%)
  - textual_merging: 462 (2.90%)
  - textual_simplification: 440 (2.76%)

- classification: 3190 (20.03%)
Details:  
  - intent_classification: 660 (4.14%)
  - sentiment_analysis: 616 (3.87%)
  - star_classification: 484 (3.039%)
  - textual_entailment: 484 (3.039%)
  - paraphrase_detection: 484 (3.039%)
  - classes_classification: 462 (2.90%)
- token classification: 1386 (8,70%)
Details:    
  - ner: 462 (2.90%)
  - pos: 462 (2.90%)
  - keywords_extraction: 462 (2.90%)
- coreference: 220 (1.38%)
- fill_mask: 528 (3.31%)
- summarization: 616 (3.87%)
- sentence_similarity: 396 (2.49%)
- question answering: 924 (5,80%)
Details:  
  - qa: 264 (1.66%)    (SQuAD 1.0 format)
  - qa_v2: 660 (4.14%) (SQuAD 2.0 format)

It was not possible to specify the themes addressed in these questions. 

### Citation
The citation will be added when the dataset is made public.