region:us
# Dataset Card for register_oscar

## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-instances)
 
### Dataset Summary

The Register Oscar dataset is a multilingual dataset, containing languaegs from the Oscar dataset that have been tagged with register information. 

8 main-level registers:
* Narrative (NA)
* Informational Description (IN)
* Opinion (OP)
* Interactive Discussion (ID)
* How-to/Instruction (HI)
* Informational Persuasion (IP)
* Lyrical (LY)
* Spoken (SP)

For further description of the labels, see (Douglas Biber and Jesse Egbert. 2018. Register variation online)

Code used to tag Register Oscar can be found at https://github.com/TurkuNLP/register-labeling

### Languages

Currently contains the following languages: Arabic, Bengali, Catalan, English, Spanish, Basque, French, Hindi, Indonesian, Portuguese, Swahili, Urdu, Vietnamese and Chinese.

For further information on the languages and data, see https://huggingface.co/datasets/oscar

## Dataset Structure

### Data Instances

```
{"id": "0", "labels": ["NA"], "text": "Zarif: Iran inajua mpango wa Saudia wa kufanya mauaji ya kigaidi dhidi ya maafisa wa ngazi za juu wa Iran\n"}
```

### Data Fields

* id: unique id of the document (from the Oscar dataset)
* labels: the list of labels assigned to the text
* text: the original text of the document (as appears in the Oscar dataset)


### Citing

```
@inproceedings{laippala-etal-2022-towards,
    title = "Towards better structured and less noisy Web data: Oscar with Register annotations",
    author = {Laippala, Veronika  and
      Salmela, Anna  and
      R{\"o}nnqvist, Samuel  and
      Aji, Alham Fikri  and
      Chang, Li-Hsin  and
      Dhifallah, Asma  and
      Goulart, Larissa  and
      Kortelainen, Henna  and
      P{\`a}mies, Marc  and
      Prina Dutra, Deise  and
      Skantsi, Valtteri  and
      Sutawika, Lintang  and
      Pyysalo, Sampo},
    booktitle = "Proceedings of the Eighth Workshop on Noisy User-generated Text (W-NUT 2022)",
    month = oct,
    year = "2022",
    address = "Gyeongju, Republic of Korea",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.wnut-1.23",
    pages = "215--221",
    abstract = {Web-crawled datasets are known to be noisy, as they feature a wide range of language use covering both user-generated and professionally edited content as well as noise originating from the crawling process. This article presents one solution to reduce this noise by using automatic register (genre) identification -whether the texts are, e.g., forum discussions, lyrical or how-to pages. We apply the multilingual register identification model by R{\"o}nnqvist et al. (2021) and label the widely used Oscar dataset. Additionally, we evaluate the model against eight new languages, showing that the performance is comparable to previous findings on a restricted set of languages. Finally, we present and apply a machine learning method for further cleaning text files originating from Web crawls from remains of boilerplate and other elements not belonging to the main text of the Web page. The register labeled and cleaned dataset covers 351 million documents in 14 languages and is available at https://huggingface.co/datasets/TurkuNLP/register{\_}oscar.},
}
```