task_categories:text-generation annotations_creators:crowdsourced language_creators:crowdsourced multilinguality:monolingual size_categories:10K<n<20k language:ru license:mit spellcheck russian arxiv:2308.09435 region:us
Russian Spellcheck Benchmark is a new benchmark for spelling correction in Russian language.
                It includes four datasets, each of which consists of pairs of sentences in Russian language. 
                Each pair embodies sentence, which may contain spelling errors, and its corresponding correction. 
                Datasets were gathered from various sources and domains including social networks, internet blogs, github commits, 
                medical anamnesis, literature, news, reviews and more.
# TODO: add citation
---
annotations_creators:
- crowdsourced
language_creators:
- crowdsourced
language:
- ru
license: mit
multilinguality:
- monolingual
size_categories:
- 10K<n<20k
task_categories:
- text-generation
pretty_name: Russian Spellcheck Benchmark
language_bcp47:
- ru-RU
tags:
- spellcheck
- russian
---

# Dataset Card for Russian Spellcheck Benchmark

## Table of Contents
- [Table of Contents](#table-of-contents)
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)
  - [Contributions](#contributions)

## Dataset Description

- **Repository:** [SAGE](https://github.com/ai-forever/sage)
- **Paper:** [arXiv:2308.09435](https://arxiv.org/abs/2308.09435)
- **Point of Contact:** nikita.martynov.98@list.ru

### Dataset Summary

Spellcheck Benchmark includes four datasets, each of which consists of pairs of sentences in Russian language. 
Each pair embodies sentence, which may contain spelling errors, and its corresponding correction. 
Datasets were gathered from various sources and domains including social networks, internet blogs, github commits, medical anamnesis, literature, news, reviews and more.

All datasets were passed through two-stage manual labeling pipeline.
 The correction of a sentence is defined by an agreement of at least two human annotators. 
 Manual labeling scheme accounts for jargonisms, collocations and common language, hence in some cases it encourages 
 annotators not to amend a word in favor of preserving style of a text.

### Supported Tasks and Leaderboards

- **Task:** automatic spelling correction.
- **Metrics:** https://www.dialog-21.ru/media/3427/sorokinaaetal.pdf.


### Languages

Russian.

## Dataset Structure

### Data Instances

#### RUSpellRU

- **Size of downloaded dataset files:** 3.64 Mb
- **Size of the generated dataset:** 1.29 Mb
- **Total amount of disk used:** 4.93 Mb

An example of "train" / "test" looks as follows
```
{
    "source": "очень классная тетка ктобы что не говорил.",
    "correction": "очень классная тетка кто бы что ни говорил",
}
```

#### MultidomainGold

- **Size of downloaded dataset files:** 15.05 Mb
- **Size of the generated dataset:** 5.43 Mb
- **Total amount of disk used:** 20.48 Mb

An example of "test" looks as follows
```
{
    "source": "Ну что могу сказать... Я заказала 2 вязанных платья: за 1000 руб (у др продавца) и это ща 1200. Это платье- голимая синтетика (в том платье в составе была шерсть). Это платье как очень плохая резинка. На свои параметры (83-60-85) я заказала С . Пока одевала/снимала - оно в горловине растянулось. Помимо этого в этом платье я выгляжу ну очень тоской. У меня вес 43 кг на 165 см роста.  Кстати, продавец отправлял платье очень долго. Я пыталась отказаться от заказа, но он постоянно отклонял мой запрос. В общем не советую.",
    "correction": "Ну что могу сказать... Я заказала 2 вязаных платья: за 1000 руб (у др продавца) и это ща 1200. Это платье- голимая синтетика (в том платье в составе была шерсть). Это платье как очень плохая резинка. На свои параметры (83-60-85) я заказала С . Пока надевала/снимала - оно в горловине растянулось. Помимо этого в этом платье я выгляжу ну очень доской. У меня вес 43 кг на 165 см роста.  Кстати, продавец отправлял платье очень долго. Я пыталась отказаться от заказа, но он постоянно отклонял мой запрос. В общем не советую.",
    "domain": "reviews",
    
}
```

#### MedSpellcheck

- **Size of downloaded dataset files:** 1.49 Mb
- **Size of the generated dataset:** 0.54 Mb
- **Total amount of disk used:** 2.03 Mb

An example of "test" looks as follows
```
{
    "source": "Кровотечения, поерации в анамнезе отрицает",
    "correction": "Кровотечения, операции в анамнезе отрицает",
}
```


#### GitHubTypoCorpusRu

- **Size of downloaded dataset files:** 1.23 Mb
- **Size of the generated dataset:** 0.48 Mb
- **Total amount of disk used:** 1.71 Mb

An example of "test" looks as follows
```
{
    "source": "## Запросы и ответа содержат заголовки",
    "correction": "## Запросы и ответы содержат заголовки",
}
```


### Data Fields

#### RUSpellRU

- `source`: a `string` feature
- `correction`: a `string` feature
- `domain`: a `string` feature


#### MultidomainGold

- `source`: a `string` feature
- `correction`: a `string` feature
- `domain`: a `string` feature

#### MedSpellcheck

- `source`: a `string` feature
- `correction`: a `string` feature
- `domain`: a `string` feature

#### GitHubTypoCorpusRu

- `source`: a `string` feature
- `correction`: a `string` feature
- `domain`: a `string` feature



### Data Splits

#### 	RUSpellRU
|   |train|test|
|---|---:|---:|
|RUSpellRU|2000|2008|

#### MultidomainGold

|   |train|test|
|---|---:|---:|
|web|386|756|
|news|361|245|
|social_media|430|200|
|reviews|584|586|
|subtitles|1810|1810|
|strategic_documents|-|250|
|literature|-|260|

#### MedSpellcheck

|   |test|
|---|---:|
|MedSpellcheck|1054|

#### GitHubTypoCorpusRu

|   |test|
|---|---:|
|GitHubTypoCorpusRu|868|


## Dataset Creation

### Source Data

#### Initial Data Collection and Normalization

The datasets are chosen in accordance with the specified criteria. 
First, domain variation: half of the datasets are chosen from different domains to ensure diversity, while the remaining half are from a single domain. 
Another criterion is spelling orthographic mistakes:
the datasets exclusively comprised mistyping, omitting grammatical or more complex errors of nonnative speakers.
- **RUSpellRU**: texts collected from ([LiveJournal](https://www.livejournal.com/media)), with manually corrected typos and errors;
- **MultidomainGold**: examples from several text sources including the open web, news, social media, reviews, subtitles, policy documents and literary works were collected:

  *Aranea web-corpus* is a family of multilanguage gigaword web-corpora collected from Internet resources. The texts in the corpora are evenly distributed across periods, writing styles and topics they cover. We randomly picked the sentences from Araneum Russicum, which is harvested from the Russian part of the web.

  *Literature* is a collection of Russian poems and prose of different classical literary works. We randomly picked sentences from the source dataset that were gathered from Ilibrary, LitLib, and Wikisource.

  *News*, as the name suggests, covers news articles on various topics such as sports, politics, environment, economy etc. The passages are randomly picked from the summarization dataset Gazeta.ru.

  *Social media* is the text domain from social media platforms marked with specific hashtags. These texts are typically short, written in an informal style and may contain slang, emojis and obscene lexis.

  *Strategic Documents* is part of the dataset the Ministry of Economic Development of the Russian Federation collected. Texts are written in a bureaucratic manner, rich in embedded entities, and have complex syntactic and discourse structures. The full version of the dataset has been previously used in the RuREBus shared task.

- **MedSpellChecker**: texts with errors from medical anamnesis;
- **GitHubTypoCorpusRu**: spelling errors and typos in commits from [GitHub](https://github.com);


### Annotations

#### Annotation process

We set up two-stage annotation project via a crowd-sourcing platform Toloka:
1. Data gathering stage: we provide the texts with possible mistakes to annotators and ask them to write the sentence correctly;
2. Validation stage: we provide annotators with the pair of sentences (source and its corresponding correction from the previous stage) and ask them to check if the correction is right.

We prepared instructions for annotators for each task. The instructions ask annotators to correct misspellings if it does not alter the original style of the text.
Instructions do not provide rigorous criteria on the matter of distinguishing the nature of an error in terms of its origin - whether it came from an urge to endow a sentence with particular stylistic features or from unintentional spelling violation since it is time-consuming and laborious to describe every possible case of employing slang, dialect, collo- quialisms, etc. instead of proper language. Instructions also do not distinguish errors that come from the geographical or social background of the source. Instead, we rely on annotators’ knowledge and understanding of a language since, in this work, the important factor is to preserve the original style of the text.
To ensure we receive qualified expertise, we set up test iteration on a small subset of the data for both stages. We manually validated the test results and selected annotators, who processed at least six samples (2% of the total test iteration) and did not make a single error. After test iteration, we cut 85% and 86% of labellers for gathering and validation stages.
We especially urge annotators to correct mistakes associated with the substitution of the letters "ё" "й" and "щ" for corresponding "е" "и" and "ш" and not to explain abbreviations and correct punctuation errors. Each annotator is also warned about potentially sensitive topics in data (e.g., politics, societal minorities, and religion).

#### Who are the annotators?

Native Russian speakers who passed the language exam.


## Considerations for Using the Data

### Discussion of Biases

We clearly state our work’s aims and
implications, making it open source and transparent. The data will be available under a public license. As our research involved anonymized textual data, informed consent from human participants was not required. However, we obtained permission to access publicly available datasets and
ensured compliance with any applicable terms of
service or usage policies.

### Other Known Limitations

The data used in our research may be limited to specific
domains, preventing comprehensive coverage of
all possible text variations. Despite these limitations, we tried to address the issue of data diversity
by incorporating single-domain and multi-domain
datasets in the proposed research. This approach
allowed us to shed light on the diversity and variances within the data, providing valuable insights
despite the inherent constraints.

We primarily focus on the Russian language. Further
research is needed to expand the datasets for a wider
range of languages.

## Additional Information

### Future plans

We are planning to expand our benchmark with both new Russian datasets and datasets in other languages including (but not limited to) European and CIS languages. 
If you would like to contribute, please contact us.

### Dataset Curators

Nikita Martynov nikita.martynov.98@list.ru

### Licensing Information

All our datasets are published by MIT License.

### Citation Information
```

@inproceedings{martynov2023augmentation,
  title={Augmentation methods for spelling corruptions},
  author={Martynov, Nikita and Baushenko, Mark and Abramov, Alexander and Fenogenova, Alena},
  booktitle={Proceedings of the International Conference “Dialogue},
  volume={2023},
  year={2023}
}

@misc{martynov2023methodology,
      title={A Methodology for Generative Spelling Correction
via Natural Spelling Errors Emulation across Multiple Domains and Languages}, 
      author={Nikita Martynov and Mark Baushenko and Anastasia Kozlova and
Katerina Kolomeytseva and Aleksandr Abramov and Alena Fenogenova},
      year={2023},
      eprint={2308.09435},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

```