task_categories:text-classification size_categories:1K<n<10K language:ru license:cc-by-4.0 fact-checking region:us
Fact-checking benchmark for the Russian Big Language Models.
---
license: cc-by-4.0
task_categories:
- text-classification
language:
- ru
tags:
- fact-checking
size_categories:
- 1K<n<10K
---

# Dataset Card for RuFacts

## Dataset Description

RuFacts is a benchmark for internal fact-checking for the Russian language. The dataset contains tagged examples labeled consistent and inconsistent. 

For inconsistent examples, ranges containing violations of facts in the source text and the generated text are also collected and presented on the [Kaggle competition page](https://www.kaggle.com/competitions/internal-fact-checking-for-the-russian-language).

Various data sources and approaches for data generation were used to create the training and test datasets for the fact-checking task. We consider the data on the sentence level and small texts. The average length of texts is 198 symbols, the minimum is 10 symbols, and the maximum is 3,402 symbols. 
The final dataset was formed using three main approaches: 
* Texts generated by a [paraphrase model](https://habr.com/ru/companies/sberdevices/articles/667106/)
* Translations of the [dataset for fact-checking](https://fever.ai/dataset/fever.html)
* Text augmentation

Translations and generated data were manually labeled via the crowd-sources platform Yandex.Toloka. We additionally manually annotate the augmented data for
the test set. The test set consists of examples from all three sources: 26% translations, 6% augmented data, and 68% generated paraphrases.

We require three criteria for the generated text to be factually consistent with the original:
1. facts are correct and not corrupted;
2. any additional facts in the generated texts are not included;
3. all the main facts are included in the generated text.

## Data Structure

### Data Fields

* `idx`: an integer
* `evidence`: a string containing the original text
* `claim`: a string containing the generated text by some genetative models
* `label`: an integer, either 0 or 1, indicating whether the facts are consistent (0) or inconsistent (1)

An example of `train`/`validation` looks as follows:
```
{'idx': 1,
 'evidence': 'Суд в Англии рассмотрит дело советского диссидента Буковского',
 'claim': 'Суд в Великобритании рассмотрит дело советского диссидента Буковского',
 'label': 0}
```
An example of `test` looks as follows:
```
{'idx': 4,
 'evidence': 'Google выплатит штраф в 200 млн долларов за сбор данных детей на YouTube.',
 'claim': 'Google заплатит $200 млн за нарушения конфиденциальности детей на YouTube.',
 'label': -1}
```
### Data Splits

|     |train | validation | test|
|-----|------|------------|-----|
|rows |4677  | 1559       | 500 |