task_categories:text-generation size_categories:1M<n<10M language:ru region:us
---
dataset_info:
  features:
  - name: url
    dtype: string
  - name: text
    dtype: string
  - name: title
    dtype: string
  - name: source
    dtype: string
  - name: timestamp
    dtype: uint64
  splits:
  - name: train
    num_bytes: 12858731888
    num_examples: 4137525
  download_size: 3669747077
  dataset_size: 12858731888
task_categories:
- text-generation
language:
- ru
size_categories:
- 1M<n<10M
---

# RuNews dataset

## Table of Contents
- [Table of Contents](#table-of-contents)
- [Description](#description)
- [Usage](#usage)
- [Data Instances](#data-instances)
- [Personal and Sensitive Information](#personal-and-sensitive-information)

## Description

**Summary:** Dataset of news from several sources:
* [Lenta.ru by yutkin](https://github.com/yutkin/Lenta.Ru-News-Dataset)
* [Several sources by buriy](https://github.com/buriy/russian-nlp-datasets/releases)
* [ODS Newsviz Tass](https://github.com/newsviz/newsviz)
* [Taiga fontanka](https://tatianashavrina.github.io/taiga_site/)
* [News from Telegram contest](https://github.com/IlyaGusev/tgcontest)

**Script:** [create_ru_news.py](https://github.com/IlyaGusev/rulm/blob/master/data_processing/create_ru_news.py)

**Point of Contact:** [Ilya Gusev](ilya.gusev@phystech.edu)

**Languages:** Russian.


## Usage

Prerequisites:
```bash
pip install datasets zstandard jsonlines pysimdjson
```

Dataset iteration:
```python
from datasets import load_dataset
dataset = load_dataset('IlyaGusev/ru_news', split="train", streaming=True)
for example in dataset:
    print(example["text"])
```

## Data Instances

```
{
  "title": "Заместитель главы района в Якутии пожаловался на пьянство начальника",
  "text": "Заместитель главы Нерюнгринского района Якутии Геннадий Ленц пожаловался руководителю республики Егору Борисову на своего начальника. Как рассказал Ленц 'Интерфаксу', Андрей Фитисов пьет на рабочем месте и 'уходит в многодневные загулы'...",
  "timestamp": 1346284800,
  "url": "https://lenta.ru/news/2012/08/30/alco/",
  "source": "lenta"
}
```

## Personal and Sensitive Information

The dataset is not anonymized, so individuals' names can be found in the dataset. Information about the original authors is included in the dataset where possible.