task_categories:summarization task_categories:feature-extraction size_categories:1B<n<10B language:as language:bh language:bn language:en language:gu language:hi language:kn language:ml language:mr language:ne language:or language:pa language:ta language:te language:ur license:cc arxiv:2305.05858 region:us
---
license: cc
task_categories:
- summarization
- feature-extraction
language:
- as
- bh
- bn
- en
- gu
- hi
- kn
- ml
- mr
- ne
- or
- pa
- ta
- te
- ur
pretty_name: varta
size_categories:
- 1B<n<10B
---

## Dataset Description

- **Repository:** https://github.com/rahular/varta
- **Paper:** https://arxiv.org/abs/2305.05858

### Dataset Summary

Varta is a diverse, challenging, large-scale, multilingual, and high-quality headline-generation dataset containing 41.8 million news articles in 14 Indic languages and English. 
The data is crawled from DailyHunt, a popular news aggregator in India that pulls high-quality articles from multiple trusted and reputed news publishers. 

### Languages

Assamese, Bhojpuri, Bengali, English, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Oriya, Punjabi, Tamil, Telugu, and Urdu.

## Dataset Structure

### Data Instances
```
{
"id":"n400000150",
"langCode":"as",
"source_url":"https://www.etvbharat.com/assamese/assam/bharat/militant-hideout-destroyed-on-srinagar-bandipora-highway/assam20220630074145729729173",
"dh_url":"https://m.dailyhunt.in/news/india/assamese/etvbharatassamese-epaper-dh6b381d65c3344bbcad9a06ee28b4ab2a/boma+nikshepeve+dhbans+kva+hl+santvasabadiv+aatmagopanasthali-newsid-n400000150"
}
```

### Data Fields

- id: unique identifier for the artilce on DailyHunt. This id will be used to recreate the dataset.
- langCode: ISO 639-1 language code
- source_url: the url that points to the article on the website of the original publisher
- dh_url: the url that points to the article on DailyHunt

### Data Splits

From every language, we randomly sample 10,000 articles each for validation and testing. We also ensure that at least 80% of a language’s data is available for training. 
Therefore, if a language has less than 100,000 articles, we restrict its validation and test splits to 10% of its size.

We also create a `small` training set by limiting the number of articles from each language to 100K.
This `small` training set with a size of 1.3M is used in all our fine-tuning experiments.
You can find the `small` training set [here](https://huggingface.co/datasets/rahular/varta/blob/main/varta/train/train_100k.json)

## Data Recreation
To recreate the dataset, follow this [README file](https://github.com/rahular/varta/tree/main/crawler#README.md).

## Misc
- Original source: https://m.dailyhunt.in/
- License: CC-BY 4.0

## Citation Information

```
@misc{aralikatte2023varta,
      title={V\=arta: A Large-Scale Headline-Generation Dataset for Indic Languages}, 
      author={Rahul Aralikatte and Ziling Cheng and Sumanth Doddapaneni and Jackie Chi Kit Cheung},
      year={2023},
      eprint={2305.05858},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```