task_categories:text-generation language:as language:brx language:bn language:doi language:en language:gom language:gu language:hi language:kha language:kn language:ks language:mai language:ml language:mni language:mr language:ne language:or language:pa language:sa language:sat language:sd language:ta language:te language:ur license:cc0-1.0 arxiv:2212.05409 region:us
    IndicCORPV2 is the largest collection of texts for Indic langauges consisting of 20.9 Billion tokens of which 14.4B tokens correspond to 23 Indic languages and 6.B tokens of Indian English content curated from Indian websites.
    @article{Doddapaneni2022towards,
  title={Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages},
  author={Sumanth Doddapaneni and Rahul Aralikatte and Gowtham Ramesh and Shreyansh Goyal and Mitesh M. Khapra and Anoop Kunchukuttan and Pratyush Kumar},
  journal={ArXiv},
  year={2022},
  volume={abs/2212.05409}
}
---
license: cc0-1.0
language:
- as
- brx
- bn
- doi
- en
- gom
- gu
- hi
- kha
- kn
- ks
- mai
- ml
- mni
- mr
- ne
- or
- pa
- sa
- sat
- sd
- ta
- te
- ur
task_categories:
- text-generation
---
# IndicCorpv2

[Github](https://github.com/AI4Bharat/IndicBERT) | [Paper](https://arxiv.org/abs/2212.05409)   
Languages: 24   
The largest collection of texts for Indic languages consisting of 20.9 billion tokens of which 14.4B tokens correspond to 23 Indic languages and 6.5B tokens of Indian English content curated from Indian websites.
<img src="https://huggingface.co/datasets/satpalsr/indicCorpv2/resolve/main/languages.png">

## Citation
```
@article{Doddapaneni2022towards,
  title={Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages},
  author={Sumanth Doddapaneni and Rahul Aralikatte and Gowtham Ramesh and Shreyansh Goyal and Mitesh M. Khapra and Anoop Kunchukuttan and Pratyush Kumar},
  journal={ArXiv},
  year={2022},
  volume={abs/2212.05409}
}
```