task_categories:automatic-speech-recognition annotations_creators:found language_creators:found multilinguality:monolingual size_categories:10K<n<100K source_datasets:extended|tilde_model source_datasets:extended|wikipedia source_datasets:extended|europarl_bilingual language:lv license:cc-by-4.0 lv latvian region:us
---
annotations_creators:
- found
language:
- lv
language_creators:
- found
license:
- cc-by-4.0
multilinguality:
- monolingual
pretty_name: Latvian text dataset
size_categories:
- 10K<n<100K
source_datasets:
- extended|tilde_model
- extended|wikipedia
- extended|europarl_bilingual
tags:
- lv
- latvian
task_categories:
- automatic-speech-recognition
task_ids: []
---

# Latvian text dataset

Data set of latvian language texts. Intended for use in AI tool development, like speech recognition or spellcheckers


## Data sources used
* Latvian Wikisource articles - https://wikisource.org/wiki/Category:Latvian
* Literary works of Rainis - https://repository.clarin.lv/repository/xmlui/handle/20.500.12574/41
* Latvian Wikipedia articles - https://huggingface.co/datasets/joelito/EU_Wikipedias
* European Parliament Proceedings Parallel Corpus - https://huggingface.co/datasets/europarl_bilingual
* Tilde MODEL Corpus - Multilingual Open Data for European Languages - https://huggingface.co/datasets/tilde_model



To get Wikipedia dataset (197MB) run. 
```
python tools/wikipedia/GetWikipedia.py
```

To get Europarl dataset (1.7GB) run. 
```
python tools/europarl/GetEuroparl.py
```

To get Tilde dataset (834MB) run. 
```
python tools/europarl/GetTilde.py
```

To combine all datasets run 
```
sh combine-all.sh
```

To clean out some junk run.
```
sh clean.sh
```

Also maybe you want to remove duplocate lines. To do so run 
```
sort lv.txt | uniq > lv-uniq.txt
```

## Notes

Possible future sources
* Parliament proceedings transcripts - https://www.saeima.lv/lv/transcripts
* Discussions of Latvian Wikipedia pages - https://lv.wikipedia.org/wiki/Special:AllPages
* Out of copyright books from LNB collection - https://data.gov.lv/dati/lv/dataset/gramatu-digitala-kolekcija

Data sets not used
* Web scrapes, as they tend to yield data from comments with improper spelling like "atrashanaas vieta" instead of "atrašanās vieta"
* Open Subtitles, as they contain data with improper spelling like "atrashanaas vieta" instead of "atrašanās vieta"

Possible issues:
* Data sets contain foreign language characters, like "蠻子" or cyrilic f.e. "Рига"