task_categories:sentence-similarity task_categories:translation language:eng language:afr language:nbl language:xho language:zul language:sot language:nso language:tsn language:ssw language:ven language:tso license:cc-by-4.0 multilingual government arxiv:2303.03750 region:us
The dataset contains editions from the South African government magazine Vuk'uzenzele. Data was scraped from PDFs that have been placed in the data/raw folder. The PDFS were obtained from the Vuk'uzenzele website.
@dataset{marivate_vukosi_2023_7598540, author = {Marivate, Vukosi and Njini, Daniel and Madodonga, Andani and Lastrucci, Richard and Dzingirai, Isheanesu Rajab, Jenalea}, title = {The Vuk'uzenzele South African Multilingual Corpus}, month = feb, year = 2023, publisher = {Zenodo}, doi = {10.5281/zenodo.7598539}, url = {https://doi.org/10.5281/zenodo.7598539} }
---
language: 
- eng
- afr
- nbl
- xho
- zul
- sot
- nso
- tsn
- ssw
- ven
- tso
pretty_name: "The Vuk'uzenzele South African Multilingual Corpus"
tags:
- multilingual
- government
license: "cc-by-4.0"
task_categories:
- sentence-similarity
- translation
arxiv: 2303.03750
---

# The Vuk'uzenzele South African Multilingual Corpus

Github: [https://github.com/dsfsi/vukuzenzele-nlp/](https://github.com/dsfsi/vukuzenzele-nlp/)

Zenodo: [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.7598539.svg)](https://doi.org/10.5281/zenodo.7598539)

Arxiv Preprint: [![arXiv](https://img.shields.io/badge/arXiv-2303.03750-b31b1b.svg)](https://arxiv.org/abs/2303.03750)

# About
The dataset was obtained from the South African government magazine Vuk'uzenzele, created by the [Government Communication and Information System (GCIS)](https://www.gcis.gov.za/). 
The original raw PDFS were obtatined from the [Vuk'uzenzele website](https://www.vukuzenzele.gov.za/).

The datasets contain government magazine editions in 11 languages, namely:

|  Language  | Code  |  Language  | Code |
|------------|-------|------------|-------|
| English    | (eng) | Sepedi     | (sep) |
| Afrikaans  | (afr) | Setswana   | (tsn) |
| isiNdebele | (nbl) | Siswati    | (ssw) |
| isiXhosa   | (xho) | Tshivenda  | (ven) |
| isiZulu    | (zul) | Xitstonga  | (tso) |
| Sesotho    | (nso) |

## Available pairings

The alignment direction is bidrectional, i.e. xho-zul is zul-xho

afr-eng; afr-nbl; afr-nso; afr-sot; afr-ssw; afr-tsn; afr-tso; afr-ven; afr-xho; afr-zul  
eng-nbl; eng-nso; eng-sot ;eng-ssw; eng-tsn; eng-tso; eng-ven; eng-xho; eng-zul  
nbl-nso; nbl-sot; nbl-ssw; nbl-tsn; nbl-tso; nbl-ven; nbl-xho; nbl-zul  
nso-sot; nso-ssw; nso-tsn; nso-tso; nso-ven; nso-xho; nso-zul  
sot-ssw; sot-tsn; sot-tso; sot-ven; sot-xho; sot-zul  
ssw-tsn; ssw-tso; ssw-ven; ssw-xho; ssw-zul  
tsn-tso; tsn-ven; tsn-xho; tsn-zul  
tso-ven; tso-xho; tso-zul  
ven-xho; ven-zul  
xho-zul  

# Disclaimer

This dataset contains machine-readable data extracted from PDF documents, from https://www.vukuzenzele.gov.za/, provided by the Government Communication Information System (GCIS). While efforts were made to ensure the accuracy and completeness of this data, there may be errors or discrepancies between the original publications and this dataset. No warranties, guarantees or representations are given in relation to the information contained in the dataset. The members of the Data Science for Societal Impact Research Group bear no responsibility and/or liability for any such errors or discrepancies in this dataset. The Government Communication Information System (GCIS) bears no responsibility and/or liability for any such errors or discrepancies in this dataset. It is recommended that users verify all information contained herein before making decisions based upon this information.

# Datasets
The datasets consist of pairwise sentence aligned data. There are 55 distinct datasets of paired sentences.
The data is obtained by comparing [LASER](https://github.com/facebookresearch/LASER) embeddings of sentence tokens between 2 languages. If the similarity is high, the sentences are deemed semantic equivalents of one another and the observation is outputted. 

Naming convention:  
The naming structure of the pairwise_sentence_aligned folder is `aligned-{src_lang_code}-{tgt_lang_code}.csv`.  
For example, `aligned-afr-zul.csv` is the aligned sentences between Afrikaans and isiZulu. 

The data is in .csv format and the columns are `src_text`,`tgt_text`,`cosine_score` where:  
- `src_text` is the source sentence
- `tgt_text` is the target sentence
- `cosine_score` is the cosine similarity score obtained by comparing the sentence embeddings, it ranges from 0 to 1

**Note:** The notion of source (src) and target (tgt) are only necessary for distinction between the languages used in the aligned pair, as the sentence semantics should be bidirectional. (hallo <-> sawubona)

# Citation
Vukosi Marivate, Andani Madodonga, Daniel Njini, Richard Lastrucci, Isheanesu Dzingirai, Jenalea Rajab. **The Vuk'uzenzele South African Multilingual Corpus**, 2023

> @dataset{marivate_vukosi_2023_7598540,
  author       = {Marivate, Vukosi and
                  Njini, Daniel and
                  Madodonga, Andani and
                  Lastrucci, Richard and
                  Dzingirai, Isheanesu
                  Rajab, Jenalea},
  title        = {The Vuk'uzenzele South African Multilingual Corpus},
  month        = feb,
  year         = 2023,
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.7598539},
  url          = {https://doi.org/10.5281/zenodo.7598539}
}

### Licence
* Licence for Data - [CC 4.0 BY](LICENSE.md)
