task_categories:translation size_categories:10K<n<100K language:en language:rw license:cc-by-2.0 region:us
---
---
license: cc-by-2.0
task_categories:
- translation
language:
- en
- rw
size_categories:
- 10K<n<100K
---


## Dataset Description
This dataset was created to develop a machine translation model for bidirectional translation between Kinyarwanda and English for education-based sentences, in particular for the [Atingi](https://www.atingi.org/) learning platform.
- **Repository:**[link](https://github.com/Digital-Umuganda/twb_nllb_project_tourism_education) to the GitHub repository containing the code for training the model on this data, and the code for the collection of the monolingual data.
- **Data Format:** TSV
- **Model:** huggingface [model link](mbazaNLP/Nllb_finetuned_education_en_kin).

  
### Dataset Summary



### Data Instances

```
118347	103384	And their ideas was that the teachers just didn't care and had no time for them.	Kandi igitekerezo cyabo nuko abarimu batabitayeho gusa kandi ntibabone umwanya.	2023-06-25 09:40:28	 223	1	3	education	coursera	72-93
```

### Data Fields

- id
- source_id
- source
- phrase
- timestamp
- user_id
- validation_state
- validation_score
- domain
- source_files
- str_ranges

### Data Splits

- **Training Data:** 58251
- **Validation Data:** 2456
- **Test Data:** 1060

## Data Preprocessing

- **Data Splitting:** To create a test set; all data sources are equally represented in terms of the number of sentences contributed to the test dataset. In terms of sentence length, the test set distribution is similar to the sentence length distribution of the whole dataset. After picking the test set, from the remaining data the train and validation data are split using sklearn's [train_test_split](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html).

## Data Collection

- **Data Collection Process:** The monolingual source sentences were obtained through web-scraping of several websites containing English sentences. 

- **Data Sources:**
  -  Coursera
  -  Atingi
  -  Wikipedia




## Dataset Creation

After collecting the monolingual dataset, human translators were employed to produce translations for the collected sentences. To ensure quality, each sentence was translated more than once, and each generated translation was assigned **validation_score** that was used to pick the best translation. The test dataset was further revised to remove or correct sentences with faulty translations.



