task_categories:text-classification task_categories:translation size_categories:10K<n<100K license:cc-by-4.0 region:us
---
license: cc-by-4.0
task_categories:
- text-classification
- translation
size_categories:
- 10K<n<100K
---

# Dataset Card for Ehn-Bible-BBC-GPT3.5

## Dataset Description

- **Repository:** https://huggingface.co/datasets/NITHUB-AI/Ehn-bible-bbc-gpt3.5/
- **Paper:** To be added
- **Point of Contact:** fortuneadekogbe@gmail.com

### Dataset Summary

This dataset card contains parallel Nigerian Pidgin and English sentences split into three files, namely: `train.csv`, `valid.csv` and `test.csv`. 
The original data was split in the ratio of 8:1:1 to obtain these files.

### Supported Tasks and Leaderboards

- Language Translation
- Language Identification

### Languages

- English
- Nigerian Pidgin

## Dataset Structure

### Data Instances

![image/png](https://cdn-uploads.huggingface.co/production/uploads/6212bf377b3af3ccd458002a/dnL7SM_Lvom534sqJBGGE.png)

### Data Fields

- English: contains sentences in the English language
- Pidgin: contains corresponding sentences in Nigerian Pidgin language

### Data Splits

- train (80%)
- validation (10%)
- test (10%)

## Dataset Creation

This section details the process involved in creating this Data.

### Curation Rationale

The data was curated first from the context of the Bible, which proved to be the largest available source of English-Nigerian Pidgin parallel sentences.
For the English sentences, The Message translation of the Bible was used because it presented the most modern form of English.
This data was, however, not versatile enough, so we scraped Pidgin data from the BBC Pidgin website. This platform provided data in wider contexts, from politics to entertainment.
Naturally, this makes the model more versatile.


### Source Data

#### Initial Data Collection and Normalization

- The data was scraped using BeautifulSoup in Python and stored in a MongoDB database
- The Bible-sourced data was split into samples by verses because that was the easiest way to retain context between parallel sentences. Primarily because sentences in English and Nigerian Pidgin were not perfect matches.
- The BBC Pidgin data was translated using Open AI's GPT3.5-turbo via the API and the [LangChain](https://python.langchain.com/) package.

#### Who are the source language producers?

- [Domot - BBC News Pidgin](https://www.bbc.com/pidgin/)
- [YouVersion PCM Bible](https://www.bible.com/bible/2516/GEN.1.PCM)
- [YouVersion Message Translation Bible](https://www.bible.com/bible/97/GEN.1.MSG)

### Personal and Sensitive Information

No additional effort was taken to remove sensitive information aside from what was done by the writers at BBC News Pidgin and the Bible.

## Considerations for Using the Data

### Social Impact of Dataset

This data makes it easier for Engineers to build language tools that work for a less literate but digitally connected Nigerian audience.

### Discussion of Biases

The data is primarily focused on News and Biblical texts. While this has a reasonably wide scope, it is quite limited, and the model will perform considerably poorly in completely alien contexts.
 
### Other Known Limitations

- The data does not contain other versions of Pidgin, like Warri Pidgin or Pidgin from other African nations.
- The data does not have sentences that contain a lot of domain-specific Jargon.

## Additional Information

### Dataset Curators

- [Fortune Adekogbe](https://www.linkedin.com/in/fortune-adekogbe)
- [Joseph Olaide](https://ng.linkedin.com/in/josepholaide)

### Citation Information

- [Domot - BBC News Pidgin](https://www.bbc.com/pidgin/)
- (Open AI GPT3.5-Turbo)[https://platform.openai.com]
  
### Contributions

We welcome contributions from individuals who understand Nigerian Pidgin to help scale up our manual data translation efforts. Motivated developers interested in building interfaces for this are also welcome.