task_categories:conversational task_categories:text-generation task_categories:text2text-generation annotations_creators:machine-generated language_creators:found multilinguality:monolingual size_categories:100K<n<1M source_datasets:extended language:bn license:cc-by-nc-sa-4.0 arxiv:2007.01852 arxiv:1606.05250 region:us
DailyDialogue (bengali) has been derived from the original English dataset.
@inproceedings{bhattacharjee-etal-2023-banglanlg,
    title = "{B}angla{NLG} and {B}angla{T}5: Benchmarks and Resources for Evaluating Low-Resource Natural Language Generation in {B}angla",
    author = "Bhattacharjee, Abhik  and
      Hasan, Tahmid  and
      Ahmad, Wasi Uddin  and
      Shahriyar, Rifat",
    booktitle = "Findings of the Association for Computational Linguistics: EACL 2023",
    month = may,
    year = "2023",
    address = "Dubrovnik, Croatia",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-eacl.54",
    pages = "726--735",
    abstract = "This work presents {`}BanglaNLG,{'} a comprehensive benchmark for evaluating natural language generation (NLG) models in Bangla, a widely spoken yet low-resource language. We aggregate six challenging conditional text generation tasks under the BanglaNLG benchmark, introducing a new dataset on dialogue generation in the process. Furthermore, using a clean corpus of 27.5 GB of Bangla data, we pretrain {`}BanglaT5{'}, a sequence-to-sequence Transformer language model for Bangla. BanglaT5 achieves state-of-the-art performance in all of these tasks, outperforming several multilingual models by up to 9{\%} absolute gain and 32{\%} relative gain. We are making the new dialogue dataset and the BanglaT5 model publicly available at https://github.com/csebuetnlp/BanglaNLG in the hope of advancing future research on Bangla NLG.",
}
---
annotations_creators:
- machine-generated
language_creators:
- found
multilinguality:
- monolingual
size_categories:
- 100K<n<1M
source_datasets:
- extended
task_categories:
  - conversational
  - text-generation
  - text2text-generation
language:
- bn
license:
- cc-by-nc-sa-4.0
---

# Dataset Card for `dailydialogue_bn`

## Table of Contents
- [Dataset Card for `dailydialogue_bn`](#dataset-card-for-dailydialogue_bn)
  - [Table of Contents](#table-of-contents)
  - [Dataset Description](#dataset-description)
    - [Dataset Summary](#dataset-summary)
    - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
    - [Languages](#languages)
    - [Usage](#usage)
  - [Dataset Structure](#dataset-structure)
    - [Data Instances](#data-instances)
    - [Data Fields](#data-fields)
    - [Data Splits](#data-splits)
  - [Dataset Creation](#dataset-creation)
    - [Curation Rationale](#curation-rationale)
    - [Source Data](#source-data)
      - [Initial Data Collection and Normalization](#initial-data-collection-and-normalization)
      - [Who are the source language producers?](#who-are-the-source-language-producers)
    - [Annotations](#annotations)
      - [Annotation process](#annotation-process)
      - [Who are the annotators?](#who-are-the-annotators)
    - [Personal and Sensitive Information](#personal-and-sensitive-information)
  - [Considerations for Using the Data](#considerations-for-using-the-data)
    - [Social Impact of Dataset](#social-impact-of-dataset)
    - [Discussion of Biases](#discussion-of-biases)
    - [Other Known Limitations](#other-known-limitations)
  - [Additional Information](#additional-information)
    - [Dataset Curators](#dataset-curators)
    - [Licensing Information](#licensing-information)
    - [Citation Information](#citation-information)
    - [Contributions](#contributions)

## Dataset Description

- **Repository:** [https://github.com/csebuetnlp/BanglaNLG](https://github.com/csebuetnlp/BanglaNLG)
- **Paper:** [**"BanglaNLG and BanglaT5: Benchmarks and Resources for Evaluating Low-Resource Natural Language Generation in Bangla"**](https://aclanthology.org/2023.findings-eacl.54/)
- **Point of Contact:** [Tahmid Hasan](mailto:tahmidhasan@cse.buet.ac.bd)

### Dataset Summary

This is a Multi-turn dialogue dataset for Bengali, curated from the original English [DailyDialogue]() dataset and using the state-of-the-art English to Bengali translation model introduced **[here](https://aclanthology.org/2020.emnlp-main.207/).**


### Supported Tasks and Leaderboards

[More information needed](https://github.com/csebuetnlp/BanglaNLG)

### Languages

* `Bengali`

### Usage
```python
from datasets import load_dataset
dataset = load_dataset("csebuetnlp/dailydialogue_bn")
```
## Dataset Structure

### Data Instances

One example from the dataset is given below in JSON format. Each element of the `dialogue` feature represents a single turn of the conversation.
  ```
  {
    "id": "130",
    "dialogue": 
    [
        "তোমার জন্মদিনের জন্য তুমি কি করবে?",
        "আমি আমার বন্ধুদের সাথে পিকনিক করতে চাই, মা।",
        "বাড়িতে পার্টি হলে কেমন হয়? এভাবে আমরা একসাথে হয়ে উদযাপন করতে পারি।",
        "ঠিক আছে, মা। আমি আমার বন্ধুদের বাড়িতে আমন্ত্রণ জানাবো।"
    ]
  }
  ```

### Data Fields

The data fields are as follows:

- `id`: a `string` feature.
- `dialogue`: a List of `string` feature.


### Data Splits

|   split   |count  |
|----------|--------|
|`train`|  11118 |
|`validation`| 1000  |
|`test`| 1000 |




## Dataset Creation

For the training set, we translated the complete [DailyDialogue](https://aclanthology.org/N18-1101/) dataset using the English to Bangla translation model introduced [here](https://aclanthology.org/2020.emnlp-main.207/). Due to the possibility of incursions of error during automatic translation, we used the [Language-Agnostic BERT Sentence Embeddings (LaBSE)](https://arxiv.org/abs/2007.01852) of the translations and original sentences to compute their similarity. A datapoint was accepted if all of its constituent sentences had a similarity score over 0.7. 

### Curation Rationale

[More information needed](https://github.com/csebuetnlp/BanglaNLG)

### Source Data

[DailyDialogue](https://arxiv.org/abs/1606.05250)

#### Initial Data Collection and Normalization

[More information needed](https://github.com/csebuetnlp/BanglaNLG)


#### Who are the source language producers?

[More information needed](https://github.com/csebuetnlp/BanglaNLG)


### Annotations

[More information needed](https://github.com/csebuetnlp/BanglaNLG)


#### Annotation process

[More information needed](https://github.com/csebuetnlp/BanglaNLG)

#### Who are the annotators?

[More information needed](https://github.com/csebuetnlp/BanglaNLG)

### Personal and Sensitive Information

[More information needed](https://github.com/csebuetnlp/BanglaNLG)

## Considerations for Using the Data

### Social Impact of Dataset

[More information needed](https://github.com/csebuetnlp/BanglaNLG)

### Discussion of Biases

[More information needed](https://github.com/csebuetnlp/BanglaNLG)

### Other Known Limitations

[More information needed](https://github.com/csebuetnlp/BanglaNLG)

## Additional Information

### Dataset Curators

[More information needed](https://github.com/csebuetnlp/BanglaNLG)

### Licensing Information

Contents of this repository are restricted to only non-commercial research purposes under the [Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC BY-NC-SA 4.0)](https://creativecommons.org/licenses/by-nc-sa/4.0/). Copyright of the dataset contents belongs to the original copyright holders.
### Citation Information

If you use the dataset, please cite the following paper:
```
@inproceedings{bhattacharjee-etal-2023-banglanlg,
    title = "{B}angla{NLG} and {B}angla{T}5: Benchmarks and Resources for Evaluating Low-Resource Natural Language Generation in {B}angla",
    author = "Bhattacharjee, Abhik  and
      Hasan, Tahmid  and
      Ahmad, Wasi Uddin  and
      Shahriyar, Rifat",
    booktitle = "Findings of the Association for Computational Linguistics: EACL 2023",
    month = may,
    year = "2023",
    address = "Dubrovnik, Croatia",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-eacl.54",
    pages = "726--735",
    abstract = "This work presents {`}BanglaNLG,{'} a comprehensive benchmark for evaluating natural language generation (NLG) models in Bangla, a widely spoken yet low-resource language. We aggregate six challenging conditional text generation tasks under the BanglaNLG benchmark, introducing a new dataset on dialogue generation in the process. Furthermore, using a clean corpus of 27.5 GB of Bangla data, we pretrain {`}BanglaT5{'}, a sequence-to-sequence Transformer language model for Bangla. BanglaT5 achieves state-of-the-art performance in all of these tasks, outperforming several multilingual models by up to 9{\%} absolute gain and 32{\%} relative gain. We are making the new dialogue dataset and the BanglaT5 model publicly available at https://github.com/csebuetnlp/BanglaNLG in the hope of advancing future research on Bangla NLG.",
}
```


### Contributions

Thanks to [@abhik1505040](https://github.com/abhik1505040) and [@Tahmid](https://github.com/Tahmid04) for adding this dataset.