task_categories:translation language_creators:expert-generated multilinguality:multilingual multilinguality:translation size_categories:1K<n<10K language:as language:bn language:brx language:doi language:en language:gom language:gu language:hi language:kn language:ks language:mai language:ml language:mr language:mni language:ne language:or language:pa language:sa language:sat language:sd language:ta language:te language:ur license:cc-by-4.0 arxiv:2305.16307 region:us
IN-22 is a newly created comprehensive benchmark for evaluating machine translation performance in multi-domain, n-way parallel contexts across 22 Indic languages. 
IN22-Conv is the conversation domain subset of IN22. It is designed to assess translation quality in typical day-to-day conversational-style applications. 
Currently, we use it for sentence-level evaluation of MT systems but can be repurposed for document translation evaluation as well.
@article{ai4bharat2023indictrans2,
  title   = {IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages},
  author  = {AI4Bharat and Jay Gala and Pranjal A. Chitale and Raghavan AK and Sumanth Doddapaneni and Varun Gumma and Aswanth Kumar and Janki Nawale and Anupama Sujatha and Ratish Puduppully and Vivek Raghavan and Pratyush Kumar and Mitesh M. Khapra and Raj Dabre and Anoop Kunchukuttan},
  year    = {2023},
  journal = {arXiv preprint arXiv: 2305.16307}
}
---
language:
- as
- bn
- brx
- doi
- en
- gom
- gu
- hi
- kn
- ks
- mai
- ml
- mr
- mni
- ne
- or
- pa
- sa
- sat
- sd
- ta
- te
- ur
language_details: >-
  asm_Beng, ben_Beng, brx_Deva, doi_Deva, eng_Latn, gom_Deva, guj_Gujr,
  hin_Deva, kan_Knda, kas_Arab, mai_Deva, mal_Mlym, mar_Deva, mni_Mtei, 
  npi_Deva, ory_Orya, pan_Guru, san_Deva, sat_Olck, snd_Deva, tam_Taml, 
  tel_Telu, urd_Arab
license: cc-by-4.0
language_creators:
- expert-generated
multilinguality:
- multilingual
- translation
pretty_name: in22-conv
size_categories:
- 1K<n<10K
task_categories:
- translation
---

# IN22-Conv

IN-22 is a newly created comprehensive benchmark for evaluating machine translation performance in multi-domain, n-way parallel contexts across 22 Indic languages. IN22-Conv is the conversation domain subset of IN22. It is designed to assess translation quality in typical day-to-day conversational-style applications. The evaluation subset consists of 1024 sentences translated across 22 Indic languages enabling evaluation of MT systems across 506 directions.

Currently, we use it for sentence-level evaluation of MT systems but can be repurposed for document translation evaluation as well.

Here is the domain distribution of our IN22-Conv evaluation subset.

<table style="width:25%">
    <tr>
        <td>domain</td>
        <td>count</td>
    </tr>
    <tr>
        <td>hobbies</td>
        <td>120</td>
    </tr>
    <tr>
        <td>daily_dialogue</td>
        <td>117</td>
    </tr>
    <tr>
        <td>government</td>
        <td>116</td>
    </tr>
    <tr>
        <td>geography</td>
        <td>114</td>
    </tr>
    <tr>
        <td>sports</td>
        <td>100</td>
    </tr>
    <tr>
        <td>entertainment</td>
        <td>97</td>
    </tr>
    <tr>
        <td>history</td>
        <td>97</td>
    </tr>
    <tr>
        <td>legal</td>
        <td>96</td>
    </tr>
    <tr>
        <td>arts</td>
        <td>95</td>
    </tr>
    <tr>
        <td>college_life</td>
        <td>94</td>
    </tr>
    <tr>
        <td>tourism</td>
        <td>91</td>
    </tr>
    <tr>
        <td>school_life</td>
        <td>87</td>
    </tr>
    <tr>
        <td>insurance</td>
        <td>82</td>
    </tr>
    <tr>
        <td>culture</td>
        <td>73</td>
    </tr>
    <tr>
        <td>healthcare</td>
        <td>67</td>
    </tr>
    <tr>
        <td>banking</td>
        <td>57</td>
    </tr>
    <tr>
        <td>total</td>
        <td>1503</td>
    </tr>
</table>

Please refer to the `Appendix E: Dataset Card` of the [preprint](https://arxiv.org/abs/2305.16307) on detailed description of dataset curation, annotation and quality control process.


### Dataset Structure

#### Dataset Fields

- `id`: Row number for the data entry, starting at 1.
- `doc_id`: Unique identifier of the conversation.
- `sent_id`: Unique identifier of the sentence order in each conversation.
- `topic`: The specific topic of the conversation within the domain.
- `domain`: The domain of the conversation.
- `prompt`: The prompt provided to annotators to simulate the conversation.
- `scenario`: The scenario or context in which the conversation takes place.
- `speaker`: The speaker identifier in the conversation.
- `turn`: The turn within the conversation.

#### Data Instances

A sample from the `gen` split for the English language (`eng_Latn` config) is provided below. All configurations have the same structure, and all sentences are aligned across configurations and splits.

```python
{
   "id": 1,
   "doc_id": 0,
   "sent_id": 1,
   "topic": "Festivities",
   "domain": "culture",
   "prompt": "14th April a holiday",
   "scenario": "Historical importance",
   "speaker": 1,
   "turn": 1,
   "sentence": "Mom, let's go for a movie tomorrow."
}
```

When using a hyphenated pairing or using the `all` function, data will be presented as follows:

```python
{
   "id": 1,
   "doc_id": 0,
   "sent_id": 1,
   "topic": "Festivities",
   "domain": "culture",
   "prompt": "14th April a holiday",
   "scenario": "Historical importance",
   "speaker": 1,
   "turn": 1,
   "sentence_eng_Latn": "Mom, let's go for a movie tomorrow.",
   "sentence_hin_Deva": "माँ, चलो कल एक फिल्म देखने चलते हैं।"
}
```

#### Sample Conversation

<table>
    <tr>
        <td>Speaker</td>
        <td>Turn</td>
    </tr>
    <tr>
        <td>Speaker 1</td>
        <td>Mom, let&#39;s go for a movie tomorrow. I don&#39;t have to go to school. It is a holiday.</td>
    </tr>
    <tr>
        <td>Speaker 2</td>
        <td>Oh, tomorrow is the 14th of April right? Your dad will also have the day off from work. We can make a movie plan!</td>
    </tr>
    <tr>
        <td>Speaker 1</td>
        <td>That&#39;s a good news! Why is it a holiday though? Are all schools, colleges and offices closed tomorrow?</td>
    </tr>
    <tr>
        <td>Speaker 2</td>
        <td>It is Ambedkar Jayanti tomorrow! This day is celebrated annually to mark the birth of Dr. B. R Ambedkar. Have you heard of him?</td>
    </tr>
    <tr>
        <td>Speaker 1</td>
        <td>I think I have seen him in my History and Civics book. Is he related to our Constitution?</td>
    </tr>
    <tr>
        <td>Speaker 2</td>
        <td>Absolutely! He is known as the father of the Indian Constitution. He was a civil rights activist who played a major role in formulating the Constitution. He played a crucial part in shaping the vibrant democratic structure that India prides itself upon.</td>
    </tr>
    <tr>
        <td></td>
        <td>...</td>
    </tr>
</table>


### Usage Instructions

```python
from datasets import load_dataset

# download and load all the pairs
dataset = load_dataset("ai4bharat/IN22-Conv", "all")

# download and load specific pairs
dataset = load_dataset("ai4bharat/IN22-Conv", "eng_Latn-hin_Deva")
```

### Languages Covered

<table style="width: 40%">
    <tr>
        <td>Assamese (asm_Beng)</td>
        <td>Kashmiri (Arabic) (kas_Arab)</td>
        <td>Punjabi (pan_Guru)</td>
    </tr>
    <tr>
        <td>Bengali (ben_Beng)</td>
        <td>Kashmiri (Devanagari) (kas_Deva)</td>
        <td>Sanskrit (san_Deva)</td>
    </tr>
    <tr>
        <td>Bodo (brx_Deva)</td>
        <td>Maithili (mai_Deva)</td>
        <td>Santali (sat_Olck)</td>
    </tr>
    <tr>
        <td>Dogri (doi_Deva)</td>
        <td>Malayalam (mal_Mlym)</td>
        <td>Sindhi (Arabic) (snd_Arab)</td>
    </tr>
    <tr>
        <td>English (eng_Latn)</td>
        <td>Marathi (mar_Deva)</td>
        <td>Sindhi (Devanagari) (snd_Deva)</td>
    </tr>
    <tr>
        <td>Konkani (gom_Deva)</td>
        <td>Manipuri (Bengali) (mni_Beng)</td>
        <td>Tamil (tam_Taml)</td>
    </tr>
    <tr>
        <td>Gujarati (guj_Gujr)</td>
        <td>Manipuri (Meitei) (mni_Mtei)</td>
        <td>Telugu (tel_Telu)</td>
    </tr>
    <tr>
        <td>Hindi (hin_Deva)</td>
        <td>Nepali (npi_Deva)</td>
        <td>Urdu (urd_Arab)</td>
    </tr>
    <tr>
        <td>Kannada (kan_Knda)</td>
        <td>Odia (ory_Orya)</td>
    </tr>
</table>


### Citation

If you consider using our work then please cite using:

```
@article{ai4bharat2023indictrans2,
  title   = {IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages},
  author  = {AI4Bharat and Jay Gala and Pranjal A. Chitale and Raghavan AK and Sumanth Doddapaneni and Varun Gumma and Aswanth Kumar and Janki Nawale and Anupama Sujatha and Ratish Puduppully and Vivek Raghavan and Pratyush Kumar and Mitesh M. Khapra and Raj Dabre and Anoop Kunchukuttan},
  year    = {2023},
  journal = {arXiv preprint arXiv: 2305.16307}
}
```

