multilinguality:monolingual language:hy arxiv:2009.12615 region:us
---
language:
- hy
task_categories: [paraphrase, paraphrase detection]
multilinguality: [monolingual]
task_ids: [paraphrase, paraphrase detection]
---

## Table of Contents
- [Table of Contents](#table-of-contents)
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
- [Dataset Structure](#dataset-structure)
- [Dataset Evaluation](#dataset-evaluation)
- [Additional Information](#additional-information)

## Dataset Description

We provide sentential paraphrase detection train, test datasets as well as BERT-based models for the Armenian language.

### Dataset Summary

The sentences in the dataset are taken from [Hetq](https://hetq.am/) and [Panarmenian](http://www.panarmenian.net/) news articles. To generate paraphrase for the sentences, we used back translation from Armenian to English. We repeated the step twice, after which the generated paraphrases were manually reviewed. Invalid sentences were filtered out, while the rest were labelled as either paraphrase, near paraphrase or non-paraphrase. Test examples were reviewed by 3 different annotators. In addition, to increase the number of non-paraphrase pairs, we padded the dataset with automatically generated negative examples, including pairs of consecutive sentences and random pairs.

## Dataset Structure

Each row consists of 2 sentences and their label. This sentences were labelled as either paraphrase, near paraphrase or non-paraphrase (with 1, 0, -1 labels respectively). The sentences are divided into train and test sets.

|Number of examples|Total|Paraphrase|Non-paraphrase (near paraphrase)|
|:-- |     :---:    | :---:  |          :---:  |
|Train   | 4233         |1339   |2683 (211) |
|Test    | 1682         |1021   |448 (213) |

### Dataset Evaluation

We finetuned Multilingual BERT on several training sets, including the proposed ARPA dataset, and evaluated their performance on our test set. During training and 
evaluation, near paraphrase and non-paraphrase pairs were combined into one class. The results are provided below:

|BERT Model | Train set |   F1   |  Acc.     |
|:-- |     :---:    |     :---:    |     :---:    |
|Multilingual BERT | ARPA train set| 84.27|  78.06|
|Multilingual BERT | Paraphraser.ru train set machine-translated into Armenian | 83.81  | 77.09  | 
|Multilingual BERT | MRPC train set machine-translated into Armenian  | 80.07        | 69.87 | 
|Multilingual BERT | All of the above combined | 84 |77.6  |


#### Additional Information

The model trained on ARPA is available for use, and can be downloaded using this [link](https://drive.google.com/uc?id=14owW5kkZ1JiNa6P-676e-QIj8m8i5e_8).

For more details about the models and dataset construction, refer to the [paper](https://arxiv.org/pdf/2009.12615).

