<commit_before>---
layout: base
title:  'Introduction'
permalink: hy/overview/introduction.html
udver: '2'
---

# Introduction

UD currently contains one treebank for Armenian (Eastern):

* The Eastern Armenian data come from the ՀայՇտեմ - ArmDT-East dataset (version 1.0): a mix of random sentences sampled from different sources and representing different genres and domains, released in several formats (local on-line newspaper and journal articles, contemporary fiction, textbooks). UD Eastern Armenian is a conversion of the ArmDT-East dataset originally annotated and manually checked by a team led by <NAME> at the Yerevan State University and by <NAME> in YerevaNN research lab (Yerevan).

Below we give a brief description of UD Eastern Armenian treebank. More information can be found in the respective README files.

## UD Armenian (Eastern)
[https://github.com/UniversalDependencies/UD_Armenian/](https://github.com/UniversalDependencies/UD_Armenian/)  

UD Armenian is based on the ՀայՇտեմ - ArmDT-East dataset (version 1.0) originally annotated by the team led by <NAME>. It consists of --- sentences and --- tokens. The annotation scheme was developed in according to the UD guidelines. All data were checked manually. Documentation is provided by <NAME> and <NAME>.

### Source of annotations

This table summarizes the origins and checking of the various columns of the CoNLL-U data.

| Column | Status |
| ------ | ------ |
| ID | Sentence segmentation and tokenization was automatically done using [HyNish-ArmDT tokenizer](https://github.com/Armtreebank/Tokenizer). Additional changes (splitting and merging) were done manually during the annotation. |
| FORM |  |
| LEMMA | Generated automatically from FORM using Eastern Armenian lexicons and then hand-corrected. |
| UPOSTAG | Generated automatically from LEMMA and then hand-corrected. |
| XPOSTAG | _ (currently unused) |
| FEATS | Generated automatically from LEMMA and then hand-corrected. |
| HEAD | Annotated manually. |
| DEPREL | Annotated manually. |
| DEPS | _ (currently unused) |
| MISC | Information about token spacing. |

### Acknowledgments

* This work became possible in part by a research grant from the Armenian National Science and Education Fund (ANSEF) based in New York, USA.
* We are deeply grateful to ANSEF, also to Lragir.am, 1in.am and Yavruhrat Publishing for letting us download and exploit their articles as text material under the terms of educational use.
<commit_msg>Update introduction.md<commit_after>---
layout: base
title:  'Introduction'
permalink: hy/overview/introduction.html
udver: '2'
---

# Introduction

UD currently contains one treebank for Eastern Armenian:

* UD_Armenian

## UD Armenian (Eastern)
[https://github.com/UniversalDependencies/UD_Armenian/](https://github.com/UniversalDependencies/UD_Armenian/)  

UD_Armenian is based on the ՀայՇտեմ - ArmDT-East dataset (version 1.0), a mix of random sentences sampled from different sources and representing different genres and domains, released in several formats (local on-line newspaper and journal articles, contemporary fiction, textbooks), originally annotated by a team led by <NAME> at the Yerevan State University. The annotation scheme was developed in according to the UD guidelines. All data were checked manually. The tokenization and POS-tagging process was carried out through alternating steps of automatic scripting and manual revision, and finally with some out-of-domain parsing modules led by <NAME> in the YerevaNN research lab. Parsing results also underwent a manual revision.

UD_Armenian comprises --- sentences and --- tokens. Documentation is provided by <NAME> and <NAME>.

The first preliminary release was issued in April 2018, for the CoNLL-2018 shared task.

### Source of annotations

This table summarizes the origins and checking of the various columns of the CoNLL-U data.

| Column | Status |
| ------ | ------ |
| ID | Sentence segmentation and tokenization was automatically done using [HyNish-ArmDT tokenizer](https://github.com/Armtreebank/Tokenizer). Additional changes (splitting and merging) were done manually during the annotation. |
| FORM |  |
| LEMMA | Generated automatically from FORM using Eastern Armenian lexicons and then hand-corrected. |
| UPOSTAG | Generated automatically from LEMMA and then hand-corrected. |
| XPOSTAG | _ (currently unused) |
| FEATS | Generated automatically from LEMMA and then hand-corrected. |
| HEAD | Annotated manually. |
| DEPREL | Annotated manually. |
| DEPS | _ (currently unused) |
| MISC | Information about token spacing. |

### Acknowledgments

* This work became possible in part by a research grant from the Armenian National Science and Education Fund (ANSEF) based in New York, USA.
* We are deeply grateful to ANSEF, also to Lragir.am, 1in.am and Yavruhrat Publishing for letting us download and exploit their articles as text material under the terms of educational use.

{'fraction_non_alphanumeric': 0.07463592233009708, 'fraction_numerical': 0.0036407766990291263, 'mean_word_length': 4.4944444444444445, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 12, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6157412', 'n_tokens_mistral': 1378, 'n_tokens_neox': 1263, 'n_words': 607}