<jupyter_start><jupyter_text>Dependency Parsing with SuParProject repo: https://github.com/yzhangcs/parserPrerequisites:* `python`: 3.7* `pytorch`: 1.4* `transformers`: 3.0We recommend installing SuPar from GitHub repository. The `pip install supar` version did not allow us to train the models.In the Issues, the author as well recommends installing SuPar from source.<jupyter_code>## If not installed earlier:
# !apt install git -y
# !apt install wget -y
# !pip install gensim
!pip install transformers
!pip install corpuscula
!pip install junky

# At the time we tested SuPar, the `pip install` version didn't work, so we
# clone the original repo with the latest version

!git clone https://github.com/yzhangcs/parser
!cd parser && python setup.py install
!wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1YV9L3AXORclrfGKuxh9LuARNhaUX5iri' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p')&id=1YV9L3AXORclrfGKuxh9LuARNhaUX5iri" -O model && rm -rf /tmp/cookies.txt
!wget http://files.deeppavlov.ai/embeddings/ft_native_300_ru_wiki_lenta_nltk_wordpunct_tokenize/ft_native_300_ru_wiki_lenta_nltk_wordpunct_tokenize.vec<jupyter_output><empty_output><jupyter_text>Loading corpus<jupyter_code>from corpuscula.corpus_utils import download_ud, syntagrus

corpus_name = 'UD_Russian-SynTagRus'
# corpus_name = 'UD_Russian-Taiga'
download_ud(corpus_name, root_dir='.')<jupyter_output><empty_output><jupyter_text>Filtering embeddingsSuPar requires a lot of CPU memory to load word embeddings. 25 Gb is not enough. If you have enough CPU memory, you can use SuPar without filtering embeddings.Otherwise, shrinking word vectors definitely helps while still showing comparable-to-SOTA results and taking no more than a couple of minutes. We implemented vector filtering in our `junky` library.<jupyter_code>import junky
junky.clear_tqdm()
fpath = 'corpus/_UD/UD_Russian-SynTagRus/ru_syntagrus-ud-{}.conllu'

train, train_lemmas = junky.get_conllu_fields(fpath.replace('{}', 'train'), fields=['LEMMA'])
dev, dev_lemmas = junky.get_conllu_fields(fpath.replace('{}', 'dev'), fields=['LEMMA'])
test, test_lemmas = junky.get_conllu_fields(fpath.replace('{}', 'test'), fields=['LEMMA'])
full_corpus = train + train_lemmas + dev + dev_lemmas + test + test_lemmas
# any pretrained word embeddings in txt format where first line is `<vocab_size> <emb_dim>`
FT_VECTORS_PATH = 'ft_native_300_ru_wiki_lenta_nltk_wordpunct_tokenize.vec'

_ = junky.filter_embeddings(pretrained_embs=FT_VECTORS_PATH, corpus=full_corpus,
                      min_abs_freq=1, save_name='filtered_vectors_freq1.vec',
                      pad_token='[PAD]', unk_token='[UNK]')<jupyter_output><empty_output><jupyter_text>Biaffine Dependency Parsing [[Link]](https://arxiv.org/abs/1611.01734) Original Paper: <NAME> and <NAME>. 2017. Deep Biaffine Attention for Neural Dependency Parsing.  Training with w2v+Bert embeddings Depending on the GPU you have, you might have the following error while running the training script:```Error: mkl-service + Intel(R) MKL: MKL_THREADING_LAYER=INTEL is incompatible with libgomp.so.1 library.	Try to import numpy first or set the threading layer accordingly. Set MKL_SERVICE_FORCE_INTEL to force it.```To avoid the problem, set the environment variable `MKL_SERVICE_FORCE_INTEL=1`. You can run the code below and then run training script withour restarting the notebook.```pythonimport osos.environ["MKL_SERVICE_FORCE_INTEL"] = "1"```<jupyter_code>!python -m supar.cmds.biaffine_dependency train -b -d 0    \
  -p ./model-bert \
  -f bert  \
  --punct  \
  --train corpus/_UD/UD_Russian-SynTagRus/ru_syntagrus-ud-train.conllu \
  --dev corpus/_UD/UD_Russian-SynTagRus/ru_syntagrus-ud-dev.conllu \
  --test corpus/_UD/UD_Russian-SynTagRus/ru_syntagrus-ud-test.conllu \
  --embed filtered_vectors_freq1.vec \
  --unk '[UNK]' \
  --n-embed 300 \
  --batch-size 1000 \
  --bert bert-base-multilingual-cased<jupyter_output><empty_output><jupyter_text>Training with w2v+character embeddings<jupyter_code>!python -m supar.cmds.biaffine_dependency train -b -d 0    \
  -p ./model-char \
  -f char  \
  --punct  \
  --train corpus/_UD/UD_Russian-SynTagRus/ru_syntagrus-ud-train.conllu \
  --dev corpus/_UD/UD_Russian-SynTagRus/ru_syntagrus-ud-dev.conllu \
  --test corpus/_UD/UD_Russian-SynTagRus/ru_syntagrus-ud-test.conllu \
  --embed filtered_vectors_freq1.vec \
  --unk '[UNK]' \
  --n-embed 300 \
  --batch-size 1000<jupyter_output><empty_output><jupyter_text>Distributed training on several GPUs<jupyter_code>!python -m torch.distributed.launch \
  --nproc_per_node=4 --master_port=10000  \
  -m supar.cmds.biaffine_dependency train -b -d 0,1 \
  -p ./model-bert-distr \
  -f bert  \
  --punct  \
  --train corpus/_UD/UD_Russian-SynTagRus/ru_syntagrus-ud-train.conllu \
  --dev corpus/_UD/UD_Russian-SynTagRus/ru_syntagrus-ud-dev.conllu \
  --test corpus/_UD/UD_Russian-SynTagRus/ru_syntagrus-ud-test.conllu \
  --embed filtered_vectors_freq1.vec \
  --unk '[UNK]' \
  --n-embed 300 \
  --batch-size 1500 \
  --bert bert-base-multilingual-cased<jupyter_output><empty_output><jupyter_text>Prediction<jupyter_code>from supar import Parser
parser = Parser.load('model')
test_dataset = parser.predict(test, pred='supar_char_depparse_test.conllu')<jupyter_output>2020-08-07 13:32:45 INFO Load the data
2020-08-07 13:32:51 INFO                                                           
Dataset(n_sentences=6491, n_batches=26, n_buckets=8)
2020-08-07 13:32:51 INFO Make predictions on the dataset
100%|####################################| 26/26 00:04<00:00,  6.33it/s
2020-08-07 13:32:56 INFO Save predicted results to supar_char_depparse_test.conllu
2020-08-07 13:32:57 INFO 0:00:04.629948s elapsed, 1401.96 Sents/s<jupyter_text>EvaluationDuring evaluation, the model makes predictions on the target corpus first, and then runs evaluation.<jupyter_code>parser = Parser.load('model-bert')
loss, metric = parser.evaluate('corpus/_UD/UD_Russian-SynTagRus/ru_syntagrus-ud-test.conllu')
parser = Parser.load('model')
loss, metric = parser.evaluate('corpus/_UD/UD_Russian-SynTagRus/ru_syntagrus-ud-test.conllu')<jupyter_output>2020-08-07 13:33:41 INFO Load the data
2020-08-07 13:33:48 INFO                                                           
Dataset(n_sentences=6491, n_batches=26, n_buckets=8)
2020-08-07 13:33:48 INFO Evaluate the dataset
2020-08-07 13:33:52 INFO loss: 0.2753 - UCM: 58.45% LCM: 47.97% UAS: 94.29% LAS: 92.15%
2020-08-07 13:33:52 INFO 0:00:04.232478s elapsed, 1533.62 Sents/s
{'fraction_non_alphanumeric': 0.1254125412541254, 'fraction_numerical': 0.04665466546654665, 'mean_word_length': 4.482730263157895, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 35, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '288241', 'license': 'CC0-1.0', 'n_tokens_mistral': 2722, 'n_tokens_neox': 2406, 'n_words': 514, 'path': 'supar/supar_depparse.ipynb', 'repo_name': 'steysie/parse-xplore', 'size': '6689'}