<jupyter_start><jupyter_text>Generate a Vecsigrafo using SwivelIn this notebook we show how to generate a Vecsigrafo based on a subset of the [UMBC corpus](https://ebiquity.umbc.edu/resource/html/id/351/UMBC-webbase-corpus).We follow the procedure described in [Towards a Vecsigrafo: Portable Semantics in Knowledge-based Text Analytics](https://pdfs.semanticscholar.org/b0d6/197940d8f1a5fa0d7474bd9a94bd9e44a0ee.pdf) and depicted in the following figure:![Generic Vecsigrafo Creation](https://github.com/hybridNLP2018/tutorial/blob/master/images/generic-vecsigrafo-creation.png?raw=1)  Tokenization and Word Sense DisambiguationThe main difference with standard swivel is that: - we use word-sense disambiguation on the text as a pre-processing step (Swivel simply uses white-space tokenization) - each 'token' in the resulting sequences is composed of a lemma and an optional concept identifier.  DisambiguatorsIf we are going to apply WSD, we will need some disambiguator strategy. Unfortunately, there are not a lot of open-source high-performance disambiguators available. At [Expert System](https://www.expertsystem.com/), we have a [state-of-the-art disambiguator](https://www.expertsystem.com/products/cogito-cognitive-technology/semantic-technology/disambiguation/) that assings **syncon**s (our version of synsets) to lemmas in the text. Since Expert System's disambiguator and semantic KG are proprietary, in this notebook we will be mostly using WordNet (although we may present some results and examples based on Expert System's results). We have implemented a lightweight disambiguation strategy, proposed by [<NAME>., <NAME>., <NAME>., & <NAME>. (2017). Embedding Words and Senses Together via Joint Knowledge-Enhanced Training. CoNLL.](http://arxiv.org/abs/1612.02703), which has allowed us to produce disambiguated corpora based on WordNet 3.1.To be able to inspect the disambiguated corpus, let's make sure we have access to WordNet in our environment by executing the following cell.<jupyter_code>import nltk
nltk.download('wordnet')
from nltk.corpus import wordnet as wn
wn.synset('Maya.n.02')<jupyter_output><empty_output><jupyter_text>TokenizationsWhen applying a disambiguator, the tokens are no longer (groups of) words. Each token can contain different types of information, we generally keep the following token information:  * `t`: text, the original text (possibly normalised, i.e. lower-cased)  * `l`: lemma, the lemma form of the word  * `g`: grammar: the grammar type  * `s`: syncon (or synset) identifier   Example WordNetWe have included a small sample of our disambiguated UMBC corpus as part of our [GitHub tutorial repo](https://github.com/HybridNLP2018/tutorial). Execute the following cell to clone the repo, unzip the sample corpus and print the first line of the corpus<jupyter_code>%cd /content/
!git clone https://github.com/HybridNLP2018/tutorial.git
%cd /content/tutorial/datasamples/
!unzip umbc_tlgs_wnscd_5K.zip
toked_corpus = '/content/tutorial/datasamples/umbc_tlgs_wnscd_5K'
!head -n1 {toked_corpus}
%cd /content/<jupyter_output><empty_output><jupyter_text>You should see, among others, the first line in the corpus, which starts with:```the%7CGT_ART mayan%7Clem_Mayan%7CGT_ADJ%7Cwn31_Maya.n.03 image%7Clem_image%7CGT_NOU%7Cwn31_effigy.n.01 ``` The file included in the github repo for this tutorial is a subset of a disambiguated tokenization for the UMBC corpus, it only contains the first 5 thousand lines of that corpus (the full corpus has about 40 million lines) as we only need it to show the steps necessary to generate embeddings.The last output, from the cell above, shows the format we are using to represent the tokenized corpus. We use white space to separate the tokens, and have URL encoded each token to avoid mixing up tokens. Since this format is hard to read, we provide a library to inspect the lines in an easy manner. Execute the following cell to display the first two lines in the corpus as a table.<jupyter_code>%cd /content/
import tutorial.scripts.wntoken as wntoken
import pandas

# open the file and produce a list of python dictionaries describing the tokens
corpus_tokens = wntoken.open_as_token_dicts(toked_corpus, max_lines=2)
# convert the tokens into a pandas DataFrame to display in table form
pandas.DataFrame(corpus_tokens, columns=['line', 't', 'l', 'g', 's', 'glossa'])<jupyter_output><empty_output><jupyter_text>Example CogitoAs a second example, analysing the original sentence:       EXPERIMENTAL STUDY  We conducted an empirical evaluation to assess the effectiveness    using Cogito gives us     ![Full Cogito Analysis of example sentence](https://github.com/hybridNLP2018/tutorial/blob/master/images/example-sentence-cogito.PNG?raw=1)We filter some of the words and only keep the lemmas and the syncon ids and encode them into the next sequence of disambiguated tokens:    en86052|experimental en2686|study en76710|conduct en86047|empirical en3546|evaluation en68903|assess     en25094|effectiveness    Vocabulary and Co-occurrence matrixNext, we need to count the co-occurrences in the disambiguated corpus. We can either: - use **standard swivel prep**: in this case each *|||* tuple will be treated as a separate token. For the example sentence from UMBC, presented above, we would then get that `mayan|lem_Mayan|GT_ADJ|wn31_Maya.n.03` has a co-occurrence count of 1 with `image|lem_image|GT_NOU|wn31_effigy.n.01`. This would result in a very large vocabulary. - use **joint-subtoken prep**: in this case, you can specify which individual subtoken information you want to take into account. In this notebook we will use **ls** information, hence each synset and each lemma are treated as separate entities in the vocabulary and will be represented with different embeddings. For the example sentence we would get that `lem_Mayan` has a co-occurrence count of 1 with `wn31_Maya.n.03`, `lem_image` and `wn31_effigy.n.01`.<jupyter_code>import os
import numpy as np<jupyter_output><empty_output><jupyter_text>Standard Swivel PrepFor the **standard swivel prep**, we can simply call `prep` using the `!python` command. In this case we have the `toked_corpus` which contains the disambiguated sequences as shown above. The output wil be a set of sharded co-occurrence submatrices as explained in the notebook for creating word vectors.We set the `shard_size` to 512 since the corpus is quite small. For larger corpora we could use the standard value of 4096.<jupyter_code>!mkdir /content/umbc/
!mkdir /content/umbc/coocs
!mkdir /content/umbc/coocs/tlgs_wnscd_5k_standard
coocs_path = '/content/umbc/coocs/tlgs_wnscd_5k_standard/'
!python tutorial/scripts/swivel/prep.py --input={toked_corpus} --output_dir={coocs_path} --shard_size=512<jupyter_output><empty_output><jupyter_text>Expected output:    ... tensorflow flags ....        vocabulary contains 8192 tokens    writing shard 256/256    Wrote vocab and sum files to /content/umbc/coocs/tlgs_wnscd_5k_standard/    Wrote vocab and sum files to /content/umbc/coocs/tlgs_wnscd_5k_standard/    done!<jupyter_code>!head -n15  /content/umbc/coocs/tlgs_wnscd_5k_standard/row_vocab.txt<jupyter_output><empty_output><jupyter_text>As the cells above show, applying standard prep results in a vocabulary of over 8K "tokens", however each token is still represented as a URL-encoded combination of the plain text, lemma, grammar type and synset (when available).  Joint-subtoken PrepFor the **joint-subtoken prep** step, we have a Java implementation that is not open-source yet (as it is still tied to proprietary code, we are working on refactoring the code so that Cogito subtokens are just a special case). However, we ***provide pre-computed co-occurrence files***.Although not open source, we describe the steps we executed to help you implement a similar pipeline. First,  we ran our implementation of subtoken prep on the corpus. Notice:  * we are only including lemma and synset information (i.e. we are not including plain text and grammar information).   * furthermore, we are filtering the corpus by     1. removing any tokens related to punctuation marks (PNT), auxiliary verbs (AUX) and articles (ART), since we think these do not contribute much to the semantics of words.     2. replacing tokens with grammar types `ENT` (entities) and `NPH` (proper names) with generic variants `grammarENT` and `grammarNPH` respectively. The rationale is that, depending on the input corpus, names of people or organizations may appear a few times, but may be filtered out if they do not appear enough times. This ensures such tokens are kept in the vocabulary and contribute to the embeddings of words nearby. The main disadvantage is that we will not have some proper names in our final vocabulary.```java $JAVA_OPTIONS net.expertsystem.word2vec.swivel.SubtokPrep \  --input C:/hybridNLP2018/tutorial/datasamples/umbc_tlgs_wnscd_5K \  --output_dir C:/corpora/umbc/coocs/tlgs_wnscd_5K_ls_f/  \  --expected_seq_encoding TLGS_WN \  --sub_tokens \  --output_subtokens "LEMMA,SYNSET" \  --remove_tokens_with_grammar_types "PNT,AUX,ART"  \  --generalise_tokens_with_grammar_types "ENT,NPH" \  --shard_size 512```The output log looked as follows:```INFO  net.expertsystem.word2vec.swivel.SubtokPrep - expected_seq_encoding set to 'TLGS_WN'INFO  net.expertsystem.word2vec.swivel.SubtokPrep - remove_tokens_with_grammar_types set to PNT,AUX,ARTINFO  net.expertsystem.word2vec.swivel.SubtokPrep - generalise_tokens_with_grammar_types set to ENT,NPHINFO  net.expertsystem.word2vec.swivel.SubtokPrep - Creating vocab for C:\hybridNLP2018\tutorial\datasamples\umbc_tlgs_wnscd_5KINFO  net.expertsystem.word2vec.swivel.SubtokPrep - read 5000 lines from C:\hybridNLP2018\tutorial\datasamples\umbc_tlgs_wnscd_5KINFO  net.expertsystem.word2vec.swivel.SubtokPrep - filtered 166152 tokens from a total of 427796 (38,839%)generalised 1899 tokens from a total of 427796 (0,444%)full vocab size 21321INFO  net.expertsystem.word2vec.swivel.SubtokPrep - Vocabulary contains 5632 tokens (21321 full count, 5913 appear > 5 times)INFO  net.expertsystem.word2vec.swivel.SubtokPrep - Flushing 1279235 co-occ pairsINFO  net.expertsystem.word2vec.swivel.SubtokPrep - Wrote 121 tmpShards to disk``` We have included the output of this process as part of the GitHub repo for the tutorial. We will unzip this folder to inspect the results:<jupyter_code>!unzip /content/tutorial/datasamples/precomp-coocs-tlgs_wnscd_5K_ls_f.zip -d /content/umbc/coocs/
precomp_coocs_path = '/content/umbc/coocs/tlgs_wnscd_5K_ls_f'<jupyter_output><empty_output><jupyter_text>The previous cell extracts the pre-computed co-occurrence shards and defines a variable `precomp_coocs_path` that points to the folder where these shards are stored.Next, we print the first 10 elements of the vocabulary to see the format that we are using to represent the lemmas and synsets:<jupyter_code>!head -n10 {precomp_coocs_path}/row_vocab.txt<jupyter_output><empty_output><jupyter_text>As the output above shows, the vocabulary we get with `subtoken prep` is smaller (5.6K elements instead of over 8K) and it contains individual lemmas and synsets (it also contains *special* elements grammarENT and grammarNPH, as described above).**More importantly**, the co-occurrence counts take into account the fact that certain lemmas co-occur more frequently with certain other lemmas and synsets, which should be taken into account when learning embedding representations.  Learn embeddings from co-occurrence matrixWith the sharded co-occurrence matrices created in the previous section it is now possible to learn embeddings by calling the `swivel.py` script. This launches a tensorflow application based on various parameters (most of which are self-explanatory) : - `input_base_path`: the folder with the co-occurrence matrix (protobuf files with the sparse matrix) generated above. - `submatrix_` rows and columns need to be the same size as the `shard_size` used in the `prep` step. - `num_epochs` the number of times to go through the input data (all the co-occurrences in the shards). We have found that for large corpora, the learning algorithm converges after a few epochs, while for smaller corpora you need a larger number of epochs.   Execute the following cell to generate embeddings for the pre-computed co-occurrences.<jupyter_code>vec_path = '/content/umbc/vec/tlgs_wnscd_5k_ls_f'
!python /content/tutorial/scripts/swivel/swivel.py --input_base_path={precomp_coocs_path} \
    --output_base_path={vec_path} \
    --num_epochs=40 --dim=150 \
    --submatrix_rows=512 --submatrix_cols=512<jupyter_output><empty_output><jupyter_text>This will take a few minutes, depending on your machine.The result is a list of files in the specified output folder, including: - the tensorflow graph, which defines the architecture of the model being trained - checkpoints of the model (intermediate snapshots of the weights) - `tsv` files for the final state of the column and row embeddings.<jupyter_code>%ls {vec_path}<jupyter_output><empty_output><jupyter_text>Convert `tsv` files to `bin` fileAs we've seen in previous notebooks, the `tsv` files are easy to inspect, but they take too much space and they are slow to load since we need to convert the different values to floats and pack them as vectors. Swivel offers a utility to convert the `tsv` files into a `bin`ary format. At the same time it combines the column and row embeddings into a single space (it simply adds the two vectors for each word in the vocabulary).<jupyter_code>!python /content/tutorial/scripts/swivel/text2bin.py --vocab={precomp_coocs_path}/row_vocab.txt --output={vec_path}/vecs.bin \
        {vec_path}/row_embedding.tsv \
        {vec_path}/col_embedding.tsv<jupyter_output><empty_output><jupyter_text>This adds the `vocab.txt` and `vecs.bin` to the folder with the vectors:<jupyter_code>%ls {vec_path}<jupyter_output><empty_output><jupyter_text>Inspect the embeddingsAs in previous notebooks, we can now use Swivel to inspect the vectors using the `Vecs` class. It accepts a `vocab_file` and a file for the binary serialization of the vectors (`vecs.bin`).<jupyter_code>from tutorial.scripts.swivel import vecs<jupyter_output><empty_output><jupyter_text>...and we can load existing vectors. Here we load some pre-computed embeddings, but feel free to use the embeddings you computed by following the steps above (although, due to random initialization of weight during the training step, your results may be different).<jupyter_code>vectors = vecs.Vecs(precomp_coocs_path + '/row_vocab.txt', 
            vec_path + '/vecs.bin')<jupyter_output><empty_output><jupyter_text>Next, let's define a basic method for printing the `k` nearest neighbors for a given word: And let's use the method on a few lemmas and synsets in the vocabulary:<jupyter_code>import pandas as pd
pd.DataFrame(vectors.k_neighbors('lem_California'))
pd.DataFrame(vectors.k_neighbors('lem_semantic'))
pd.DataFrame(vectors.k_neighbors('lem_conference'))
pd.DataFrame(vectors.k_neighbors('wn31_conference.n.01'))<jupyter_output><empty_output><jupyter_text>Note that using the Vecsigrafo approach gets us very different results than when using standard swivel (notebook 01): * the results now include concepts (synsets), besides just words. Without further information, this makes interpreting the results harder since we now only have the concept id, but we can search for these concepts in the underlying KG (WordNet in this case) to explore the semantic network and get further information. Of course, results may not be very good, since these have been derived from a very small corpus (5K lines from UMBC). In the excercise below, we encourage you to download and inspect pre-computed embeddings based on the full UMBC corpus.<jupyter_code>pd.DataFrame(vectors.k_neighbors('lem_semantic web'))
pd.DataFrame(vectors.k_neighbors('lem_ontology'))<jupyter_output><empty_output><jupyter_text>Conclusion and ExercisesIn this notebook we generated a vecsigrafo based on a disambiguated corpus. The resulting embedding space combines concept ids and lemmas.We have seen that the resulting space: 1. may be harder to inspect due to the potentially opaque concept ids 2. clearly different than standard swivel embeddings The question is: are the resulting embeddings *better*? To get an answer, in the next notebook, we will look at **evaluation methods for embeddings**.  Exercise 1: Explore full precomputed embeddingsWe have also pre-computed embeddings for the full UMBC corpus. The provided `tar.gz` file is about 1.1GB, hence downloading it may take several minutes.<jupyter_code>full_precomp_url = 'https://zenodo.org/record/1446214/files/vecsigrafo_umbc_tlgs_ls_f_6e_160d_row_embedding.tar.gz'
full_precomp_targz = '/content/umbc/vec/tlgs_wnscd_ls_f_6e_160d_row_embedding.tar.gz'
!wget {full_precomp_url} -O {full_precomp_targz}<jupyter_output><empty_output><jupyter_text>Next, we have to unpack the vectors:<jupyter_code>!tar -xzf {full_precomp_targz} -C /content/umbc/vec/
full_precomp_vec_path = '/content/umbc/vec/vecsi_tlgs_wnscd_ls_f_6e_160d'
%ls /content/umbc/vec/vecsi_tlgs_wnscd_ls_f_6e_160d/<jupyter_output><empty_output><jupyter_text>The data only includes the `tsv` version of the vectors, so we need to convert these to the binary format that Swivel uses. And for that, we aso need a `vocab.txt` file,  which we can derive from the tsv as follows:<jupyter_code>with open(full_precomp_vec_path + '/vocab.txt', 'w', encoding='utf_8') as f:
  with open(full_precomp_vec_path + '/row_embedding.tsv', 'r', encoding='utf_8') as vec_lines:
    vocab = [line.split('\t')[0].strip() for line in vec_lines]
    for word in vocab:
      print(word, file=f)<jupyter_output><empty_output><jupyter_text>Let's inspect the vocabulary:<jupyter_code>!wc -l {full_precomp_vec_path}/vocab.txt
!grep 'wn31_' {full_precomp_vec_path}/vocab.txt | wc -l
!grep 'lem_' {full_precomp_vec_path}/vocab.txt | wc -l<jupyter_output><empty_output><jupyter_text>As we can see, the embeddings have a vocabulary of just under 1.5M entries, 56K of which are synsets and most of the rest are lemmas.Next, convert the `tsv` into the swivel's binary format. This can take a couple of minutes.<jupyter_code>!python /content/tutorial/scripts/swivel/text2bin.py --vocab={full_precomp_vec_path}/vocab.txt --output={full_precomp_vec_path}/vecs.bin \
        {full_precomp_vec_path}/row_embedding.tsv<jupyter_output><empty_output><jupyter_text>Now, we are ready to load the vectors.<jupyter_code>vecsi_wn_umbc = vecs.Vecs(full_precomp_vec_path + '/vocab.txt', 
            full_precomp_vec_path + '/vecs.bin')
pd.DataFrame(vecsi_wn_umbc.k_neighbors('lem_California'))
pd.DataFrame(vecsi_wn_umbc.k_neighbors('lem_semantic'))
pd.DataFrame(vecsi_wn_umbc.k_neighbors('lem_conference'))
print(wn.synset('conference.n.01').definition())
pd.DataFrame(vecsi_wn_umbc.k_neighbors('wn31_conference.n.01'))
print(wn.synset('conference.n.03').definition())
pd.DataFrame(vecsi_wn_umbc.k_neighbors('wn31_conference.n.03'))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0786056049213944, 'fraction_numerical': 0.017666543982333457, 'mean_word_length': 5.542827657378741, 'pattern_counts': {'":': 0, '<': 93, '<?xml version=': 0, '>': 94, 'https://': 9, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '35563', 'license': 'MIT', 'n_tokens_mistral': 6211, 'n_tokens_neox': 5750, 'n_words': 2092, 'path': '03_vecsigrafo.ipynb', 'repo_name': 'flaviomerenda/tutorial', 'size': '19046'}