<jupyter_start><jupyter_text>Evaluating Word (and Concept) EmbeddingsIn the previous notebooks we have seen how to generate word, knowledge graph and joint (word-concept) embeddings.We also saw that it is easy to explore the resulting embedding spaces using cosine similarity and selecting the *k-nearest neighbours*.In this notebook we look further into how (word) embeddings are evaluated. In particular, we look into the following methods:  - **Visual Exploration**: whereby (a subsection of) the embeddings are displayed  - **Intrinsic Evaluation**: whereby the embeddings are used to perform a token-based task and the results are compared with a gold standard.    + **Word Prediction**: whereby we look into using a test corpus to evaluate the embeddings by defining a word prediction task.  - **Extrinsic Evaluation**: whereby a new model is learned (using the embeddings as inputs) to perform a complex task.   KG embeddings tend to be evaluated using **graph completion** tasks, which we will also discuss briefly.  Recommended papers in this area[<NAME>., <NAME>., <NAME>., & <NAME>. (2015). Evaluation methods for unsupervised word embeddings. In EMNLP (pp. 298–307). Association for Computational Linguistics.](http://anthology.aclweb.org/D/D15/D15-1036.pdf) Provides a good overview of methods and introduces terminology to refer to different types of evaluations.[<NAME>., <NAME>., & <NAME>. (2014). Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors. In ACL (pp. 238–247).](http://anthology.aclweb.org/P/P14/P14-1023.pdf) Focuses mostly on *intrinsic* evaluations. Showed that predictive models (like word2vec) produced better results than count models (based on co-occurrence counting).[<NAME>., <NAME>., & <NAME>. (2015). Improving Distributional Similarity with Lessons Learned from Word Embeddings. Transactions of the Association for Computational Linguistics, 3(0), 211–225.](https://www.transacl.org/ojs/index.php/tacl/article/view/570) Studied how various implementation or optimization 'details' used in predictive models, which were not needed or used in count models affect the performance of the resulting embeddings. Example of such details are: negative sampling, dynamic context windows, subsampling and vector normalization. The paper shows that once such details are taken into account, the difference between count and predictive models is not that large.<jupyter_code>%cd /content/tutorial
!git pull
%cd /content/
!git clone https://github.com/HybridNLP2018/tutorial.git<jupyter_output>fatal: destination path 'tutorial' already exists and is not an empty directory.<jupyter_text>Visual ExplorationUse dimensionality reduction algorithms such as t-SNE and PCA to visualize (a subset) of the embedding space to project points to a 2-D or 3-D space.[Embedding Projector](http://projector.tensorflow.org/) - Pros:   - Can give you a sense of whether the model has correctly learned meaningful relations. Especially if you have a small number of pre-categorized words.   - Easy to explore the space - Cons:   - Subjective: neighbourhoods may look good, but are they? There is no gold standard   - Works best for a small subset of the embedding space. But who decides which subset?   - resulting projection can be deceiving: what looks close in 3-D space can be far in 300-D space (and vice-versa).  Intrinsic Evaluation**Intrinsic** evaluations are those where you can use embeddings to perform relatively simple, word-related tasks.Schnabel et al. distinguish between: - **Absolute intrinsic**: you have a (human annotated) gold standard for a particular task and use the embeddings to make predictions. - **Comparative intrinsic**: you use the embedding space to present predictions to humans, who then rate them. Mostly used when there is no gold standard available. Tasks: - **Relatedness**: How well do embeddings capture human-perceived word similarity? Datasets typically consist of triples: two words and a similarity score (e.g. between 0.0 and 1.0). Several available datasets, although interpretation of 'word similarity' can vary. - **Synonym detection**: Can embeddings select a synonym for a given word and a set of options? Datasts are n-tuples where the first word is the input word and the other `n-1` words are the options. Only one of the options is a synonym. - **Analogy**: Do embeddings encode relations between words? Datasets are 4-tuples: the first two words define the relation, the third word is the source of the query and the fourth word is the solution. Good embeddings should predict an embedding close to the solution word. - **Categorization**: Can embeddings be clustered into hand-annotated categories? Datasets are word-category pairs. Standard clustering algorithms can then be used to generate k-clusters and the purity of the clusters can be computed. - **Selectional preference**: Can embeddings predict whether a noun-verb pair is more likely to represent a verb-subject or a verb-object relation? E.g. people-eat is more likely to be found as a verb-subject.  Compute Relatedness ScoreSwivel comes with a `eval.mk` script that downloads and unzips various relatedness and analogy datasets. The script also compiles an `analogy` executable. It assumes you have a unix environment and tools such as `wget`, `tar`, `unzip` and `egrep`, as well as `make` and a `c++` compiler.For convenience, we have included various relatedness datasets as part of this repo in `eval-datastets/relatedness`. We assume you have generated vectors as part of previous notebooks, which we will test here.<jupyter_code>import os
%ls /content/tutorial/datasamples/relatedness/
%cp /content/umbc/coocs/tlgs_wnscd_5K_ls_f/row_vocab.txt /content/umbc/vec/tlgs_wnscd_5k_ls_f/vocab.txt
umbc_5k_vec = '/content/umbc/vec/tlgs_wnscd_5k_ls_f/'
umbc_full_vec = '/content/umbc/vec/vecsi_tlgs_wnscd_ls_f_6e_160d/'<jupyter_output><empty_output><jupyter_text>You can use Swivel's `wordsim.py` to produce metrics for the k-cap embeddings we produced in previous notebooks:<jupyter_code>!python /content/tutorial/scripts/swivel/wordsim.py --vocab={umbc_5k_vec}vocab.txt \
  --embeddings={umbc_5k_vec}vecs.bin \
  --word_prefix="lem_" \
  /content/tutorial/datasamples/relatedness/*.ws.tab  
%ls {umbc_full_vec}vocab.txt
!python /content/tutorial/scripts/swivel/wordsim.py --vocab=/content/umbc/vec/vecsi_tlgs_wnscd_ls_f_6e_160d/vocab.txt \
  --embeddings={umbc_full_vec}vecs.bin \
  --word_prefix="lem_" \
  /content/tutorial/datasamples/relatedness/*.ws.tab<jupyter_output>/content/umbc/vec/vecsi_tlgs_wnscd_ls_f_6e_160d/vocab.txt
Opening vector with expected size 1499136 from file /content/umbc/vec/vecsi_tlgs_wnscd_ls_f_6e_160d/vocab.txt
vocab size 1499136 (unique 1499125)
read rows
1433 of 2034 pairs found
0.401 /content/tutorial/datasamples/relatedness/rarewords.ws.tab
999 of 999 pairs found
0.276 /content/tutorial/datasamples/relatedness/simlex999.ws.tab
3494 of 3500 pairs found
0.191 /content/tutorial/datasamples/relatedness/simverb3500.ws.tab
250 of 252 pairs found
0.529 /content/tutorial/datasamples/relatedness/ws353rel.ws.tab
202 of 203 pairs found
0.649 /content/tutorial/datasamples/relatedness/ws353sim.ws.tab
{'fraction_non_alphanumeric': 0.08022724123597062, 'fraction_numerical': 0.024663987806567825, 'mean_word_length': 4.985074626865671, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '147454', 'license': 'MIT', 'n_tokens_mistral': 2201, 'n_tokens_neox': 1988, 'n_words': 850, 'path': '04_evaluating_word_and_kg_embeddings.ipynb', 'repo_name': 'flaviomerenda/tutorial', 'size': '7252'}