<jupyter_start><jupyter_text>Import packages You'll first need to install either the ```spaCY``` medium or large model!->> terminal```cd cds-languagesource ./lang101/bin/activatepython -m spacy download en_core_web_mddeactivate```<jupyter_code># preprocessing
import os
import pandas as pd
from tqdm import tqdm # progress bar

# nlp
import spacy
nlp = spacy.load("en_core_web_md") # medium model that contains word vectors which the small model does not contain

# gensim
from gensim.models import Word2Vec
import gensim.downloader<jupyter_output><empty_output><jupyter_text>Using pretrained vectors in ```spaCy``` spaCy comes with pretrained word embeddings generated with word2vec. Each word has a vector representing it. The word embedding models in spaCy are not as accurate as Gensim's models, but they are more simple and intuitive. The spaCy word embeddings are not that useful, because spaCy does not make it clear how they have been trained.<jupyter_code># Looking at the vector for the word "denmark"
nlp("denmark").vector<jupyter_output><empty_output><jupyter_text>^This vector is an array of 300 points that encode the word "denmark". Each number is a weight that has been calcualted with the word2vec skipgram model. Skipgram trains a logistic classifier on each word, and then the weights that are trained then become this vector for the word. Hence, we have 300 weights/nodes representing a single words. Every word has a unique representation - unique weights.   __Comparing individual words__ We can calculate similarity between words.<jupyter_code># Creating spacy nlp objects for different words.
banana = nlp("banana")
apple = nlp("apple")
scotland = nlp("scotland")
denmark = nlp("denmark")<jupyter_output><empty_output><jupyter_text>__Inspect word similarities__ With spaCy we can investigate the similarity between word vectors<jupyter_code>banana.similarity(apple)
banana.similarity(scotland)<jupyter_output><empty_output><jupyter_text>^Banana and apple are very similar while banana and Scotland are very dissimilar.<jupyter_code>denmark.similarity(scotland)<jupyter_output><empty_output><jupyter_text>__Document similarities__ With spaCy we can also look at the similarity between sentences/documents. Hence, not just individual words. spaCy does this by taking the vector for each word in the document and then average these vectors, so we can an average vector for each docuemnt. We can then compare these averaged vectors for documents with each other to see how similar documents are to each other. Keep in mind that the results should be taken with a grain of salt, because averaging word vectors is not that informative. Once we have documents consisting of many words, averaging their vectors does not make much sense because it introduces a lot of variance/randomness, and comparing documents in this way is not very useful.<jupyter_code>doc1 = nlp("I like bananas")
doc2 = nlp("I like apples")
doc3 = nlp("I come from Scotland")
doc4 = nlp("I live in Denmark")
doc1.similarity(doc3)
doc3.similarity(doc4)<jupyter_output><empty_output><jupyter_text>Working with ```gensim``` If you want more accurate word embeddings we need to work with the word embeddings provided by gensim. Gensim provides different pretrained word embeddings (see below). __Download pretrained models__<jupyter_code>list(gensim.downloader.info()['models'].keys())<jupyter_output><empty_output><jupyter_text>^These are the different pretrained word embedding models provided by gensim. We can see that there are three different approaches represented: word2vec, GloVe, and fastText. These are different in terms of how they are trained on the data. Their output is essentially the same, but the algorithm used to calcualte the embeddings/weights are not the same across the approaches. FastText models contain subword embeddings which word2vec does not. This means that fastText splits words into subwords which allows for greater generalizability, which allows the model to capture out-of-vocabulary words which is something the word2vec model is not able to.  __Download a pretrained model__<jupyter_code># Download the GloVe model that has been trained on the wiki gigaword corpus
pretrained_vectors = gensim.downloader.load('glove-wiki-gigaword-100')<jupyter_output>[=========-----------------------------------------] 18.3% 23.4/128.1MB downloaded<jupyter_text>In many cases it is useful to train your own word embedding model, but for downstream NLP tasks, e.g. text classification, we can just use a pretrained model that has been trained on a very large corpus to turn a text into a numerical representation.  __Inspect vector for specific word__<jupyter_code># Now we can inspect the vector for the word denmark generated by this particular model
pretrained_vectors['denmark']<jupyter_output><empty_output><jupyter_text>__Find most similar words to target__ With gensim we can find the most similar words to a target word rather than comparing two specific words. This allows us to explore the semantic space of a particular word.<jupyter_code>pretrained_vectors.most_similar('denmark')<jupyter_output><empty_output><jupyter_text>We can see that the most similar words to "denmark" are very close in terms of geography.  __Compare specific words__ We can also compare two specific words.<jupyter_code>pretrained_vectors.similarity('denmark', 'scotland')
pretrained_vectors.similarity('denmark', 'sweden')<jupyter_output><empty_output><jupyter_text>__Vector algebra__We can use word embeddings to study structural relationships. *Man* is to *king* as *woman* is to ...<jupyter_code>pretrained_vectors.most_similar(positive=['king', 'woman'], 
                                negative=['man'], 
                                topn=1)<jupyter_output><empty_output><jupyter_text>This is just simple algebra: [V(king) - V(man)] + V(woman)We subract the vector for man from the vector for king and add the vector for women and we get the vector for queen.Here we are working with gender as a dimension.<jupyter_code>pretrained_vectors.most_similar(positive=['walk', 'swim'], 
                           negative=['walked'], 
                           topn=1)<jupyter_output><empty_output><jupyter_text>Once again it is just simple algebra:[V(walk) - V(walked)] + V(swam)Here we are working with tense as a dimension.<jupyter_code>pretrained_vectors.most_similar(positive=['berlin', 'denmark'], 
                           negative=['germany'], 
                           topn=1)<jupyter_output><empty_output><jupyter_text>[V(berlin) - V(germany)] + V(denmark)here we are working with nationality as a dimension. __Odd one out!__ We can also take a list of words and find the word that does not match - the word that is most dissimilar to the others (furthest away in the embedding space)<jupyter_code>pretrained_vectors.doesnt_match(["france", "germany", "dog", "japan"])<jupyter_output>/home/cds-au617836/cds-language-forked/lang101/lib/python3.6/site-packages/gensim/models/keyedvectors.py:877: FutureWarning: arrays to stack must be passed as a "sequence" type such as list or tuple. Support for non-sequence iterables such as generators is deprecated as of NumPy 1.16 and will raise an error in the future.
  vectors = vstack(self.word_vec(word, use_norm=True) for word in used_words).astype(REAL)<jupyter_text>Word embedding models are capturing something meaningful about words and langauge in general, which can be seen from the word association tasks (e.g. man is to king as woman is to queen). We are also able to encode grammatical information (e.g. walk is to walked as swim is to swam). Hence, there is some kind of structural information in the language that the word embedding model is able to capture in terms of semantic relationships and co-occurrence.   Train your own models Training your own model with gensim is simple. __Load data with pandas__<jupyter_code>filename = os.path.join("..", "data", "labelled_data", "fake_or_real_news.csv")
data = pd.read_csv(filename)
data.head()<jupyter_output><empty_output><jupyter_text>__Tokenize with ```spaCy```__<jupyter_code>sentences = []

for post in tqdm(data["text"]):
    # create a temporary list
    tmp_list = []
    # create spaCy doc object
    doc = nlp(post.lower()) # we need everything to be lowercase to not consider the same word as different words just because of casing
    # loop over
    for token in doc:
        tmp_list.append(token.text)
    # append tmp_list to sentences
    sentences.append(tmp_list)<jupyter_output>100%|██████████| 6335/6335 [15:07<00:00,  6.98it/s]<jupyter_text>__Train model with ```gensim```__<jupyter_code>model = Word2Vec(sentences=sentences,  # input data
                 size=50,              # embedding size (the number of dimensions)
                 window=3,             # context window (the number of words before and after the target word)
                 sg=1,                 # cbow or skip-gram (cbow=0, sg=1)
                 negative=5,           # number of negative samples - the more negative samples, the longer the training will take, because it is going to classify more labels
                 min_count=3,          # remove rare words. Words that appear less than 3 times will be excluded.
                 workers=6)            # number of CPU processes/cores. More cores allows for more parallel processing.<jupyter_output><empty_output><jupyter_text>__Inspect most similar word__<jupyter_code>model.wv.most_similar('obama', topn=10)<jupyter_output><empty_output><jupyter_text>Part of the problem with word embedding models is that they are data-hungry - it takes a lot of data to create useful representation of data. This is something we often run into when we are training our own models. Hence, when training your own model we need to consider whether we have enough data, and how we perform the preprocessing.  __Compare words__<jupyter_code>model.wv.similarity('jesus', 'god')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07758707216701796, 'fraction_numerical': 0.00913379504165412, 'mean_word_length': 4.626199887069452, 'pattern_counts': {'":': 0, '<': 79, '<?xml version=': 0, '>': 80, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '416719', 'license': 'MIT', 'n_tokens_mistral': 2895, 'n_tokens_neox': 2720, 'n_words': 1131, 'path': 'notebooks/session9_WordEmbeddings_au617836.ipynb', 'repo_name': 'sofieditmer/cds-language', 'size': '9963'}