<jupyter_start><jupyter_text>Class 6: Word embeddings  Preparation for class---So was sick friday so didn't get the chance to get this out before the weekend. So you really don't have to prepare anything, but I might be worth going through the gensim section code to get an overview before class and to make sure that everything work as intended.  Gensim--- You can install gensim simply using: ```pypip install gensim```  Using Gensimto start us of we can download the word embeddings we will use for class using:<jupyter_code>import gensim.downloader as api

# Pre-trained vectors based on Wikipedia 2014 + Gigaword, 
# 5.6B tokens, 400K vocab, 
# uncased (https://nlp.stanford.edu/projects/glove/).
# embedding size 50 (so fairly small, but easy to work with)
word_emb = api.load("glove-wiki-gigaword-50")
# try out other word embeddings, see what is available here: https://github.com/RaRe-Technologies/gensim-data<jupyter_output>[==================================================] 100.0% 376.1/376.1MB downloaded<jupyter_text>We can the parts of the word embedding class here, but I do encourage you to inspect it a bit more using e.g. `dir(word_emb)` or `help(word_emb)`:<jupyter_code>print(type(word_emb))

# inspecting the vocabulary
vocab = word_emb.index_to_key.keys()
vocab = sorted(list(vocab)) # sort the vocab
print(vocab[:10])
print(vocab[51000: 51010])
print(vocab[-10:])<jupyter_output><class 'gensim.models.keyedvectors.Word2VecKeyedVectors'><jupyter_text>We can now extract word embeddings for words in the vocabulary:<jupyter_code># fetch the word embedding for class
glass = word_emb["glass"]
print(type(glass))
print(glass.shape) # fairly small word embedding

print("aarhus" in word_emb) # check if word in vocabulary<jupyter_output><class 'numpy.ndarray'>
(50,)
True<jupyter_text>As you probably know from the lectures the similarity between word embedding is calculated using cosine similarity, which you might recall we implemented in the introduction week. We can use this to calculate the similarity between a couple of vectors here:<jupyter_code>import numpy as np
def cosine(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

for w1, w2 in [("dog", "dog"), ("dog", "glass"), ("dog", "cat")]:
    sim = cosine(word_emb[w1], word_emb[w2])
    print(f"the similarity between {w1} and {w2} is {round(sim, 2)}")

# You can also calculate this using gensim:
word_emb.similarity('dog', 'glass')<jupyter_output>the similarity between dog and dog is 1.0
the similarity between dog and glass is 0.27000001072883606
the similarity between dog and cat is 0.9200000166893005<jupyter_text>You could imagine doing this for all words in the corpus to get the most similar words to `dog`, using gensim this can be done simply using:<jupyter_code>print(word_emb.most_similar("dog", topn=10))<jupyter_output>[('cat', 0.9218005537986755), ('dogs', 0.8513159155845642), ('horse', 0.7907583713531494), ('puppy', 0.7754921913146973), ('pet', 0.7724707722663879), ('rabbit', 0.7720813751220703), ('pig', 0.7490061521530151), ('snake', 0.7399188280105591), ('baby', 0.7395570874214172), ('bite', 0.7387937903404236)]<jupyter_text>This function also helps you do analogies, *"woman is to ____ what man is to king"* or rephrased *king-man + woman = ?*:<jupyter_code># you can also do analogies using this function:
sim = word_emb.most_similar(positive = ["woman", "king"], negative=["man"], topn=1)
print(sim)<jupyter_output>[('queen', 0.8523603677749634)]<jupyter_text>Naturally the phrase *"king - man + woman = ?"* is nonsense, you can't do mathematics with words, but you can with word embeddings. So let us just go through what is going on here:We start of taking the word embedding of king $w_{king}$ substract the word embedding for man $w_{man}$ and then add the word embedding for women $w_{woman}$, thus we have: $$w_{anology} = w_{king} - w_{man} + w_{woman}$$Naturally it would be extremely coincidental that we were to have $w_{anology}$ being equal to $w_{queen}$, what we actually do is calculate the similarity between all vectors and $w_{anology}$ and find that $w_{queen}$ is the closest to that one, thus: $$w_{anology} \sim w_{queen}$$Do note that we are working with vectors and that these are [cummutative](https://en.wikipedia.org/wiki/Commutative_property) (e.g. $v_1 + v_2 = v_2 + v_1$) thus it is just as valid to write:$$w_{anology} = w_{king} + w_{woman} - w_{man} = w_{king} + (w_{woman} - w_{man})$$Where you can interpret $w_{woman} - w_{man}$ as the *gender component* assuming that everything else about the usage of man and woman is similar i.e. they both refer to human, typically adult and seen as a neutral gendered description of a person. Even in simple cases such as this one it is trivially not true, *man* can even refer to a member of mankind (*god cares for all men*) an authority (*the man*) or can be used as an exclamation (*oh man*).      That sounds like a problem, what do we do to solve it? Glad you ask. Well there are multiple and probably more than I don't know of. One way is to concieve the other meanings as noise in the *gender component* and one way to remove it is to average over mulitple gender components (another one could be $w_{girl} - w_{boy}$), you could probably imagine others as well. Another way is to disambiguate the word embeddings (by created the word embedding for each different meaning of the word), not a trivial task, but one place where you can start is including the part-of-speech tag and thus distingushing between e.g. *desert* as verb or *desert* as a noun. Word sense disambiguation is still an active area of reasearch.   Plotting word embeddingsWe naturally can't plot a word embedding with an embedding size larger than three so what we can do instead is reduce the dimensionality e.g. using a PCA, which some of you might be familiar with.     What the hell is a PCA? It is apparently a part of experimental methods II, but I must admit I don't really recall hearing much about it either. PCA stands for principal component analysis and reduces a matrix (a bunch of datapoints) set of principal components which are features which seek to explain the most variance in the data while remaining uncorrelated. For instance imagine if you had height, gender, education, parents education as variables we know that there is a lot of redudancy of information (gender correlated with height and education with parents education), thus you could imagine reducing that to two components a *'gender'*-component and a *'social status'*-component. This is what PCA tries to do. If you want to know more there is a [20 minutes video](https://www.youtube.com/watch?v=FgakZw6K1QQ&t=2s) or a [5 minute video](https://www.youtube.com/watch?v=HMOI_lkzW08) both are by <NAME>, which some might find frustratingly pedagogic while other might (like me) find him hilarious.       Whouldn't it be better to use a another dimensionality reduction approach? Probably, PCA isn't really intended for plotting. T-SNE is a good alternative as it preserves local structure (but disregards global structure). However PCA is quick as constitutes a very solid baseline and as it is only for inspection I don't believe you loose much. I have made a function for it here which you are free to use, but do take a moment to read through it.<jupyter_code>def plot_word_embeddings(words, embedding):
    """Plot the words embedding in 2D using a scatter plot and PCA

    Args:
        words (List[str]): List of words
        embedding: Your word embedding as  gensim keyed vectors object.

    Returns:
        A matplotlib plot object
    """
    # extract word embedding matrix
    X = embedding[words]

    # apply a dimensionality reduction method of your choosing e.g. PCA
    from sklearn.decomposition import PCA
    pca = PCA(n_components=2)
    X_pca = pca.fit_transform(X)

    import pandas as pd
    df = pd.DataFrame(X_pca, index=words, columns=['x', 'y']) # create a dataframe for plotting

    # create a plot object
    import matplotlib.pyplot as plt
    fig = plt.figure()
    ax = fig.add_subplot(1, 1, 1)

    # add point for the words
    ax.scatter(df['x'], df['y'])

    # add word label to each point
    for word, pos in df.iterrows():
            ax.annotate(word, pos)
    
    return ax

words_to_plot = ["man", "woman", "queen", "king", "boy", "girl", "actor", "actress", "male", "female"]
ax = plot_word_embeddings(words=words_to_plot, embedding=word_emb)
ax.plot()<jupyter_output><empty_output><jupyter_text>Plan for class---- We will go through the `super().__init__()` to get an initial understanding.- Work on exercises- Last 15 minutes: present solutions<!-- Some exercises- most similar - king - man + woman = queen- NERNER (anonymization)- download a word embedding- visualizing word embeddings (PCA)find the most similar word embeddingsplan for class:    - We will go through the `super().__init__()` to get an initial understanding.    - Classify named entities -->  Understanding Super class initializationWe will go through this example at the start of class<jupyter_code>class person():
    def __init__(self, name, year_of_birth):
        self.name = name
        self.age = 2021-year_of_birth
    
    def say_hello(self):
        print(f"Hello my name is {self.name} and I am {self.age} years old")

class researcher(person):
    def __init__(self, loc, employment):
        self.loc = loc
        self.employment = employment
        
    def say_hello(self):
        print(f"Hello I am currently a {self.employment}, at {self.loc}")


kenneth = researcher("Aarhus", "phd")
kenneth.say_hello()
# problem 1:
    # I want to reuse code from the init of person without copy pasting
# problem 2:
    # I want to be able to call the say_hello function of the parent class
    # e.g. kenneth.say_ordinary_hello()<jupyter_output>Hello I am currently a phd, at Aarhus<jupyter_text>What does the pytorch nn.Module init then do? Well from the [source code](https://pytorch.org/docs/stable/_modules/torch/nn/modules/module.htmlModule) we find the following:```py    def __init__(self):        """        Initializes internal Module state, shared by both nn.Module and ScriptModule.        """        torch._C._log_api_usage_once("python.nn_module")        self.training = True        self._parameters = OrderedDict()        self._buffers = OrderedDict()        self._non_persistent_buffers_set = set()        self._backward_hooks = OrderedDict()        self._is_full_backward_hook = None        self._forward_hooks = OrderedDict()        self._forward_pre_hooks = OrderedDict()        self._state_dict_hooks = OrderedDict()        self._load_state_dict_pre_hooks = OrderedDict()        self._modules = OrderedDict()```Which notably sets a couple of internal states (noticably self.training=True), and log it to the pytorch API.  Exercises--- you are not required to solve these tasks in any particular order. Exercise 6 is more challenging than the rest so if you are up for a challenge go for it.1) Skim/read through the Gensim section to get an overview of how to work with word embeddings. 2) What is to woman what man to doctor (according to the word embedding)? Is this problematic?3) Discuss how you could use word embeddings to find pluralis of a word  - apply it to three cases  - plot these, does there seem to be a *plural component*?3) Discuss how you would find the antonym of a word  - apply it to three cases  - plot these, does there seem to be a *antonym component*?4) You can also use word embedding to detect the odd one out for instance which word does not belong in *Ross, Kenneth, Tim, glass*? You can do this simply using: `word_emb.doesnt_match("kenneth ross tim glass".split())`. Try this out on three samples.  - Typically word embeddings are quite intuitive, discuss or write down your best answer to how one might find the odd one out?     Answer The function works by:- taking the mean of all the word-embeddings.- calculate the similarity (typically using cosine-similarity) from the mean to each word- return the most dissimilar word (i.e. the one with the highest cosine-distance from that mean vector).Do note that while your answer might not be the same it might still work as there are multiple ways of doing this. For instance you could also add up the distance between each word and every other word and see which one is the furthest away from all other points.5) Which word embedding is closest to the word embedding of *corona*; *virus* or *beer*? Justify your answer before you test it using the word embedding.  - What does this tell you about word embeddings?6) Word embedding for token classification  - We will here use token embeddding for classifying part-of-speech tags (i.e. whether a word is a noun or a verb), but this approach could just as well be used to classify whether a word is named entity or whether it is positive or negative. We will here use the English dependency treebank. In the utilities section below I have provided some functions to get you started, but I recommend solving it in 3 steps:    1) transform the words to word embeddings    2) Train a classifier to predict the pos-tag using e.g. logistic regression    3) calculate the performance metrics on the held out test set (how well does it perform on unseen words?)  Utilities for tagging POS using word embeddingsThis contain a bunch of utility functions for getting started using word embeddings for classifying part of speech tags.Including a 1) logistic classifier (or a softmax classifier), 2) a function to load in the dataset as a list of token, pos-tag pairs, and 3) a code snippet for turning the label into one hot vectors.  You can read more about the individual [pos-tags](https://universaldependencies.org/docsv1/en/pos/all.html) here.<jupyter_code>import os

from spacy.training import Corpus
import spacy

# change this so it fits your file structure:
path = os.path.join("..", "data", "English Dependency Treeback", "en_ewt-ud-train.spacy")
# and use en_ewt-ud-dev.spacy and en_ewt-ud-test.spacy for the dev and test set.

def load_pos_data(path: str) -> list:
    corpus = Corpus(path)

    nlp = spacy.blank("en")

    def extract_token_label_pair(example):
        e = example.to_dict()
        tokens = e["token_annotation"]["ORTH"]
        entity_labels = e["token_annotation"]["POS"]
        return zip(tokens, entity_labels)
    train_samples = [(token, label) for example in corpus(nlp) for token, label in extract_token_label_pair(example)]

    return train_samples

train = load_pos_data(path)
# exploring the output
print(train[:10])

tokens, labels = list(zip(*train))
print(tokens[:10])
# convert labels to one hot
import pandas as pd
df = pd.get_dummies(labels)
y = df.to_numpy() # One hot representation of the labels
df.head()
import torch
import torch.nn as nn


class Logistic(nn.Module):
    def __init__(self, n_input_features: int, n_output_features: int):
        super().__init__()
        self.linear = nn.Linear(n_input_features, n_output_features)
        self.cost = nn.CrossEntropyLoss()

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return self.linear(x)
    
    def fit(self, X: torch.Tensor, y: torch.Tensor, epochs = 1000):
        optimizer = torch.optim.AdamW(self.parameters())

        for epoch in range(epochs):
            # forward
            y_hat = self.forward(X)

            # backward
            loss = self.cost(y_hat, y)
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()

            # print
            if (epoch + 1) % 100 == 0:
                print(f"epoch: {epoch+1}, loss = {loss.item():.4f}")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09032997834118996, 'fraction_numerical': 0.021212893362211747, 'mean_word_length': 4.107026675341574, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 36, 'https://': 7, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '286716', 'license': 'MIT', 'n_tokens_mistral': 4801, 'n_tokens_neox': 4458, 'n_words': 1993, 'path': 'syllabus/classes/class6/class6.ipynb', 'repo_name': 'rmxrmx/NLP-E21', 'size': '15704'}