<jupyter_start><jupyter_text>Homework 2 Word EmbeddingsWelcome to Homework 2! The homework contains several tasks. You can find the amount of points that you get for the correct solution in the task header. Maximum amount of points for each homework is _six_.The **grading** for each task is the following:- correct answer - **full points**- insufficient solution or solution resulting in the incorrect output - **half points**- no answer or completely wrong solution - **no points**Even if you don't know how to solve the task, we encourage you to write down your thoughts and progress and try to address the issues that stop you from completing the task.When working on the written tasks, try to make your answers short and accurate. Most of the times, it is possible to answer the question in 1-3 sentences.When writing code, make it readable. Choose appropriate names for your variables (`a = 'cat'` - not good, `word = 'cat'` - good). Avoid constructing lines of code longer than 100 characters (79 characters is ideal). If needed, provide the commentaries for your code, however, a good code should be easily readable without them :)Finally, all your answers should be written only by yourself. If you copy them from other sources it will be considered as an academic fraud. You can discuss the tasks with your classmates but each solution must be individual.**Important!:** **before sending your solution, do the `Kernel -> Restart & Run All` to ensure that all your code works.**  More than a wordSo far, we studied how we can represent separate words as dense vectors, also called _embeddings_. But what if we need to find similar documents? How do we represent a whole document, that can have thousands of words as a vector? In this homework, you will learn and try different techniques of document representation.But first, let's start with imports. In this homework, you are going to use [Gensim](https://radimrehurek.com/gensim/) library. The installation should not cause any troubles. However, **install Cython before installing Gensim!** This will allow Gensim to use efficient calculations and reduce the training time from several minutes to several seconds. Thus, the installation procedure should be the following:`pip install Cython``pip install gensim`<jupyter_code>import csv
from pathlib import Path
from collections import namedtuple, Counter
import numpy as np

from gensim.models.doc2vec import Doc2Vec, TaggedDocument

import spacy
nlp = spacy.load('en_core_web_sm')<jupyter_output><empty_output><jupyter_text>Task 0. Data (0 points)Load the data and get familiar with its structure. We provided all the functions for loading the dataset.The data is a compilation of movie plot synopses from IMDb and Wikipedia. You can find the original data [here](https://www.kaggle.com/cryptexcode/mpst-movie-plot-synopses-with-tags/datampst_full_data.csv), as well as the paper describing it: [<NAME>., <NAME>., <NAME>., & <NAME>. (2018). MPST: A corpus of movie plot synopses with tags. arXiv preprint arXiv:1802.07858.](https://www.aclweb.org/anthology/L18-1274/).The data was modified as follows: only `title`, `plot_synopsis`, `tags`, and `split` colums were left. All the synopses were tokenize, lemmatized, cleared from the stop words with Spacy. Final data was reduced to 1000 training samples and 500 test samples.Run the cells below to load the data and see an example of the first training sample:<jupyter_code>MovieInfo = namedtuple('MovieInfo', ['title', 'synopsis_preprocessed', 'synopsis_original', 'tags'])

data_path = Path('movie_plots.csv')

train_set, test_set = [], []

with data_path.open(encoding='utf-8', newline='') as csvfile:
    reader = csv.DictReader(csvfile)
    for row in reader:
        tags = [tag.strip() for tag in row['tags'].split(',')]
        movie_info = MovieInfo(
            row['title'], row['synopsis_preprocessed'], row['synopsis_original'], tags
        )
        if row['split'] == 'train':
            train_set.append(movie_info)
        elif row['split'] == 'test':
            test_set.append(movie_info)
print(f"Train set has {len(train_set)} entries")
print(f"Test set has {len(test_set)} entries")
print("Movie info example")
print("-" * 50)
print("Title:")
print(train_set[0].title)
print("-" * 50)
print("Preprocessed synopsis:")
print(train_set[0].synopsis_preprocessed)
print("-" * 50)
print("Original synopsis:")
print(train_set[0].synopsis_original)
print("-" * 50)
print("Tags:")
print(train_set[0].tags)<jupyter_output><empty_output><jupyter_text>Task 1. TF-IDF (4 points)TF-IDF is one of the techniques to represent a text as a vector. TF-IDF means Term Frequency - Inversed Document Frequency. The hyphen between two term does not mean substraction, the terms are multiplies with each other. For this approach, we are going to create a document matrix, that is going to have a TF-IDF value for each word in every document. For each document, this value is going to be different. Let's dive into this gradually, starting with the first part - term frequency. As you might've guesses from the name, term frequency shows us how many times each word appears in a document. But just a simple counter is going to prioritize very frequent words, like `is` or `me`, so we are going to normalize it. It gives us the following formula for the term frequency:`tf(t, d) = count of t in d / number of words in d`where `t` is the term, or a word, and `d` is a document.  Task 1.1. Build a vocab (0.5 points)Before we start let's build a vocabulary, containing all the unique words in our training set.Complete the `build_vocab()` function, so it takes a list of texts, each of them represented by a list of tokens (`texts = [['token1', 'token2', ... 'token_i'], ['token1', 'token2', ... 'token_i'], ..., ['token1', 'token2', ... 'token_i']]`) and returns a `dict` with a word as a key and its index as a number (`vocab = {'and': 0, 'me': 1, ..., 'zimbabwe': 546}`).<jupyter_code>def build_vocab(texts):
    ...
    return vocab
def get_preprocessed_synopses(data_set):
    return [entry.synopsis_preprocessed.split() for entry in data_set]
vocab = build_vocab(get_preprocessed_synopses(train_set))
print("Total length of the vocab:", len(vocab))<jupyter_output><empty_output><jupyter_text>Task 1.2. Count TF (1 point)With the vocabulary built, construct a term frequency matrix. This matrix should have a shape of `(number of documents, size of vocab)` and contain a count for each term. For example, is a word `cat` is seen 12 times in the document 100, you will have `tf[100, vocab['cat']] = 12`.Complete the `tf()` function below and make sure you get the correct output.<jupyter_code>def tf(texts, vocab):
    # Store the TF value here
    tf = np.zeros((len(texts), len(vocab)))
    
    # Store the length of each document here
    doc_lens = np.zeros(len(texts))
    
    ...
    
    tf = tf / doc_lens[:, np.newaxis]
    return tf
train_tf = tf(get_preprocessed_synopses(train_set), vocab)

print("Shape of the train TF matrix should be (1000, 33537):", train_tf.shape)
print("Sum of all TFs for the first synopsis should be 1:", np.sum(train_tf[0, :]))
print()

test_tf = tf(get_preprocessed_synopses(test_set), vocab)

print("Shape of the train TF matrix should be (500, 33537):", test_tf.shape)
print("Sum of all TFs for the first synopsis should be 1:", np.sum(test_tf[0, :]))<jupyter_output><empty_output><jupyter_text>Task 1.3. Normalize your TF (0.5 points)Good news is that we can already find similar documents with only our TF matrix! We can do it by calculating a cosine similarity between the vectors of two documents. In our case, the vectors for the documents are the rows of the TF matrix that contain the frequency value for each term. But as we learned with word embeddings, to make the cosine value be relevant, we need to normalize all the vectors, i.e. make the all of length 1.To normalize a vector, we need to divide each value by the length of the vector. Please, use the [`np.linalg.norm`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.norm.html) function to find the length of the vector. Read the documentation and choose the correct `axis` value.When dividing the `tf` matrix by our lengths vector, you might want to add `[:, np.newaxis]` to avoid shape mismatching errors (see the `tf()` function).Complete the `normalize()` function below:<jupyter_code>def normalize(tf):
    return ...
train_tf_norm = normalize(train_tf)
test_tf_norm = normalize(test_tf)<jupyter_output><empty_output><jupyter_text>Task 1.4. Cosine similarity (0.5 points)Finish the `cosine_similarity()` function. It should take one vector as a `query` argument, the matrix of vectors for each document as a `pool` argument, and the number of the most similar documents to return as a `k` argument. For each document in the pool we caclulate a cosine similarity with the query, which is a dot product of the corresponding two vectors.The return value of the function should the a numpy array of length `k`, containing the indices for the most similar documents sorted in descending order, i.e. from the most similar document to the least similar one. In this fucntion you might want to use `np.argsort` and `np.dot` functions.<jupyter_code>def cosine_similarity(query, pool, k=10):
    return ...<jupyter_output><empty_output><jupyter_text>Sanity check of our `cosine_similarity()` function. We check the most similar documents to the first document in our training set. Obviously, the same document should be the most similar. The output should be something like this: `array([  0, 177, 187, 344, 407, 518, 766, 861, 255, 316], dtype=int64)`. If the first element is not `0`, it means that you did something wrong during the previous steps!<jupyter_code>cosine_similarity(train_tf_norm[0, :], train_tf_norm)<jupyter_output><empty_output><jupyter_text>We are also going to introduce a very naive and simple method to evaluate the quality of our predictions. For each predicted text, we are going to look at its tags and mark it as correct if at least one tag in the query and prediction is the same.<jupyter_code>def accuracy(pred, gold):
    assert len(pred) == len(gold)
    correct = 0
    total = 0
    for i in range(len(pred)):
        total += 1
        if any(x in pred[i] for x in gold[i]):
            correct += 1
    return correct / total<jupyter_output><empty_output><jupyter_text>Let's see the accuracy for our TF classifier:<jupyter_code>tf_pred = []
gold = []
for i, test_entry in enumerate(test_tf_norm):
    most_similar = cosine_similarity(test_entry, train_tf_norm, k=1)
    gold.append(test_set[i].tags)
    tf_pred.append(train_set[most_similar[0]].tags)
print(accuracy(tf_pred, gold))<jupyter_output><empty_output><jupyter_text>Task 1.5. Count DF (1 point)Let's proceed to the second part of the TF-IDF which is inversed document frequency. But before we inverse something, let's find the document frequency first. We define the document frequency as `df(t) = occurrence of t in documents`. To find it, we can use a simple but very slow approach of iterating through each document and comparing each word in it against the vocabulary. There is an alternative way to do it really fast, only using the `tf` matrix that we already constructed above. The `df()` function should return an array of length of our `vocab`, where for each respective word index, we have a value representing the number of documents with word was seen in.Try to find a `df` array with one-liner. Hint: You may want to use `np.sum()` function. Also, remember that you can use equality operators (``, `==`) with numpy arrays.The correct one-line solution will get 1 point, the slow iterative solution with only get 0.5 points!<jupyter_code>def df(tf):
    return ...
train_df = df(train_tf)<jupyter_output><empty_output><jupyter_text>Your `train_df` array should look like this: `array([25,  2,  6, ...,  3,  1, 31])`. You will probably have different numbers in the array since the initialization of the dictionary is different on your machine.<jupyter_code>train_df<jupyter_output><empty_output><jupyter_text>Task 1.6. Count IDF (0.5 points)We almost finished in our journey into TF-IDF land. Let's finally calculate the inversed document frequency. The formula for it is `idf(t) = N/df` where `N` is the total number of documents and `df` is the document frequency that we calculated before. But if `N` is some really large number, our IDF term will become huge. Thus, we are going to take a `log` of it. Finally, the formula that we are going to use is: `idf(t) = log(N/(df + 1))`. `log` here is a natural logarithm, i.e. with base `e`.If we try to think what IDF does, it gives more weight to rare words in the document collection and less to the words that appear in many documents. The assumtion is that the first type of words and my characteristic of a document that the second type.The final IDF vector should be of the length of our vocabulary.Implement the `idf()` function below:<jupyter_code>def idf(texts, df):
    return ...
train_idf = idf(get_preprocessed_synopses(train_set), train_df)
print(train_idf.shape)<jupyter_output><empty_output><jupyter_text>Finally, TF-IDF is just a multiplication of two terms that we calculated before:<jupyter_code>def tfidf(tf, idf):
    return tf * idf
train_tfidf = tfidf(train_tf, train_idf)
train_tfidf_norm = normalize(train_tfidf)<jupyter_output><empty_output><jupyter_text>Sanity check again. The most similar document should be `0`. If it's different, check your previous code for mistakes!<jupyter_code>cosine_similarity(train_tfidf_norm[0, :], train_tfidf_norm)<jupyter_output><empty_output><jupyter_text>Since we want to compare test texts, which are unseen, we need to have the same vocabulary size as our train matrix. Thus, we are going to calculate the TF term for each test text and we are going to use the IDF term from our training set. For now, we are just ignoring the words that are in the test set but not in the train set.<jupyter_code>test_tfidf = tfidf(test_tf, train_idf)
test_tfidf_norm = normalize(test_tfidf)
tfidf_pred = []
for i, test_entry in enumerate(test_tfidf_norm):
    most_similar = cosine_similarity(test_entry, train_tfidf_norm, k=1)
    tfidf_pred.append(train_set[most_similar[0]].tags)
    
print(accuracy(tfidf_pred, gold))<jupyter_output><empty_output><jupyter_text>Task 2. Doc2Vec (1 point)TF-IDF maybe a good solution for short query but it is most likely inefficient for long queries. This is because TF-IDF is a bag-of-words (BOW) algorithm. As the name suggests, all the words are just compiled in out vocabulary, like in a bag, and the order in which they appear in text does not affect the resulting vector. Thus `Mary likes John` and `John likes Mary` are going to have identical TF-IDF vectors. Another problem is that is the size of the vocabulary is large, we are going to have many zeros that don't represent anything useful. Another algorithm is Doc2Vec. This algorithm was described by the same people that invented Word2Vec in their paper [<NAME>., & <NAME>. (2014, January). Distributed representations of sentences and documents. In International conference on machine learning (pp. 1188-1196).](https://cs.stanford.edu/~quocle/paragraph_vector.pdf).The main idea is very similar to the Word2Vec model. Actually, the models are almost identical, with the only difference being a paragraph matrix added to the input. This matrix is going to be the embedding for the document. In this homework, we are going to train a Doc2Vec model using Gensim package.First, we need to prepare train and test sets. Train set should be a collection of `TaggedDocument` objects, that contain the text and the index of each text. This index is going to correspond to the additional paragraph matrix row, containing the document vector. The test set is just a list of texts.<jupyter_code>doc2vec_train = []

for i, text in enumerate(get_preprocessed_synopses(train_set)):
    doc2vec_train.append(TaggedDocument(text, [i]))
    
doc2vec_test = [text for text in get_preprocessed_synopses(test_set)]<jupyter_output><empty_output><jupyter_text>Next, to train the model, we just need to specify hyperparameters, build the vocabulary and run the `train()` function. Your task is to read the [Doc2Vec documentation](https://radimrehurek.com/gensim/models/doc2vec.htmlgensim.models.doc2vec.Doc2Vec), [Doc2Vec tutorial](https://radimrehurek.com/gensim/auto_examples/tutorials/run_doc2vec_lee.htmlsphx-glr-auto-examples-tutorials-run-doc2vec-lee-py), and the paper above. Then, find the hyperparameters that, in your opinion, suit best our dataset.**Leave the seed as 1234!**Try to achieve the accuracy > 0.55<jupyter_code>model = Doc2Vec(..., seed=1234)
model.build_vocab(doc2vec_train)
model.train(doc2vec_train, total_examples=model.corpus_count, epochs=model.epochs)<jupyter_output><empty_output><jupyter_text>Sanity check that the most similar text in the training set is the text itself. The first index should be `0`.<jupyter_code>vector = model.infer_vector(doc2vec_train[0].words)
model.docvecs.most_similar([vector])
doc2vec_pred = []
for i, test_entry in enumerate(doc2vec_test):
    vector = model.infer_vector(test_entry)
    most_similar = model.docvecs.most_similar([vector])
    doc2vec_pred.append(train_set[most_similar[0][0]].tags)<jupyter_output><empty_output><jupyter_text>Try to achieve the accuracy > 0.55<jupyter_code>print(accuracy(doc2vec_pred, gold))<jupyter_output><empty_output><jupyter_text>Task 3. Write your own plot (1 point)Try to come up with your own movie plot. Then run three different algorithms (TF, TF-IDF, Doc2Vec) to find similar plots from the collection. Your plot should not be very long, three sentences should be enough unless your imagination strives for more :)Answer the following question:**In your opinion, which algorithm performed better in this task? Explain your choice.**Your answer here<jupyter_code># Put your title here
my_title = "The Last Meow"

# Put your plot here
my_plot = """It's been a long time since humans think they rule over the cats. 
They've been teasing and humiliating them for too long. 
Now, the time has come for the cat army to rise and destroy the humanity once and for all.""" 
def preprocess(text):
    doc = nlp(text)
    preprocessed_text = synopsis = [token.lemma_.lower() for sent in doc.sents for token in sent 
                    if not token.is_stop and not token.is_punct and not token.is_space]
    return preprocessed_text
plot_preprocessed = preprocess(my_plot)
my_plot_tf = tf([plot_preprocessed], vocab)
print("Similar plots by TF:\n")
print("Author's title:", my_title)
print("Author's plot:", my_plot)
print("-" * 50)
for idx in cosine_similarity(normalize(my_plot_tf)[0], train_tf_norm, k=3):
    print(f'Candidate {idx}:')
    title = train_set[idx].title
    print('Title:', title)
    plot = train_set[idx].synopsis_original
    print('Plot:', plot)
    tags = train_set[idx].tags
    print('Tags:', tags)
    print("-" * 50)
my_plot_tfidf = tfidf(my_plot_tf, train_idf)
print("Similar plots by TF-IDF:\n")
print("Author's title:", my_title)
print("Author's plot:", my_plot)
print("-" * 50)
for idx in cosine_similarity(normalize(my_plot_tfidf)[0], train_tf_norm, k=3):
    print(f'Candidate {idx}:')
    title = train_set[idx].title
    print('Title:', title)
    plot = train_set[idx].synopsis_original
    print('Plot:', plot)
    tags = train_set[idx].tags
    print('Tags:', tags)
    print("-" * 50)
my_plot_vector = model.infer_vector(plot_preprocessed)
print("Similar plots by TF-IDF:\n")
print("Author's title:", my_title)
print("Author's plot:", my_plot)
print("-" * 50)
for idx in model.docvecs.most_similar([vector], topn=3):
    idx = idx[0]
    print(f'Candidate {idx}:')
    title = train_set[idx].title
    print('Title:', title)
    plot = train_set[idx].synopsis_original
    print('Plot:', plot)
    tags = train_set[idx].tags
    print('Tags:', tags)
    print("-" * 50)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09259074092590741, 'fraction_numerical': 0.012548745125487452, 'mean_word_length': 4.879776601998824, 'pattern_counts': {'":': 0, '<': 87, '<?xml version=': 0, '>': 90, 'https://': 7, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '390690', 'license': 'MIT', 'n_tokens_mistral': 6153, 'n_tokens_neox': 5818, 'n_words': 2491, 'path': 'homeworks/hw2/Homework2.ipynb', 'repo_name': '501Good/tartu-nlp-2020', 'size': '20026'}