<jupyter_start><jupyter_text>Dataset Details - Standford's GloVe pre-trained word vectorsGloVe is an unsupervised learning algorithm for obtaining vector representations for words. Training is performed on aggregated global word-word co-occurrence statistics from a corpus. The GloVe pre-trained word vectors dataset contains English word vectors pre-trained on the combined Wikipedia 2014 + Gigaword 5th Edition corpora (6B tokens, 400K vocab). All tokens are in lowercase. This dataset contains 50-dimensional, 100-dimensional and 200-dimensional pre trained word vectors. In this problem we are going to use the 50-dimensional dataset.   \ 0. Get an overview on what Glove isRead up the documentation on glove embeddings, esp. where it gets applied here: https://nlp.stanford.edu/projects/glove/  Load DatasetLet's load the dataset first. Each row is indexed as a word vector. Dimension of word vectors is 50. How many words are there in this dataset? Print a few words and see what they are. You don't need to code anything here, just understand the data structure.<jupyter_code>import pandas as pd
import numpy as np
import sys
import csv

csv.field_size_limit(sys.maxsize)
# Load GloVe pre-trained vectors 
# local_file1="./glove/glove.6B.50d.txt" # Make sure this file exists!
filePATH = '/content/drive/MyDrive/Machine_Learning/HW4/glove.6B.50d.txt' 
df = pd.read_csv(filePATH,sep=' ', index_col=0, header=None, engine='python', on_bad_lines = 'skip', quoting = csv.QUOTE_NONE)
print("Dataset shape - Rows: %d, Cols: %d" % (df.shape[0], df.shape[1]))

words = list(df.index)
print("print a few words in the dataset:", words[30:40])<jupyter_output>Dataset shape - Rows: 400000, Cols: 50
print a few words in the dataset: ['be', 'has', 'are', 'have', 'but', 'were', 'not', 'this', 'who', 'they']<jupyter_text>\ 1. Print the first few 11 rows of the pandas data frame below<jupyter_code># Your code HERE - It should execute as expected! 
# (Search for a pandas functionality that can help you do this!)
df.head(11)<jupyter_output><empty_output><jupyter_text>\ 2. Words SimilaritySimilar words have similar embeddings (or vector values). We can use cosine similarity i.e. cos(u,v) = u.v/(|u||v|) to measure vector similarity. u.v is dot product of vectors, |u| is L2 norm of u. Remember, we spoke about computing similarity based on cosine-similarity (as another alternative to correlation) in class?1. Normalize matrix df by norm of word vectors. 1. Define a function to find words similarity to a given word.1. Use the function defined to find the word in examples that is most similar to "happy".<jupyter_code># Calculate norm of word vectors
vector_norm = np.sqrt(np.square(df).sum(axis=1))

# What would be the dimension of the vector_norm array?
print("length of vector norm:", len(vector_norm), ", Each Vector will contribute towards an element of norm vector")
# Normalize matrix df by norm using .div()
dfn = df.div(vector_norm, axis = 0)
print("dimension of normalised df: ", dfn.shape)
# Define a function to find words similar to a given word in a normalized dataframe
def find_word_similarity(word, examples,dataframe):
    """
    Input: word - one string
    examples - List of strings
    dataframe - An indexed normalized dataframe
    """
    ## YOUR CODE HERE
    dotwithexample = dataframe.loc[examples].dot(dataframe.loc[word])
    dotwithexample2 = dotwithexample.sort_values(ascending = False)
    # Calculate dot product of each word in examples to the given word, sorted by value high to low
    # Once you have the sorted values of dot products (notice because of normalization, the dot product is the cosine similarity!),
    # obtain the words corresponding to the sorted values and call it similar_words
    similar_words = dotwithexample2

    # Return words similar to the given word
    return similar_words
    
examples = ["sad", "bad", "evil", "healthy", "ill",
            "beaming", "cheerful", "joyful", "radiant", "glad", "upset",
            "disco", "probably", "hardly", "ephemeral", "close", "cleaning", 
            "maths", "word", "distribution"]

# Use above function to calculate examples' similarity to happy (both "happy" and words in examples are in dfn)
print ( find_word_similarity("happy", examples,dfn) )
examples = ["sad", "bad", "evil", "healthy", "ill",
            "beaming", "cheerful", "joyful", "radiant", "glad", "upset",
            "disco", "probably", "hardly", "ephemeral", "close", "cleaning", 
            "maths", "word", "distribution"]

from sklearn.metrics.pairwise import cosine_similarity
similarity2 = pd.DataFrame(cosine_similarity(np.array(df.loc['happy']).reshape(1,50),np.array(df.loc[examples])),columns = examples)
similarity2 = similarity2.T.sort_values(by=[0], ascending=False)
print(similarity2)<jupyter_output>0
glad          0.865877
hardly        0.816272
probably      0.747581
bad           0.708395
sad           0.689063
healthy       0.640579
word          0.599150
cheerful      0.575719
upset         0.566424
joyful        0.555032
close         0.554029
ill           0.522978
evil          0.452148
disco         0.324669
beaming       0.289510
cleaning      0.246022
distribution  0.160149
radiant       0.134971
ephemeral     0.132886
maths        -0.011991<jupyter_text>\ 3. Goodness of similarityComment on the how good the glove embeddings are on finding similar words to a given word using cosine similarity? Glove and word2vec embeddings are based on co-occurence of words in senetences across hundreds of thousands of documents on the web. Would this help explain your observations on word similarity?What if you replace happy with sad, how do the results change? **Interpretation:**Maths got negative similarity with happy. In general, it is often seen that most of the people don't like math or have troubles with math. This similarity seems appropriate. similarly, glad got maximum similarity with happy which also makes sense. The similarity between happy - (hardly and sad) establishes the fact the Glove considers co-occurences in global and well as local corpus of words. sentiments are not factored in this approach of similarity.<jupyter_code>examples1 = ["happy", "bad", "evil", "healthy", "ill",
            "beaming", "cheerful", "joyful", "radiant", "glad", "upset",
            "disco", "probably", "hardly", "ephemeral", "close", "cleaning", 
            "maths", "word", "distribution"]
print ( find_word_similarity("sad", examples1,dfn) )<jupyter_output>0
hardly          0.707888
happy           0.689063
bad             0.664580
glad            0.644321
joyful          0.587249
word            0.559652
probably        0.535477
cheerful        0.531766
upset           0.501137
ill             0.487844
evil            0.401493
healthy         0.372976
disco           0.362746
close           0.311398
radiant         0.244188
ephemeral       0.234799
beaming         0.194717
cleaning        0.135319
maths           0.030129
distribution    0.008951
dtype: float64<jupyter_text>similarity between math and sad increased which again justify the co-occurence situation. similarity of disco with sad is greater than with happy, this might be because people tend to go to a disco when they are sad and hence, these words are used more frequently. Cleaning has higher similarity to Happy. **Takeaway:** Similarity using GloVe embedding find the words which have similar connotation but doesn't captures the sentiment invloved with the words.   \ 4. Correlations(This question is more of a reflection and building your intuition on how correlations we spoke in class connects to a real-world data set -  Open ended!)What are some of the most correlated words from the similarity search you did earlier to the word "happy" and "sad". Likewise, what are some of the most uncorrelated words to "happy" and "sad". Does this make sense? How would you improve the results ? If "happy" were a random variable and "sad" was a random variable - What factors make the correlation between "happy" and "sad" (as you computed above) high?  Most similar words to "happy" are(>70): 1. glad2. hardly 3. probably 4. bad Most uncorrelated words to "happy" are(<0.15): 1. maths2. ephemeral  3. radiant  4. distribution Most similar words to "sad" are(>60): 1. hardly2. happy 3. bad4. glad Most uncorrelated words to "sad" are(<0.15): 1. distribution 2. maths3. cleaning some of these are doesn't make sense as they represents similar category of feeling but they don't necessarily represents same feeling. But vectors in GloVe embedding can be great starting point. 1. It can also be improved by increasing the dimensinality of the each vector. 2. Results of glove embedding can be improved by factoring sentiments in the similarity calculation. If "Happy" and "sad" where random variables, 2 factors will result in high correaltion: 1. Probability of co-occurence. They belong to same category "emotions". so, there is high chance of using these words in a particular context. 2. It is also imperative that, given a sentence happy and sad can be used interchangably to represent two different emotions. example, student was happy with the output of his code. student was sad with the output of his code. all the other words remain same around happy and sad.   \ 5. Find nearest neighbourhoodIt is helpful to compute the nearest neighbors to a word based on the cosine similarity that we defined earlier, so that given a word we can compute which are the other words which are most similar to it. Sometimes, the nearest neighbors according to this metric reveal overlap of concepts or topics that a word shares. E.g. government might be related to the word politics because they both share topics related to public policy, politicians, parties, elections, etc. The idea is whatever embeddings we are using - word2vec or glove is "hopefully" able to capture these correlations right!1. Define a function to find the top n similar words to a given word. You can use either dot product of vectors or cosine_simialrity function. Note the search space for words is coming from your pandas data frame (so unlike the similarity problem we worked on earlier, we are not restricted to only a few words to search from - the search space here is the entire vocab captured in your data frame).1. Find 20 nearest neighborhood for words 'duck' and 'animal'.1. Find neighborhood intersection of 'duck' and 'animal', to find which words are similar to both 'duck' and 'animal'. This is also related to a similarity measure called "Jaccard Similarity" - Read up on this here: https://en.wikipedia.org/wiki/Jaccard_index<jupyter_code># define a function to find the top n similar words to a given word in the 'df'

# PART 1
def find_most_similar(df, word, n):
    """
    INPUT: 
    df: Given Data frame
    word: String
    n: Number of similar words to return
    OUTPUT:
    the list of similar words to return
    """
    ## YOUR CODE HERE
    # define and compute the most similar words
    # Use a similarity measure like cosine similarity (like earlier) to do so
    df = df.loc[~df.index.duplicated()]
    words = np.array(df.index)
    similarity = pd.DataFrame(cosine_similarity(np.array(df.loc[word]).reshape(1,50),np.array(df.loc[words])),columns = words)
    similarity = similarity.T.sort_values(by=[0], ascending=False)
    similarity_sorted = similarity.head(n)
    
    return similarity_sorted


# PART 2
# find top 40 similar words to duck
simil1 = find_most_similar(df, "duck", 40)
# find top 40 similar words to animal
simil2 = find_most_similar(df, "animal", 40)

# PART 3
# find the intersection of simil1 and simil2
#intersection =  (concat function of pandas is helpful here)
combined = pd.concat([simil1, simil2], axis=1)
combined.columns = ['duck', 'animal']
combined_final = combined.dropna(how='any')
print (combined_final)

word_labels = ['duck', 'animal'] + list(combined_final.index)
print("\nword labels are:\n", word_labels)<jupyter_output>duck    animal
pig   0.739596  0.735997
cat   0.703251  0.684206
dog   0.693291  0.725226
fish  0.676019  0.728633
meat  0.655953  0.675570
bird  0.643547  0.800324

word labels are:
 ['duck', 'animal', 'pig', 'cat', 'dog', 'fish', 'meat', 'bird']<jupyter_text>\ 6 Word analogiesSuppose you know the word vectors for King, Man and Woman. What is your intuitive answer for the 'riddle', King - Man + Woman = ? Let's go through below steps to derive the answer for this 'riddle' using the word embeddings.1. Use vector arithmetic to define a new vector which equals to k - m + w (e.g. king, man and woman combination).2. Calculate similarity of all the words in the corpus to the new vector and sort them by their similarity high to low. 3. Return the top n vectors which have the highest similarity to the new vector.1. Find the answers for the riddles,     1. good:bad::up:?    1. germany:merkel::america:?<jupyter_code>def solve_riddle(x, y, a, n, dataframe):
    ## YOUR CODE HERE
    # calculate the vector of a + y - x, where a, x, y are in dataframe
    vec_a = np.array(dataframe.loc[a])
    vec_x = np.array(dataframe.loc[x])
    vec_y = np.array(dataframe.loc[y])

    cal_vec = vec_a + vec_y - vec_x
    # calculate distance of words in dataframe to cal_vec, sorted by similarity high to low
    df = dataframe.loc[~dataframe.index.duplicated()]
    df = df.drop([a, x, y])
    words = np.array(df.index)
    similarity = pd.DataFrame(cosine_similarity(np.array(cal_vec).reshape(1,50),
                                                np.array(df.loc[words])),columns = words)
    similarity = similarity.T.sort_values(by=[0], ascending=False)
    similarity_sorted = similarity.head(n)

    # return top n words and distance that closest to cal_vec
    return similarity_sorted 

# Call the solve_riddle function to compute the top answers
print ("King - Man + Woman =\n", solve_riddle("man", "woman", "king", 5,df) )

## YOUR CODE HERE
# Solve the other two riddles
# good:bad::up:?
# # PART 4
print ("\ngood:bad::up:?\n", solve_riddle("good", "bad", "up", 5,df) )

# # germany:merkel::america:?
print ("\ngermany:merkel::america:?\n",solve_riddle("germany", "merkel", "america", 5,df))<jupyter_output>King - Man + Woman =
                  0
queen     0.860958
daughter  0.768451
prince    0.764070
throne    0.763497
princess  0.751273

good:bad::up:?
                  0
down      0.850167
falling   0.813844
out       0.792837
dropping  0.782064
off       0.778428

germany:merkel::america:?
                 0
obama    0.694289
barack   0.682594
hillary  0.660910
bush     0.657911
clinton  0.655765
{'fraction_non_alphanumeric': 0.08724878249537005, 'fraction_numerical': 0.042664105905754854, 'mean_word_length': 3.726094003241491, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 22, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '552486', 'license': 'MIT', 'n_tokens_mistral': 4736, 'n_tokens_neox': 4222, 'n_words': 1817, 'path': 'SolvePuzzleUsingGlove/PuzzlesWithGlove.ipynb', 'repo_name': 'krmayankb/MachineLearning', 'size': '14579'}