<jupyter_start><jupyter_text>Statistical Methods For Comparisons  Contents1. [Goal](Goal)1. [Motivations](Motivations)1. [Defining a clearer space](Defining-a-clearer-space)    1. [Defining Length](Defining-length)        1. [L-1](L-1)        1. [L-2](L-2)    1. [What is Distance?](Distance)        1. [Cosine](Cosine)1. [Evaluation Update](Quick-Evaluation-Update)1. [Nearest Neighbors](Nearest-Neighbors)<jupyter_code>import utils
import vsm
import pandas as pd
import numpy as np
import os
import random
W = pd.read_csv('giga5.csv',index_col=0) #read in raw giga word dataset
DATA_HOME = os.path.join('data/data', 'wordrelatedness')
eval_df = pd.read_csv( # read in evaluation dataset
    os.path.join(DATA_HOME, "wordrelatedness-dev.csv"))
giga5 = pd.read_csv('giga5.csv',index_col=0)
def distance2pred(pred_df): # haven't added these to vsm.py yet
    lis = [-1*i for i in pred_df['prediction']]
    pred_df['prediction'] = pd.Series(lis)
    return pred_df
def random_scorer(x1, x2):
    return random.random()<jupyter_output><empty_output><jupyter_text>GoalWe've seen how W, a set of word vectors derived from raw co-occurance counts seems to carry an interesting embedding space. This is our initial VSM. Before capturing nuance, let's first focus on getting the basics right. That can be done by making more use of the initial data. Doing so gives us our first real baseline on the word evaluation test.<jupyter_code>pred_df, score = vsm.word_relatedness_evaluation(eval_df, giga5, distfunc=vsm.cosine)
random_pred_df, random_score = vsm.word_relatedness_evaluation(eval_df, giga5, distfunc=random_scorer)
pred_df = distance2pred(pred_df)
pred_df['prediction'] = abs(1 - pred_df['prediction'])
pred_df['percent error'] = abs(pred_df['score']-pred_df['prediction'])*100
pred_df.rename(columns={'score': 'relatedness score'},inplace=True)
pred_df.rename(columns={'prediction': 'vsm prediction'},inplace=True)
print(f'VSM Score Percent: {round(score*100,2)} % similar to human answers')
print(f'Random Score Percent: {round(abs(random_score*100),2)} % similar to human answers')
print('Glimpse of how our VSM answered')
pred_df[59:90]<jupyter_output>VSM Score Percent: 27.76 % similar to human answers
Random Score Percent: 0.71 % similar to human answers
Glimpse of how our VSM answered<jupyter_text>MotivationsThe significant jump in score from blind guessing reflects the basic meaning these word embedding take on in a well-defined space. Scrolling through the lowest error predictions shows how it has begun capturing basic connections. Let's apply some statistical thinking to this space to obtain the above result.  Defining a clearer space  Defining length L-1 norm, Magnitude If we want to treat the colorful embeddings in Tableau as true vectors, we'll have to define them so they're more appropriate in vector space. As of now, the identity of each embedding comes from the co-occurance values present in its components (the band widths). In the Trio viz, notice the summed co-occurance values are unique for each embedding.The Y-Axis for 'age' tops out at 500k while 'old' is almost twice as large. These values represent the embedding's *magnitude*, the the sum of absolute values. It might make sense to use magnitude, Length-1(L1), as our notion of length.- L-1$$\|u\|_{1} = {\sum_{i=1}^{w} |{u_{i}|}}$$ L-2 norm, LengthImagine, however, we're looking at two words that are roughly synonyms. How about 'cookie' and 'biscuit'? We can be idealistic and assume there's no noise for now. Since they share so many of the same usages, as we run along their rows and compare the two, we'd find the co-occurance values (band widths) to be pretty much the same at each component. However, they're **not** identical. If I wanted to dip a treat in tea, and asked a Brit to bring some biscuits, I might wind up with soggy Oreos. We want to capture and amplify these slight differences in usage. Instead of summing the co-occurances at face values, we'll sum the *squares* of each component and take the square root at the end. This notion of length, L-2, is also more compatible with statistical Least Squares methods and happens to be the standard Linear Algebra definition of vector length.- L-2  $$\|u\|_{2} = \sqrt{\sum_{i=1}^{w} u_{i}^{2}}$$  Defining DistanceThere are even more ways to think about distance. We could treat the space like our physical reality and use euclidean distance. There's the grid-like Manhattan distance, and probability-oriented KL divergence. The slew of options reflects what makes ML interesting but also why new problems can feel overwhelming. The decisions, from matrix design to hyperparameter choices, are not formulaic. Instead, they're informed by the domain and your downstream tasks. Since ours is to make meaningful vector comparisons, we want to start addressing the noise from common words. The cliff between high and medium usage dwarfes a lot of nuance. We can normalize the word vectors so the components are expressed as a percentage of its length (like comparing by per capita). Then we could just use euclidean distance to express space between normalized vectors.- Normalization of vector *u* with **w** components$$\textbf{normalize}(u_w) =\left[   \frac{u_{1}}{\|u\|_{2}},   \frac{u_{2}}{\|u\|_{2}},   \ldots   \frac{u_{w}}{\|u\|_{2}}  \right]$$- Euclidean Distance between vectors *u* and *v* $$\textbf{euclidean}(u, v) = \sqrt{\sum_{i=1}^{w}|u_{i} - v_{i}|^{2}}$$ The end result is a space in which the components of our word vectors have more meaning, since they share a common reference. Moreover, instead of just length as an identity, the *orientation* of these vectors in euclidean space have meaning. We can sort of think of them as arrows in 3-D space pointing in different directions. CosineIf you are more familiar with Linear Algebra, you might notice we can achieve the same effect in one transformation. We take the dot product of the two vectors (giving us euclidean-like orientation), control for length by dividing this value by the product of their norms (normalizing them). That's the cosine distance between the two. Instead of talking about how far apart things are, it makes more sense to talk about how similar they are. So we simply flip the script and subtract the result from 1.- *Cosine Distance* $$\textbf{Cosine}(u, v) =  \frac{\sum_{i=1}^{w} u_{i} \cdot v_{i}}{\|u\|_{2} \cdot \|v\|_{2}}$$- **Cosine Similarity**$$\textbf{CosineSim}(u, v) = 1 - \frac{\sum_{i=1}^{w} u_{i} \cdot v_{i}}{\|u\|_{2} \cdot \|v\|_{2}}$$ SummaryI'll use Cosine Similarity since it's more practical, but it's the same exact thing as normalizing and taking Euclidean distance.    Nearest NeighborsSince our vectors have meaningful orientation now, we can start doing what Semantle (https://semantle.novalis.org/) does. We can search for a word in our VSM and inspect the embeddings that are closest to it! Our hope is that words sharing a local space have something in common. It's a quick way to get a glimpse of what the landscape looks like. Nearest neighbors for various Age and Beauty are shown below.<jupyter_code>pd.DataFrame(vsm.neighbors('age',(W))).head(10)
pd.DataFrame(vsm.neighbors('beauty',(W))).head(10)<jupyter_output><empty_output><jupyter_text>As you can see, the closest words for each concept are not exactly inspiring. We address this in the next notebook, *Leveraging Probability*.Here's a sneak peak at at where this work will take us.<jupyter_code>pd.DataFrame(vsm.neighbors('age',vsm.pmi(W))).head(10)
pd.DataFrame(vsm.neighbors('beauty',vsm.pmi(W))).head(10)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0982460767506264, 'fraction_numerical': 0.011604905710141104, 'mean_word_length': 5.086677367576244, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '337288', 'license': 'MIT', 'n_tokens_mistral': 2281, 'n_tokens_neox': 2132, 'n_words': 977, 'path': 'Statistical Methods (pg. 2).ipynb', 'repo_name': 'Jelan1/Vector-Space-Model', 'size': '7583'}