<jupyter_start><jupyter_text>Traditional NLP  wordNetwordNet - a thesaurus containing lists of synonym sets and hypernyms ("is a" relationships).But WordNet never works that well.- missed nuances- missed new words (wicked, badass, ninja, etc.)- subjective- requires human labro to create and adapt- can't compute accurate word similarity- in additional with one-hot-encoding it gives very huge space of similarity synonym sets containing "good"<jupyter_code>poses = {
    'n': 'noun',
    'v': 'verb',
    's': 'adj (s)',
    'a': 'adj',
    'r': 'adv'
}

for synset in wn.synsets('good'):
    pos = poses[synset.pos()]
    lemas = ', '.join(l.name() for l in synset.lemmas())
    print(f'{pos}: {lemas}')<jupyter_output>noun: good
noun: good, goodness
noun: good, goodness
noun: commodity, trade_good, good
adj: good
adj (s): full, good
adj: good
adj (s): estimable, good, honorable, respectable
adj (s): beneficial, good
adj (s): good
adj (s): good, just, upright
adj (s): adept, expert, good, practiced, proficient, skillful, skilful
adj (s): good
adj (s): dear, good, near
adj (s): dependable, good, safe, secure
adj (s): good, right, ripe
adj (s): good, well
adj (s): effective, good, in_effect, in_force
adj (s): good
adj (s): good, serious
adj (s): good, sound
adj (s): good, salutary
adj (s): good, honest
adj (s): good, undecomposed, unspoiled, unspoilt
adj (s): good
adv: well, good
adv: thoroughly, soundly, good<jupyter_text>e.g hypernyms of "panda"<jupyter_code>panda = wn.synset('panda.n.01') # lemma, pos, synset_index_str 
hyper = lambda s: s.hypernyms()
list(panda.closure(hyper))<jupyter_output><empty_output><jupyter_text>Distributional symanticsA word's meaning is given by the words that frequently appear close-by> "You shall know a word by the company it keeps" (<NAME> 1957)build a sense vector for each word, chosen so that it's similar to vectors of words that appear in similar context.`Word vectors` -> `word embeddings`, or `word representation`  Word2VecMikolov 2013_`corpus` one body of text, `corpora` multiple bodies of text, because it is latin_For a center word $c$ and a context word $o$$$P(o|c) = \frac{\exp(u_0^T v_c)}{\sum_{W \in{V}}{\exp(u_w^T v_c)}}$$- $v_w$ when $w$ is a center word- $u_w$ when $w$ is a context wordsimilarty between words / normalize over entire vocab and give probability distribution.It is softmax function, which gives prob distribution$$\frac{\delta}{\delta{v_c}} \log{P(o|c)} = u_0 - \sum_{x \in{V}} p(x|c) u_x$$$u_0$ obsorbe representation of context word and we subtruction from that the context should look like (other words on their probability) - expected context wordTODO: try to do the same for $u_c$ (context word vector)  Gensimstarts from LDA-style (Latent Dirichlet allocation) topics models and grew into SVD (Singular value decomposition) and neural word representation ME: lets use pretrained model from gensim [repos](https://github.com/RaRe-Technologies/gensim-data). We use the same model which is used in lectures<jupyter_code>import gensim.downloader as api

info = api.info()  # show info about available models/datasets
model = api.load('glove-wiki-gigaword-100')  # download the model and return as object ready for use
model.most_similar('cat')
def display_pca_scatterplot(model, words=None, sample=0):
    if words == None:
        if sample > 0:
            words = np.random.choice(list(model.vocab.keys()), sample)
        else:
            words = [ word for word in model.vocab ]
        
    word_vectors = np.array([model[w] for w in words])

    twodim = PCA().fit_transform(word_vectors)[:,:2]
    
    plt.figure(figsize=(6,6))
    plt.scatter(twodim[:,0], twodim[:,1], edgecolors='k', c='r')
    for word, (x,y) in zip(words, twodim):
        plt.text(x+0.05, y+0.05, word)
display_pca_scatterplot(model, 
                        ['coffee', 'tea', 'beer', 'wine', 'brandy', 'rum', 'champagne', 'water',
                         'spaghetti', 'borscht', 'hamburger', 'pizza', 'falafel', 'sushi', 'meatballs',
                         'dog', 'horse', 'cat', 'monkey', 'parrot', 'koala', 'lizard',
                         'frog', 'toad', 'monkey', 'ape', 'kangaroo', 'wombat', 'wolf',
                         'france', 'germany', 'hungary', 'luxembourg', 'australia', 'fiji', 'china',
                         'homework', 'assignment', 'problem', 'exam', 'test', 'class',
                         'school', 'college', 'university', 'institute'])
rn.sample([word for word in model.vocab if 'woman' in word], 10)
# display_pca_scatterplot(model, rn.sample([word for word in model.vocab if 'bio' in word], 10))
display_pca_scatterplot(model, [
    'man','woman',
    'king', 'queen',
    'brother', 'sister',
    'nephew', 'niece',
    'prince', 'princess',
    'sir', 'madam',
    'uncle', 'aunt',
])<jupyter_output><empty_output><jupyter_text>- Skip Grams (SG) - predict context words give center word- Continues Bag of Words (CBOW) - predict center word from (bag of) context words- Negative sampling vs Naive softmax> Main idea: train binary logistic regressions for a true pair (center word and word in its context window) versus several noise pairs (the center word paired with a random word)$$J_{neg - sample}(o, v_c, U) = - \log{(\sigma(u_0^T v_c)} - \sum_{k=1}^{K}\log{(\sigma(-u_k^T v_c))}$$- take k negative samples (using word probabilities)- maximize probability that real outside word appears, minimize prob. that random words appear around center word- $P(w)=U(w)^{3/4} / Z$, where normalization term $Z = \sum U(w_i)$. What makes unigram distribution $U(w)$ raised to the 3/4 power, and makes less frequent words be sampled more often  Count based technicsWord2vec technics similar to traditional- `window`- create matrix of count how many times each word co-occurrence with other and captures both syntactic (POS) andsemantic information- `document` leads to Latent Semantic Analysis dimensionality reductionuse can use Singular Value Decomponsition (SVD) of co-occurrence matrix X. And get only few the most important k singular values, in order to generalize. But it is expensive to cumpute for big matrixes like X  Example<jupyter_code>sentenses = [
    'I like deep learning',
    'I like NLP',
    'I enjoy flying',
    'I like swimming'
]

words = list({w for s in sentenses for w in s.split()})
print(words)
tuples = [(w1, w2) for s in sentenses for w1, w2 in zip(s.split(), s.split()[1:])]
print(tuples)
X = np.zeros((len(words), len(words)))
for w1, w2 in tuples:
    X[words.index(w1), words.index(w2)]+=1
    X[words.index(w2), words.index(w1)]+=1    
X<jupyter_output><empty_output><jupyter_text>now is time for singular value decomposition<jupyter_code>U, s, Vh = np.linalg.svd(X, full_matrices=False)
plt.figure(figsize=(6,6))
print(s)
plt.plot(s);<jupyter_output>[ 3.59290673e+00  3.59290673e+00  1.15708206e+00  1.15708206e+00
  8.67284475e-01  8.67284475e-01  5.60343953e-17 -0.00000000e+00]<jupyter_text>lets draw it for 2 the most important singular values<jupyter_code>plt.figure(figsize=(6,6))
coords = np.array([[U[idx, 0], U[idx, 1]] for idx, word in enumerate(words)])
plt.scatter(coords[:,0], coords[:,1], edgecolors='k', c='r')
for idx, word in enumerate(words):
    plt.text(U[idx, 0] - 0.005, U[idx, 1] + 0.02, word)<jupyter_output><empty_output><jupyter_text>how can we improve it- the problem of frequent words (the, he, has,...) which have too much impact. use "sealing" function $min(X,t)$, with $t \approx 100$, or ignore them- ramped windows that count closer words more- use Pearson correlations instead of counts, then set negative values to 0 Analogy as test coin to Rohde (2005)  Count based vs direct prediction Ccount based- LSA, HAL,- COALS, Hellinger-PCA- pros:    - Fast train    - Effcient usage of statistics- cons:    - Primarily used to capture word similarity    - Disproportionate importance given to large counts direct prediction- Skip-gram/CBOW- NNLM, HLBL, RNN- procs:    - Generate improved performance on other tasks    - Can capture complex patterns beyond word similarity    - could train fast and parallel- cons:    - Scales with corpus size    - Inefficient usage of statistics  Encoding meaning in vector differences Combine both count based and direct predictionRatios of co-occurrence probabilities can encode meaning components. Divide probability different words in context of word `a` and word `b`. How to capture ratios of co-occurrence probabilities as linear meaning components in a word vector space?Log-bilinear model. Probability of word `i`, given word `j`:$$w_i \cdot w_j = \log{P(i|j)}$$then we can have vector differences between words `a` and `b`$$w_x \cdot (w_a - w_b) = \log{\frac{P(x|a)}{P(x|b)}}$$ GloVe$$J = \sum_{i,j=1}^{V}f(X_{ij}) (w_i^T \tilde{w}_j + b_i + \tilde{b}_j - \log{X_{ij}})^2$$- Fast training- scalable to huge corpa- Good performance even with small corpus and small vectorsSo dot product of $w$ should be close similar to $X$ co-occurance probability, but with some bies for both words based on popularity.$f$ should capping $X$ on some max value.  How to evaluate word vectors?- intrinsic    - Evaluation on a specific/intermediate subtask    - Fast to compute    - Helps to understand that system    - Not clear if really helpful unless correlation to real task is established- extrinsic    - Evaluation on a real task    - Can take a long time to compute accuracy    - Unclear if the subsystem is the problem or its interaction or other subsystems    - If replacing exactly one subsystem with another improves accuracy Winning!    - eg. named entity recognition: finding a person, organization or location  Intrinsic word vector evalution Word vector analogiescosine distance and angle between vectors of words`a:b :: c:?`(`man:woman :: king:?`)$$d = argmax_{i} \frac{(x_b - x_a + x_c)^T x_i}{||x_b - x_a + x_c||}$$- Evaluate word vectors by how well their cosine distance after addition captures intuitive semantic and syntactic analogy questions- Trick: Discarding the input words from the search!- Problem: What if the information is there but not linear?<jupyter_code>def display_pca_scatterplot_of_pair_words(model, word_analogies, figsize=(6,6)):
    def get_seq(a):
        if isinstance(a, dict):
            return a['seq']
        else:
            return a
    words = [w for analog_sequence in word_analogies for w in get_seq(analog_sequence)]
    word_vectors = np.array([model[w] for w in words])

    twodim = PCA().fit_transform(word_vectors)[:,:2]
    
    plt.figure(figsize=figsize)
    plt.scatter(twodim[:,0], twodim[:,1], edgecolors='k', c='r')
    for word, (x, y) in zip(words, twodim):
        plt.text(x+0.05, y+0.05, word)

    for analog_sequence in word_analogies:
        if isinstance(analog_sequence, dict):
            color = analog_sequence['color']
            analog_sequence = analog_sequence['seq']
            print('custom seq', color)
        else:
            color = 'green'
        coords = np.array(list(zip(*(twodim[words.index(w)].tolist() for w in analog_sequence))))
#         idx_a = words.index(w_a)
#         idx_b = words.index(w_b)
# #         Arrow doesn't look nice as simple dash lines
# #         x_a, y_a = twodim[idx_a]
# #         x_b, y_b = twodim[idx_b]
# #         plt.arrow(x_a, y_a, x_b - x_a, y_b - y_a, head_width=0.05, head_length=0.1)
#         coords = list(zip(twodim[idx_a], twodim[idx_b]))
        plt.plot(
            coords[0],
            coords[1],
            scalex=False, scaley=False,
            color=color, marker='o', linestyle='dashed'
        )

display_pca_scatterplot_of_pair_words(model, [
    ('man','woman'),
    ('king', 'queen'),
    ('brother', 'sister'),
    ('nephew', 'niece'),
    ('prince', 'princess'),
    ('sir', 'madam'),
    ('uncle', 'aunt'),
])
plt.title('analogies graph, based on "glove-wiki-gigaword-100" model')
display_pca_scatterplot_of_pair_words(model, [
#     ('small', 'smaller', 'smallest'),
    ('slow', 'slower', 'slowest'), 
    ('short', 'shorter', 'shortest'),
#     ('strong', 'stronger', 'strongest'),    
    ('soft', 'softer', 'softest'),     
    ('dark', 'darker', 'darkest'),
    ('smart', 'smarter', 'smartest'),
    ('loud', 'louder', 'loudest'),
    # irregular comparatives don't do as well as regular
    {'seq': ('good', 'better', 'best'), 'color': 'orange'},
    {'seq': ('little', 'less', 'least'), 'color': 'red'}
])
plt.title('superlatives graph, based on "glove-wiki-gigaword-100" model')<jupyter_output>custom seq orange
custom seq red<jupyter_text>We have something different with irregular comparative and superlative forms<jupyter_code>display_pca_scatterplot_of_pair_words(model, [
    ('good', 'better', 'best'),
    ('little', 'less', 'least'),
    ('bad', 'worse', 'worst'),
    ('far', 'further', 'furthest'),
    {'seq': ('far', 'farther', 'farthest'), 'color': 'blue'}, # different form
])
plt.title('superlatives graph, based on "glove-wiki-gigaword-100" model')<jupyter_output>custom seq blue<jupyter_text>Data set with analogies `word2vec questions-words.txt` but it could be googled, for example here .The problem is different cities may have same name  lets take a look on other prebuilt models from Gensiminitially I have used `glove-wiki-gigaword-100` _Wikipedia 2014 + Gigaword 5 (6B tokens, uncased)_.But there are alternatives (they are much bigger):- `glove-wiki-gigaword-300` with bigger vector size _Wikipedia 2014 + Gigaword 5 (6B tokens, uncased)_- `fasttext-wiki-news-subwords-300` _Wikipedia 2017, UMBC webbase corpus and statmt.org news dataset (16B tokens)_- `word2vec-google-news-300` _Google News (about 100 billion words)_- `conceptnet-numberbatch-17-06-300` _ConceptNet, word2vec, GloVe, and OpenSubtitles 2016_<jupyter_code>fasttext_wiki_news_subwords_300_model = api.load('fasttext-wiki-news-subwords-300')
fasttext_wiki_news_subwords_300_model.most_similar('cat')
glove_wiki_gigaword_300_model = api.load('glove-wiki-gigaword-300')
glove_wiki_gigaword_300_model.most_similar('cat')<jupyter_output>/usr/local/lib/python3.6/site-packages/smart_open/smart_open_lib.py:398: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function
  'See the migration notes for details: %s' % _MIGRATION_NOTES_URL<jupyter_text>hm, on irregular words look even worse on big "fasttext-wiki-news-subwords-300"<jupyter_code>display_pca_scatterplot_of_pair_words(fasttext_wiki_news_subwords_300_model, [
    ('good', 'better', 'best'),
    ('little', 'less', 'least'),
    ('bad', 'worse', 'worst'),
    ('far', 'further', 'furthest'),
    {'seq': ('far', 'farther', 'farthest'), 'color': 'blue'}, # different form
], figsize=(9,9))
plt.title('superlatives graph, based on "fasttext-wiki-news-subwords-300" model')
display_pca_scatterplot_of_pair_words(fasttext_wiki_news_subwords_300_model, [
#     ('small', 'smaller', 'smallest'),
    ('slow', 'slower', 'slowest'), 
    ('short', 'shorter', 'shortest'),
    ('strong', 'stronger', 'strongest'),    
    ('soft', 'softer', 'softest'),     
    ('dark', 'darker', 'darkest'),
    ('smart', 'smarter', 'smartest'),
#     ('loud', 'louder', 'loudest'),
    # irregular comparatives don't do as well as regular
    {'seq': ('good', 'better', 'best'), 'color': 'orange'},
    {'seq': ('little', 'less', 'least'), 'color': 'red'}
], figsize=(9,9))
plt.title('superlatives graph, based on "fasttext-wiki-news-subwords-300" model')
display_pca_scatterplot_of_pair_words(glove_wiki_gigaword_300_model, [
    ('good', 'better', 'best'),
    ('little', 'less', 'least'),
    ('bad', 'worse', 'worst'),
    ('far', 'further', 'furthest'),
    {'seq': ('far', 'farther', 'farthest'), 'color': 'blue'}, # different form
])
plt.title('superlatives graph, based on "glove-wiki-gigaword-300" model')
display_pca_scatterplot_of_pair_words(glove_wiki_gigaword_300_model, [
#     ('small', 'smaller', 'smallest'),
    ('slow', 'slower', 'slowest'), 
    ('short', 'shorter', 'shortest'),
#     ('strong', 'stronger', 'strongest'),    
    ('soft', 'softer', 'softest'),     
    ('dark', 'darker', 'darkest'),
    ('smart', 'smarter', 'smartest'),
    ('loud', 'louder', 'loudest'),
    # irregular comparatives don't do as well as regular
    {'seq': ('good', 'better', 'best'), 'color': 'orange'},
    {'seq': ('little', 'less', 'least'), 'color': 'red'}
], figsize=(9,9))
plt.title('superlatives graph, based on "glove-wiki-gigaword-300" model')<jupyter_output>custom seq orange
custom seq red
{'fraction_non_alphanumeric': 0.14423773858419908, 'fraction_numerical': 0.01812031891761295, 'mean_word_length': 4.276290630975144, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 40, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '94476', 'license': 'MIT', 'n_tokens_mistral': 5806, 'n_tokens_neox': 5457, 'n_words': 1715, 'path': 'src/notebooks/01-intro.ipynb', 'repo_name': 'hyzhak/CS224n', 'size': '16560'}