<jupyter_start><jupyter_text>Learning word embeddings - word2vec\- [<NAME>](https://saurabhmathur96.github.io/) The aim of this experiment is to use the algorithm developed by [*<NAME> et al.*](http://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf) to learn high quality vector representations of text.  The skip-gram model Given,a sequence of words $ w_1, w_2, .., w_T $, predict the next word.The objective is to maximize average log probability.$$ AverageLogProbability = \frac{1}{T} \sum_{t=1}^{T} \sum_{-c \leqslant j\leqslant c, j \neq 0} log\ p (w_{t+j} | w_t) $$where $ c $ is the length of context.  Basic skip-gram modelThe basic skip-gram formulation defines $ p (w_{t+j} | w_t) $ in terms of softmax as -$$ p (wo | wi) = \frac{ exp(v'^{T} _{wo} \cdot v_{wi})  }{ \sum^{W}_{w=1}  exp(v'^{T} _{w} \cdot v_{wi} ) } $$where $vi$ and $vo$ are input and output word vectors.This is extremely costly and this impractical as, W is huge ( ~ $10^5-10^7$ terms ).   There are three proposed methods to get around this limitation.- Heirarchial softmax- Negative sampling- Subsample frequent words I'm using Google's Tensorflow library for the implementation<jupyter_code>import tensorflow as tf<jupyter_output><empty_output><jupyter_text>For the data, I'm using the [text8 dataset](http://mattmahoney.net/dc/textdata) which is a 100MB sample of cleaned English Wikipedia dump on Mar. 3, 2006<jupyter_code>import os, urllib

    
def fetch_data(url):
    
    filename = url.split("/")[-1]
    datadir = os.path.join(os.getcwd(), "data")
    filepath = os.path.join(datadir, filename)
    
    if not os.path.exists(datadir):
        os.makedirs(datadir)
    if not os.path.exists(filepath):
        urllib.urlretrieve(url, filepath)
    
    return filepath

url = "http://mattmahoney.net/dc/text8.zip"
filepath = fetch_data(url)
print ("Data at {0}.".format(filepath))
    

import os, zipfile

def read_data(filename):
    with zipfile.ZipFile(filename) as f:
        data = tf.compat.as_str(f.read(f.namelist()[0])).split()
    return data


words = read_data(filepath)<jupyter_output><empty_output><jupyter_text>Take only the top $c$ words, mark rest as UNK (unknown).<jupyter_code>def build_dataset(words, vocabulary_size):
    count = [[ "UNK", -1 ]].extend(
        collections.Counter(words).most_common(vocabulary_size))
    word_to_index = { word: i for i, (word, _) in enumerate(count) }
    data = [word_to_index.get(word, 0) for word in words] # map unknown words to 0
    unk_count = data.count(0) # Number of unknown words
    count[0][1] = unk_count
    index_to_word= dict(zip(word_to_index.values(), word_to_index.keys()))
    
    return data, count, word_to_index, index_to_word

vocabulary_size = 50000
data, count, word_to_index, index_to_word = build_dataset(words, vocabulary_size)
print ("data: {0}".format(data[:5]))
print ("count: {0}".format(count[:5]))
print ("word_to_index: {0}".format(word_to_index.items()[:5]))
print ("index_to_word: {0}".format(index_to_word.items()[:5]))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14147286821705427, 'fraction_numerical': 0.016472868217054265, 'mean_word_length': 4.471731448763251, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '114094', 'license': 'MIT', 'n_tokens_mistral': 1123, 'n_tokens_neox': 1069, 'n_words': 288, 'path': 'Learning word embeddings - word2vec.ipynb', 'repo_name': 'saurabhmathur96/jupyter-notebook', 'size': '3111'}