<jupyter_start><jupyter_text>Word Embedding![Word Embedding](https://cdn-images-1.medium.com/max/800/0*g24VvkPOJPaYDw6W.jpg)Photo Credit: https://cdn.pixabay.com/photo/2016/03/09/09/14/books-1245690_960_720.jpg Word Embedding is silver bullet to resolve many NLP problem. Most of modern NLP architecture adopted word embedding and giving up bag-of-word (BoW), Latent Dirichlet Allocation (LDA), Latent Semantic Analysis (LSA) etc. After reading this article, you will understand:- History of Word Embedding- Word Embedding Design- Apply off-the-shelf word embedding model- Embedding Visualization- Take Away  History of Word EmbeddingTraditionally, we use bag-of-word to represent a feature (e.g. TF-IDF or Count Vectorize). Besides BoW, we can apply LDA or LSA on word feature. However, they have some limitations such as high dimensional vector, sparse feature. Word Embedding is a dense feature in low dimensional vector. It is proved that word embedding provides a better vector feature on most of NLP problem.In 2013, Mikolov et al. made Word Embedding popular. Eventually, word embedding is state-of-the-art in NLP. He released the word2vec toolkit and allowing us to enjoy the wonderful pre-trained model. Later on, gensim provide a amazing wrapper so that we can adopt different pre-trained word embedding models which including Word2Vec (by Google), GloVe (by Stanford), fastText (by Facebook).12 years before Tomas et al. introduces Word2Vec, Bengio et al. published a paper [1] to tackle language modeling and it is the initial idea of word embedding. At that time, they named this process as "learning a distributed representation for words". ![](https://cdn-images-1.medium.com/max/800/1*FZVMHwCLO3fFo7FvMyA94Q.png)Capture from A Neural Probabilistic Language Model [2] (Benigo et al, 2003) In 2008, Ronan and Jason [3] introduce a concept of pre-trained model and showing that it is a amazing approach for NLP problem. Word embedding became famous unitl Tomas released pre-trained model (Word2Vec) in 2013. ![](https://cdn-images-1.medium.com/max/800/1*D6A44ZN5_zwTyuCAODM0fA.png)Capture from A Unified Architecture for Natural Language Processing [3] (Collobert & Weston, 2008) Timeline:- 2001: Bengio et al. introduced a concept of word embedding- 2008: Ronan and Jason  introduced a concept of pre-trained model- 2013: Mikolov et al. released pre-trained model which is Word2Vec  Word Embedding Design Low Dimensional![](https://food.fnr.sndimg.com/content/dam/images/food/fullset/2014/3/17/0/FNM_040114-KidsCake-rainbow-recipe_s4x3.jpg.rend.hgtvcom.616.462.suffix/1395082987380.jpeg)Photo Credit: https://www.foodnetwork.com/recipes/food-network-kitchen/four-layer-birthday-cake-3363221To tackle the high dimensional issue, word embedding use pre-defined vector space such as 300 to present every word. For demo purpose, I use 3 dimension to represent the following words:- Apple: [1.11, 2.24, 7.88]- Orange: [1.01, 2.04, 7.22]- Car: [8.41, 2.34, -1.28]- Table: [-1.41, 7.34, 3.01]As pre-defined the vector space (i.e. 3 in the above demo), number of dimension (or feature) is fixed no matter how large the corpus is. Comparing to BoW, number of dimension will be increased when unique word increase. Imagining we have 10k unique words in our documents, number of feature in BoW is 10k (without filtering high/ low frequency word) while the dimension can be keep as 3 in our demo. Semantic Relationship![](https://cdn-images-1.medium.com/max/1600/1*oF1QyMamN5jXCXfffSRrqA.png)Photo Credit: https://gointothestory.blcklst.com/similar-but-different-c722f39d923dIn general, the word vector encodes semantic relationship among words. It is a very important concept on word embedding as it benefits on tacking NLP problem. Word vectors will be closed if they have similar meaning. For example, buy and purchase will be closer. Unlike BoW, it only represent 0 or 1 (Counting having a word or not approach) and it cannot represent whether two wordings have similar meaning or not.In the above example, you may notice that Apple's vector and Orange's vector are closed than others meanwhile Apple's vector is far way from Car's vector relatively. Continuous bag-of-words (CBOW) & Skip-gramMikolov et al proposed two new architectures [4] which reducing computation complexity and including additional context. CBOW is that using both n words before and after target word (w). For instance, "the word vector encodes semantic relationship among words". If the window (n) is 3, here is the subset of prediction list:- Case 1, Before Words: {Empty}, After Words: (word, vector, encodes), Predict Word: "the"- Case 2, Before Words: (the), After Words: (vector, encodes semantic), Predict Word: "word"Skip-gram uses the opposite approach which use the target word to predict n words before and after target word. For instance, "the word vector encodes semantic relationship among words". If the window (n) is 3, here is the subset of prediction list:- Case 1, Predict Word: "the", Words: (word, vector, encodes)- Case 2, Predict Word: "word", Words: (the, vector, encodes, semantic)![](https://cdn-images-1.medium.com/max/800/1*QwiTOcVmwesADjQ3zMvSjA.png)Capture from Efficient Estimation of Word Representations in Vector Space (Tomas et al., 2013)Negative Sampling Negative SamplingInstead of leveraging all other words as negative label training records. Mikolov et al. proposed to use suitable small amount of negative training record to train the model. So that the whole operation become much faster. If you are not familiar with negative sampling, you may check out this article for more information.  Apply off-the-shelf word embedding modelIntroduced history and model architecture, how can we use word embedding to tackle NLP problem?There are two approaches to deal with word embedding:- Leveraging off-the-shelf model- Building a domain specific model.This article will take the first approach. Selecting 3 well-known pre-trained models and leveraging gensim to load those model. Gensim, well known NLP library, already implement interface to deal with these 3 models.<jupyter_code>!pip install tensorflow
import datetime
import numpy as np
import os

import gensim
from gensim.test.utils import datapath, get_tmpfile
from gensim.scripts.glove2word2vec import glove2word2vec

import tensorflow as tf
from tensorflow.contrib.tensorboard.plugins import projector

print('gensim Version: %s' % (gensim.__version__))

class WordEmbedding:
    __author__ = "<NAME>"
    __copyright__ = "Copyright 2018, <NAME>"
    __credits__ = ["<NAME>"]
    __license__ = "Apache"
    __version__ = "2.0"
    __maintainer__ = "<NAME>"
    __email__ = "<EMAIL>"

    def __init__(self, verbose=0):
        self.verbose = verbose
        
        self.model = {}
        
    def convert(self, source, ipnut_file_path, output_file_path):
        if source == 'glove':
            input_file = datapath(ipnut_file_path)
            output_file = get_tmpfile(output_file_path)
            glove2word2vec(input_file, output_file)
        elif source == 'word2vec':
            pass
        elif source == 'fasttext':
            pass
        else:
            raise ValueError('Possible value of source are glove, word2vec, fasttext')
        
    def load(self, source, file_path):
        print(datetime.datetime.now(), 'start: loading', source)
        if source == 'glove':
            self.model[source] = gensim.models.KeyedVectors.load_word2vec_format(file_path)
        elif source == 'word2vec':
            self.model[source] = gensim.models.KeyedVectors.load_word2vec_format(file_path, binary=True)
        elif source == 'fasttext':
            self.model[source] = gensim.models.wrappers.FastText.load_fasttext_format(file_path)
        else:
            raise ValueError('Possible value of source are glove, word2vec, fasttext')
            
        print(datetime.datetime.now(), 'end: loading', source)
            
        return self
    
    def get_model(self, source):
        if source not in ['glove', 'word2vec', 'fasttext']:
            raise ValueError('Possible value of source are glove, word2vec, fasttext')
            
        return self.model[source]
    
    def get_words(self, source, size=None):
        if source not in ['glove', 'word2vec', 'fasttext']:
            raise ValueError('Possible value of source are glove, word2vec, fasttext')
        
        if source in ['glove', 'word2vec']:
            if size is None:
                return [w for w in self.get_model(source=source).vocab]
            else:
                results = []
                for i, word in enumerate(self.get_model(source=source).vocab):
                    if i >= size:
                        break
                        
                    results.append(word)
                return results
            
        elif source in ['fasttext']:
            if size is None:
                return [w for w in self.get_model(source=source).wv.vocab]
            else:
                results = []
                for i, word in enumerate(self.get_model(source=source).wv.vocab):
                    if i >= size:
                        break
                        
                    results.append(word)
                return results
        
        return Exception('Unexpected flow')
    
    def get_dimension(self, source):
        if source not in ['glove', 'word2vec', 'fasttext']:
            raise ValueError('Possible value of source are glove, word2vec, fasttext')
        
        if source in ['glove', 'word2vec']:
            return self.get_model(source=source).vectors[0].shape[0]
            
        elif source in ['fasttext']:
            word = self.get_words(source=source, size=1)[0]
            return self.get_model(source=source).wv[word].shape[0]
        
        return Exception('Unexpected flow')
    
    def get_vectors(self, source, words=None):
        if source not in ['glove', 'word2vec', 'fasttext']:
            raise ValueError('Possible value of source are glove, word2vec, fasttext')
        
        if source in ['glove', 'word2vec', 'fasttext']:
            if words is None:
                words = self.get_words(source=source)
            
            embedding = np.empty((len(words), self.get_dimension(source=source)), dtype=np.float32)            
            for i, word in enumerate(words):
                embedding[i] = self.get_vector(source=source, word=word)
                
            return embedding
        
        return Exception('Unexpected flow')
    
    def get_vector(self, source, word, oov=None):
        if source not in ['glove', 'word2vec', 'fasttext']:
            raise ValueError('Possible value of source are glove, word2vec, fasttext')
            
        if source not in self.model:
            raise ValueError('Did not load %s model yet' % source)
        
        try:
            return self.model[source][word]
        except KeyError as e:
            raise
            
            #TODO
#             if oov is None:
#                 raise
            
#             if 'not in vocabulary' in str(e):
#                 if oov == ''

    def build_visual_metadata(self, embedding, words, file_dir, 
                              metadata_name='metadata.csv', project_model_name='model.ckpt'):
        # Create output directory if not exist
        if not os.path.exists(file_dir):
            os.makedirs(file_dir)

        # Build graph
        tf.reset_default_graph()
        sess = tf.InteractiveSession()

        embedding_graph = tf.Variable([0.0], name='embedding')
        place = tf.placeholder(tf.float32, shape=embedding.shape)

        set_embedding_graph = tf.assign(embedding_graph, place, validate_shape=False)
        sess.run(tf.global_variables_initializer())
        sess.run(set_embedding_graph, feed_dict={place: embedding})

        # Build metadata
        with open(os.path.join(file_dir, metadata_name), 'w') as f:
            for word in words:
                f.write(word + '\n')

        # Build projector
        summary_writer = tf.summary.FileWriter(file_dir, sess.graph)
        config = projector.ProjectorConfig()
        embedding_conf = config.embeddings.add()
        embedding_conf.tensor_name = 'embedding:0'
        embedding_conf.metadata_path = metadata_name
        projector.visualize_embeddings(summary_writer, config)

        # Save model
        saver = tf.train.Saver()
        saver.save(sess, os.path.join(file_dir, project_model_name))

        # Clear
        sess.close()

        
downloaded_glove_file_path = '../text/stanford/glove/glove.6B.50d.txt'
glove_file_path = '../text/stanford/glove/glove.840B.300d.vec'

word2vec_file_path = '../text/google/word2vec/GoogleNews-vectors-negative300.bin'
fasttext_file_path = '../text/facebook/fasttext/wiki.en.bin'

word_embedding = WordEmbedding()
# You may need to convert text file (downloaed from GloVe website) to vector format
# word_embedding.convert(
#      source='glove', ipnut_file_path=downloaded_glove_file_path, output_file_path=glove_file_path)<jupyter_output><empty_output><jupyter_text>Word2Vec[Word2Vec](https://code.google.com/archive/p/word2vec/) is trained on google news and provided by Google. Based on 100 billion words from Google News data, they trained model with 300 dimensions.Mikolov et al. use skip-gram and negative sampling to build this model which is released in 2013. GloVeGlobal Vectors for Word Representation ([GloVe](https://nlp.stanford.edu/projects/glove/)) is provided by Stanford NLP team. Stanford provides various models from 25, 50 , 100, 200 to 300 dimensions base on 2, 6, 42,  840 billion tokens.Stanford NLP team apply word-word co-occurrence probability to build the embedding. In other word, if two words are co-exist many time, both words may have similar meaning so the matrix will be closer. fastText[fastText](https://fasttext.cc/) is released by Facebook which provides 3 models with 300 dimensions. One of the pre-trained model is trained with subword. For example, "difference", it will be trained by "di", "dif", "diff" and so on.<jupyter_code>word_embedding.load(source='word2vec', file_path=word2vec_file_path)
word_embedding.load(source='glove', file_path=glove_file_path)
word_embedding.load(source='fasttext', file_path=fasttext_file_path)
for source in ['glove', 'word2vec', 'fasttext']:
    print('Source: %s' % (source))
    print(word_embedding.get_vector(source=source, word='apple'))
source = 'word2vec'

embedding = word_embedding.get_vectors(source=source)
words = word_embedding.get_words(source=source)
sub_embedding = embedding[:100000]
sub_words = words[:100000]<jupyter_output><empty_output><jupyter_text>Embedding VisualizationOne of state-of-the-art NLP is word embedding, what is it actually? It is a matrix and the simplest way is x and y coordinate but we have 300 dimensions not 2 dimensions. We can visualize it by using principal component analysis (PCA) or T-distributed Stochastic Neighbor Embedding (t-SNE). By leveraging TensorBoard, visualization can be presented easily.<jupyter_code>word_embedding.build_visual_metadata(embedding=sub_embedding, words=sub_words, file_dir='./word_embedding')<jupyter_output><empty_output><jupyter_text>![](https://cdn-images-1.medium.com/max/800/1*glwOAs3oK5IOOegT9ru7sw.png)<jupyter_code>"""
    To start the tensorboard.
    1. Open terminal
    2. Go to parent directory of file_dir (e.g. parent directory of word_embedding)
    3. execute "tensorboard --logdir=word_embedding" (e.g. the value of --logdir should be same 
        as what your provide in previous step)
    4. Open browser to access http://localhost:6006 (depending on your host, the default port is 6006)
"""<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0888719608963372, 'fraction_numerical': 0.022598870056497175, 'mean_word_length': 3.2395048439181915, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 24, 'https://': 13, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '300637', 'license': 'Apache-2.0', 'n_tokens_mistral': 4780, 'n_tokens_neox': 4427, 'n_words': 1566, 'path': 'SCG/nlp-word_embedding.ipynb', 'repo_name': 'talk2sunil83/UpgradLearning', 'size': '15778'}