<jupyter_start><jupyter_text>---word2vec=== ---Pop Quiz---Do computers prefer numbers or words?__Numbers__word2vec is currently the best algorithm to map words (strings) to numbers (vectors of floats). ---By The End Of This Notebook You Should Be Able To:---- Describe why word2vec is popular and powerful- Explain how word2vec is a neural network- Understand the common architectures of word2vec- Apply word vectors to "do math" on words ----Why is word2vec so popular?----1. Organizes word by semantic meaning.2. Turns text into a numerical form that Deep Learning Nets and machine learning algorithms can in-turn use. ----How does word2vec work?---- >“You shall know a word>by the company it keeps”> \- <NAME> 1957 ---Distributional Hypothesis---> Words that are used and occur in the same contexts tend to have similar meanings  Example:> ... government debt problems are turning into __banking__ crises...  > ... Europe governments needs unified __banking__ regulation to replace the hodgepodge of debt regulations...The words: _government_, _regulation_ and _debt_ probably represent some aspect of _banking_ since they frequently appear near by.The words: _Pokeman_ and _tublar_ probably don't represent some aspect of _banking_ since they don't frequently appear near by. ----How does word2vec model the Distributional Hypothesis?---word2Vec is a very simple neural network:Input = text corpus  Output = vector for each word ----word2vec as a compression algorithm-----Note the bow-tie shape. That is is an __autoencoder__. Autoencoders compress sparse representations into dense representation. Learns the mapping that best preserves the structure of the original space. ---Story time...--- The man and woman meet each other ...They become king and queen ...They got old and stop talking to each other. Instead, they read books and magazines ...<jupyter_code>corpus = """The man and woman meet each other ...
         They become king and queen ...
         They got old and stop talking to each other. Instead, they read books and magazines ...
         """
import matplotlib.pyplot as plt
import numpy as np

%matplotlib inline
# Let's hand assign the words to vectors
important_words = ['queen', 'book', 'king', 'magazine', 'woman', 'man']

vectors = np.array([[0.1,   0.3],  # queen
                    [-0.5, -0.1],  # book
                    [0.2,   0.2],  # king
                    [-0.3, -0.2],  # magazine
                    [-0.5,  0.4],  # car
                    [-0.45, 0.3]]) # bike
plt.plot(vectors[:,0], vectors[:,1], 'o')
plt.xlim(-0.6, 0.3)
plt.ylim(-0.3, 0.5)
for word, x, y in zip(important_words, vectors[:,0], vectors[:,1]):
    plt.annotate(word, (x, y), size=12)<jupyter_output><empty_output><jupyter_text>---Check for understanding---How many dimensions are data represented in? There are 2 dimensions.  How many dimensions would we need to represent for typical word vectors? 5Typically you would use n-1 word vectors, a baseline word would be coded as all zeros.<jupyter_code># Encode each word using 1-hot encoding
{'queen': [0, 0, 0, 0, 0],
 'book': [0, 0, 0, 0, 1],
 'king': [0, 0, 0, 1, 0],
 'magazine': [0, 0, 1, 0, 0],
 'woman': [0, 1, 0, 0, 0],
 'man': [1, 0, 0, 0, 0],
}<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1167231290421928, 'fraction_numerical': 0.02740991684631968, 'mean_word_length': 3.2962962962962963, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '271527', 'license': 'Apache-2.0', 'n_tokens_mistral': 995, 'n_tokens_neox': 920, 'n_words': 431, 'path': 'jupyterhub/notebooks/zz_under_construction/zz_old/TensorFlow/Word2Vec/1_word2vec_intro.ipynb', 'repo_name': 'just4jc/pipeline', 'size': '3252'}