<jupyter_start><jupyter_text>Intro and setup This notebook demonstrates the use of word2vec in Python using the  [gensim libraries](https://github.com/RaRe-Technologies/gensim).  Information is available on the [gensim website](https://radimrehurek.com/gensim/index.html) along with tutorials and the [API](https://radimrehurek.com/gensim/apiref.html). You can install them to your local machine using the command:```pip install --upgrade gensim```Of course if you are using a jupyter notebook, that doesn't mean it is installed in the correct place.  Make sure the `sys.path` command yields a location that matches the location of where the `pip install` occured.  Because the `sys.path` includes the current working directory, I use a `-t` flag to place the libraries in a location I can make the current working directory.  You could also theroretically append to `sys.path`  Background Word2Vec was first presented in the paper: <NAME>., <NAME>., <NAME>., & <NAME>. (2013). *Efficient estimation of word representations in vector space*. arXiv preprint arXiv:1301.3781.  Extensions were presented in the paper: <NAME>., <NAME>., <NAME>., <NAME>., & <NAME>. (2013). *Distributed representations of words and phrases and their compositionality*. In Advances in neural information processing systems (pp. 3111-3119).The original C code was published on Google Code, but was subsequently moved to https://github.com/tmikolov/word2vec<jupyter_code># turn off pretty printing to get horizontal display - optional, but I'm saving space for display
%pprint
import os<jupyter_output><empty_output><jupyter_text>Commands like `os.getcwd()` and `os.chdir()` can help you navigate to the proper location.<jupyter_code># gensim library also requires numpy, scipy, requests, docutils
os.listdir()<jupyter_output><empty_output><jupyter_text>Data  import dataThere are lots of possible data sources.  Gensim includes specific loaders for the [Brown corpus](https://radimrehurek.com/gensim/models/word2vec.htmlgensim.models.word2vec.BrownCorpus) and <NAME>'s [text8 corpus](https://radimrehurek.com/gensim/models/word2vec.htmlgensim.models.word2vec.Text8Corpus).  The Brown corpus is in the  widely used [Natural Language ToolKit](https://www.nltk.org/).  Text8 is a [cleaned text](http://mattmahoney.net/dc/textdata.html) 100MB selection of Wikipedia, getting rid of html tags, tables, spelling out numbers, etc.  It can be downloaded at http://mattmahoney.net/dc/text8.zip.<jupyter_code>with open('/home/milton/data/wikidump/text8.txt', 'r') as f:
    raw_data = f.read()
type(raw_data)
# yah, it is 100 MB
len(raw_data)
# what does it look like?
raw_data[:1000]
words = raw_data.split()
len(words)
# how many unique words?
len(set(words))
# the kinds of words you might expect from a Wikipedia corpus
list(set(words))[:100]<jupyter_output><empty_output><jupyter_text>Gensim<jupyter_code>import gensim
import logging
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)<jupyter_output><empty_output><jupyter_text>Setup data`gensim.models` takes a corpus broken into sentences.  I'm using the `Text8Corpus` iterator that comes as part of the `word2vec` class.  You can use any other data as long as you create an iterable to yield sentences.<jupyter_code>sentences = gensim.models.word2vec.Text8Corpus('/home/milton/data/wikidump/text8.txt')<jupyter_output><empty_output><jupyter_text>run model<jupyter_code># run model
model = gensim.models.Word2Vec(sentences, size = 100, window = 6, min_count=5, workers=3)<jupyter_output>2018-11-02 10:02:12,052 : INFO : collecting all words and their counts
2018-11-02 10:02:12,056 : INFO : PROGRESS: at sentence #0, processed 0 words, keeping 0 word types
2018-11-02 10:02:17,292 : INFO : collected 253854 word types from a corpus of 17005207 raw words and 1701 sentences
2018-11-02 10:02:17,292 : INFO : Loading a fresh vocabulary
2018-11-02 10:02:17,670 : INFO : effective_min_count=5 retains 71290 unique words (28% of original 253854, drops 182564)
2018-11-02 10:02:17,670 : INFO : effective_min_count=5 leaves 16718844 word corpus (98% of original 17005207, drops 286363)
2018-11-02 10:02:17,829 : INFO : deleting the raw counts dictionary of 253854 items
2018-11-02 10:02:17,837 : INFO : sample=0.001 downsamples 38 most-common words
2018-11-02 10:02:17,838 : INFO : downsampling leaves estimated 12506280 word corpus (74.8% of prior 16718844)
2018-11-02 10:02:18,046 : INFO : estimated required memory for 71290 words and 100 dimensions: 92677000 bytes
2018-11-02 10:02:18,046 : [...]<jupyter_text>Note that due to the stochastic nature of network training, the weights in the models will be different.  Each trained model will yield different results below in [Distance from mean](distance-from-mean) or [Similarity](similarity)<jupyter_code># save it as binary
model.save('savedModelText8')
print(model)<jupyter_output>Word2Vec(vocab=71290, size=100, alpha=0.025)<jupyter_text>Vocabulary<jupyter_code># get list of word vectors
words = list(model.wv.vocab)
# get sorted list of word vectors
words = list(model.wv.index2word)
len(words)
words[:200]
# check the index for a word
model.wv.vocab['one'].index<jupyter_output><empty_output><jupyter_text>Vectors<jupyter_code>model.wv?
model.wv.get_vector('one')
len(model.wv.get_vector('one'))<jupyter_output><empty_output><jupyter_text>Distance from mean<jupyter_code>model.wv.doesnt_match?
# find word in list that is farthest from the mean
model.wv.doesnt_match("breakfast cereal dinner lunch".split())
model.wv.doesnt_match("cook janitor pilot sport teacher".split())
model.wv.doesnt_match("joy time peace angst".split())
model.wv.doesnt_match("joy timely peace angst".split())
model.wv.doesnt_match("joy peace angst".split())<jupyter_output>/home/milton/programming/python/gensim/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.
  if np.issubdtype(vec.dtype, np.int):<jupyter_text>Similarity  Cosine similarity<jupyter_code>model.wv.similarity?
model.wv.similarity('woman', 'man')
model.wv.similarity('woman', 'tree')
model.wv.similarity('tree', 'shrub')
model.wv.similarity('tree', 'bush')
# distance is just the opposite of similarity
model.wv.distance('woman', 'tree')
model.wv.distance('woman', 'tree') + model.wv.similarity('woman', 'tree')
# closest by cosine similarity
model.wv.similar_by_word('woman', topn=10)
# closest by cosine similarity
model.wv.similar_by_word('she', topn=10)
model.wv.most_similar?
model.wv.most_similar(positive=['woman'], topn=10)
model.wv.most_similar(negative=['woman'], topn=10)
model.wv.most_similar(positive=['woman', 'king'], topn=10)
model.wv.most_similar(positive=['woman', 'king'], negative=['man'], topn=10)<jupyter_output>/home/milton/programming/python/gensim/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.
  if np.issubdtype(vec.dtype, np.int):<jupyter_text>Multiplicative combination<jupyter_code>model.wv.most_similar_cosmul?
model.wv.most_similar_cosmul(positive=['woman', 'king'], negative=['man'], topn=10)
model.wv.most_similar(positive=['woman', 'king'], negative=['man'], topn=10)
model.wv.most_similar_cosmul(positive=['woman', 'king'], topn=10)
model.wv.most_similar(positive=['woman', 'king'], topn=10)<jupyter_output>/home/milton/programming/python/gensim/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.
  if np.issubdtype(vec.dtype, np.int):
{'fraction_non_alphanumeric': 0.12326656394453005, 'fraction_numerical': 0.053158705701078585, 'mean_word_length': 6.1987060998151575, 'pattern_counts': {'":': 0, '<': 54, '<?xml version=': 0, '>': 54, 'https://': 7, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '544631', 'license': 'MIT', 'n_tokens_mistral': 2906, 'n_tokens_neox': 2575, 'n_words': 730, 'path': 'word2vecTalk.ipynb', 'repo_name': 'ultradian/word2vec', 'size': '7823'}