<jupyter_start><jupyter_text>[View in Colaboratory](https://colab.research.google.com/github/schwaaweb/aimlds1_11-NLP/blob/master/W11_CCS_AG_NLP_Coding_Challenge__3_Live_Coding_Solution.ipynb)  Coding Challenge 3: Natural Language Processing In this Coding Challenge, you will cover **Word2vec **which is a popular algorithm for building vector representations of words (i.e. word embeddings). The concept behind Word2Vec is quite straightforward - an assumption is made that the meaning of a word can be inferred by the *context it appears in* or *the company it keeps*. This is similar to stating: “tell me about your friends, and I will tell who you are”. If **2 **words  have very similar neighbors (meaning: the context in which it is used is similar), then the words are most likely quite similar.In this Coding Challenge, you will go through the process of training a Word2vec model with a sample set of documents and then examine certain attributes of the model. After that, you will train a Word2vec model with a large corpus of text and then ascertain the similarity among words in the corpus.<jupyter_code># https://radimrehurek.com/gensim/install.html
#!pip install --upgrade gensim
%%time
import nltk
nltk.download('all')<jupyter_output>[nltk_data] Downloading collection 'all'
[nltk_data]    | 
[nltk_data]    | Downloading package abc to
[nltk_data]    |     /Users/darwinm/nltk_data...
[nltk_data]    |   Package abc is already up-to-date!
[nltk_data]    | Downloading package alpino to
[nltk_data]    |     /Users/darwinm/nltk_data...
[nltk_data]    |   Package alpino is already up-to-date!
[nltk_data]    | Downloading package biocreative_ppi to
[nltk_data]    |     /Users/darwinm/nltk_data...
[nltk_data]    |   Package biocreative_ppi is already up-to-date!
[nltk_data]    | Downloading package brown to
[nltk_data]    |     /Users/darwinm/nltk_data...
[nltk_data]    |   Package brown is already up-to-date!
[nltk_data]    | Downloading package brown_tei to
[nltk_data]    |     /Users/darwinm/nltk_data...
[nltk_data]    |   Package brown_tei is already up-to-date!
[nltk_data]    | Downloading package cess_cat to
[nltk_data]    |     /Users/darwinm/nltk_data...
[nltk_data]    |   Package cess_cat is already up-to-date!
[nl[...]<jupyter_text>**Step 1: ** Tokenize the sample set of documents<jupyter_code>%%time
# Step 1

import gensim

raw_content = ['The dog ran up the steps and entered the owner\'s room to check if the owner was in the room.',
             'My name is <NAME>, commander of the Machine Learning program at Lambda school.',
             'I am creating the curriculum for the Machine Learning program and will be teaching the full-time Machine Learning program.',
            'Machine Learning is one of my favorite subjects.',
            'I am excited about taking the Machine Learning class at the Lambda school starting in April.',
                'When does the Machine Learning program kick-off at Lambda school?',
                'The batter hit the ball out off AT&T park into the pacific ocean.',
                'The pitcher threw the ball into the dug-out.']

from nltk.tokenize import word_tokenize
sentences = [word_tokenize(text) for text in raw_content]
print(sentences)<jupyter_output>[['The', 'dog', 'ran', 'up', 'the', 'steps', 'and', 'entered', 'the', 'owner', "'s", 'room', 'to', 'check', 'if', 'the', 'owner', 'was', 'in', 'the', 'room', '.'], ['My', 'name', 'is', 'Thomson', 'Comer', ',', 'commander', 'of', 'the', 'Machine', 'Learning', 'program', 'at', 'Lambda', 'school', '.'], ['I', 'am', 'creating', 'the', 'curriculum', 'for', 'the', 'Machine', 'Learning', 'program', 'and', 'will', 'be', 'teaching', 'the', 'full-time', 'Machine', 'Learning', 'program', '.'], ['Machine', 'Learning', 'is', 'one', 'of', 'my', 'favorite', 'subjects', '.'], ['I', 'am', 'excited', 'about', 'taking', 'the', 'Machine', 'Learning', 'class', 'at', 'the', 'Lambda', 'school', 'starting', 'in', 'April', '.'], ['When', 'does', 'the', 'Machine', 'Learning', 'program', 'kick-off', 'at', 'Lambda', 'school', '?'], ['The', 'batter', 'hit', 'the', 'ball', 'out', 'off', 'AT', '&', 'T', 'park', 'into', 'the', 'pacific', 'ocean', '.'], ['The', 'pitcher', 'threw', 'the', 'ball', 'into', 'the', 'dug-ou[...]<jupyter_text>**Step 2: ** Train the Word2vec model with tokenized content; size of the word vectors is 5; the word should show-up at least once in the raw content<jupyter_code>%%time
# Step 2
from gensim.models.word2vec import Word2Vec

model = Word2Vec(sentences, min_count=1, size=5)
dir(model)<jupyter_output>CPU times: user 12.6 ms, sys: 3.43 ms, total: 16 ms
Wall time: 55.4 ms<jupyter_text>**Step 3: **Output the number of words as well as the list of words in the model's vocabulary<jupyter_code># Step 3
print(model)
print(list(model.wv.vocab))
print(len(model.wv.vocab))<jupyter_output>Word2Vec(vocab=69, size=5, alpha=0.025)
['The', 'dog', 'ran', 'up', 'the', 'steps', 'and', 'entered', 'owner', "'s", 'room', 'to', 'check', 'if', 'was', 'in', '.', 'My', 'name', 'is', 'Thomson', 'Comer', ',', 'commander', 'of', 'Machine', 'Learning', 'program', 'at', 'Lambda', 'school', 'I', 'am', 'creating', 'curriculum', 'for', 'will', 'be', 'teaching', 'full-time', 'one', 'my', 'favorite', 'subjects', 'excited', 'about', 'taking', 'class', 'starting', 'April', 'When', 'does', 'kick-off', '?', 'batter', 'hit', 'ball', 'out', 'off', 'AT', '&', 'T', 'park', 'into', 'pacific', 'ocean', 'pitcher', 'threw', 'dug-out']
69<jupyter_text>**Step 4: **Output the vector of words for the following tokens: **a)** curriculum, **b)** ocean, and **c) **pitcher<jupyter_code># Step 4
print(model.wv['curriculum', 'ocean', 'pitcher'])<jupyter_output>[[ 0.0429332  -0.0387117  -0.0377904   0.07507995  0.03391911]
 [ 0.06513041 -0.09689079 -0.09367523 -0.0006524  -0.0025307 ]
 [ 0.00753346 -0.02386155 -0.0763839  -0.06631807  0.00179267]]<jupyter_text>**Step 5:** Now we are going to train the model with more data - larger corpus i.e. the 20 newsgroups text dataset. Fetch the data from the training subset*Reference*: http://scikit-learn.org/stable/datasets/index.html<jupyter_code>%%time
# Step 5
from sklearn.datasets import fetch_20newsgroups
text_from_corpus = fetch_20newsgroups(subset='train')<jupyter_output>CPU times: user 354 ms, sys: 148 ms, total: 502 ms
Wall time: 1.17 s<jupyter_text>**Step 6:** Output the metadata for the data that is fetched (investigate the object and what you can do with it)<jupyter_code># Step 6
print(dir(text_from_corpus))
print(text_from_corpus.description)<jupyter_output>['DESCR', 'data', 'description', 'filenames', 'target', 'target_names']
the 20 newsgroups by date dataset<jupyter_text>**Step 7: ** Output the  of posts across the different categories<jupyter_code># Step 7
len(text_from_corpus.data)<jupyter_output><empty_output><jupyter_text>**Step 8**: Tokenize the body of text for each post<jupyter_code># Step 8
import string

def process_text(text):
  """Remove punctuation, lowercase, and tokenize text."""
  # TODO: check for special cases like "I'll"
  text = "".join([char.lower() for char in text
                  if char not in string.punctuation])
  return word_tokenize(text)

sentences = [process_text(document) for document in text_from_corpus.data]<jupyter_output><empty_output><jupyter_text>**Step 9**: Train the Word2vec model - words should show up at least 3 times in the corpus of textand the size of each word vector is 200 (i.e. dimension = 200)Reference" Scroll down to the section "A closer look at the parameter settings" to review the parameters that can be set<jupyter_code># Step 9
news_model = Word2Vec(sentences, min_count=3, size=200)<jupyter_output><empty_output><jupyter_text>**Step 10**:  List the number of words in the model's vocabulary<jupyter_code># Step 10
print(len(news_model.wv.vocab))<jupyter_output>43312<jupyter_text>**Step 11:** Examine word similarity to the word "Christ" (find other words most similar to it)<jupyter_code># Step 11
news_model.wv.most_similar('christ')<jupyter_output><empty_output><jupyter_text>**Step 12**: Examine document similarity with Doc2vec to any body of text of your choice*Reference*: https://radimrehurek.com/gensim/models/doc2vec.html<jupyter_code>%%time
# Step 12

# We need to train a doc2vec model with the 20 Newsgroup dataset
# One of arguments to the model is a "TaggedDocument" so we will first go ahead and create a Tagged document

# Import TaggedDocument
from gensim.models.doc2vec import TaggedDocument

# Tokenize each of the posts within the newsgroups
sentences = [process_text(document) for document in text_from_corpus.data]

# Create a list of tagged_documents
# Every item within the tagged_documents list is a tokenized version of the posts 
tagged_documents_list = []
for i, sent in enumerate(sentences):
    tagged_documents_list.append(TaggedDocument(sent, ["sent_{}".format(i)]))

# Examine the first item within tagged_document_lists
# print(tagged_documents_list[0])

# Train the model with the list of Tagged Documents
# size of the vector is 300
doc2vec_model = gensim.models.doc2vec.Doc2Vec(tagged_documents_list,
                                              vector_size=300)

# Get the vector representation for a new document
vec_representation = doc2vec_model.infer_vector('I love test driving luxury cars.'.split())
#print(vec_representation)

#Determine the documents (posts) similar to the new document/post
doc2vec_model.docvecs.most_similar([vec_representation])
# Examine the first document in the list above to gauge the similarity
print(tagged_documents_list[8027])<jupyter_output>TaggedDocument(['nntppostinghost', 'surtifiuiono', 'from', 'thomas', 'parsli', 'thomaspifiuiono', 'subject', 're', 'my', 'gun', 'is', 'like', 'my', 'american', 'express', 'card', 'inreplyto', 'vikingiastateedu', 'dan', 'sorensons', 'message', 'of', 'mon', '19', 'apr', '1993', '085242', 'gmt', 'organization', 'dept', 'of', 'informatics', 'university', 'of', 'oslo', 'norway', '1qjmnuinnlmdc<EMAIL>heldcom', 'cmm0902734911642thomaspsurtifiuiono', 'viking734945095ponderouscciastateedu', 'cmm0902735132009thomaspsurtifiuiono', 'viking735209562ponderouscciastateedu', 'lines', '51', 'originator', 'thomaspsurtifiuiono', 'i', 'dont', 'remember', 'the', 'figures', 'exactly', 'but', 'there', 'were', 'about', '3500', 'deaths', 'in', 'texas', 'in', '1991', 'that', 'was', 'caused', 'by', 'guns', 'this', 'is', 'more', 'than', 'those', 'beeing', 'killed', 'in', 'caraccidents', 'yes', 'there', 'could', 'be', 'that', 'low', 'sentences', 'or', 'high', 'poverty', 'could', 'influence', 'the', 'figures', 'but[...]
{'fraction_non_alphanumeric': 0.15358490566037736, 'fraction_numerical': 0.030754716981132076, 'mean_word_length': 5.203042715038034, 'pattern_counts': {'":': 0, '<': 46, '<?xml version=': 0, '>': 46, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '189857', 'license': 'Unlicense', 'n_tokens_mistral': 3863, 'n_tokens_neox': 3553, 'n_words': 1143, 'path': 'W11_CCS_AG_NLP_Coding_Challenge__3_Live_Coding_Solution.ipynb', 'repo_name': 'schwaaweb/aimlds1_11-NLP', 'size': '10607'}