<jupyter_start><jupyter_text>Table of Contents1&nbsp;&nbsp;Introducing Distributed Word Vectors2&nbsp;&nbsp;Training and Saving Your Model3&nbsp;&nbsp;Exploring the Model Results4&nbsp;&nbsp;Numeric Representations of Words5&nbsp;&nbsp;From Words To Paragraphs, Attempt 1: Vector Averaging6&nbsp;&nbsp;From Words to Paragraphs, Attempt 2: Clustering source: https://www.kaggle.com/c/word2vec-nlp-tutorialpart-2-word-vectors  Introducing Distributed Word VectorsNow down to the nitty-gritty! First, we read in the data with pandas, as we did in Part 1. Unlike Part 1, we now use unlabeledTrain.tsv, which contains 50,000 additional reviews with no labels. When we built the Bag of Words model in Part 1, extra unlabeled training reviews were not useful. However, since Word2Vec can learn from unlabeled data, these extra 50,000 reviews can now be used.<jupyter_code>import pandas as pd

# Read data from files 
train = pd.read_csv( "data/labeledTrainData.tsv", header=0, 
 delimiter="\t", quoting=3 )

test = pd.read_csv( "data/testData.tsv", header=0, delimiter="\t", quoting=3 )

unlabeled_train = pd.read_csv( "data/unlabeledTrainData.tsv", header=0, 
 delimiter="\t", quoting=3 )

# Verify the number of reviews that were read (100,000 in total)
print ("Read %d labeled train reviews, %d labeled test reviews, " \
 "and %d unlabeled reviews\n" % (train["review"].size,  
 test["review"].size, unlabeled_train["review"].size ))<jupyter_output>Read 25000 labeled train reviews, 25000 labeled test reviews, and 50000 unlabeled reviews<jupyter_text>The functions we write to clean the data are also similar to Part 1, although now there are a couple of differences. First, to train Word2Vec it is better not to remove stop words because the algorithm relies on the broader context of the sentence in order to produce high-quality word vectors. For this reason, we will make stop word removal optional in the functions below. It also might be better not to remove numbers, but we leave that as an exercise for the reader.<jupyter_code># Import various modules for string cleaning
from bs4 import BeautifulSoup
import re
from nltk.corpus import stopwords

def review_to_wordlist( review, remove_stopwords=False ):
    # Function to convert a document to a sequence of words,
    # optionally removing stop words.  Returns a list of words.
    #
    # 1. Remove HTML
    review_text = BeautifulSoup(review,"lxml").get_text()
    #  
    # 2. Remove non-letters
    review_text = re.sub("[^a-zA-Z]"," ", review_text)
    #
    # 3. Convert words to lower case and split them
    words = review_text.lower().split()
    #
    # 4. Optionally remove stop words (false by default)
    if remove_stopwords:
        stops = set(stopwords.words("english"))
        words = [w for w in words if not w in stops]
    #
    # 5. Return a list of words
    return(words)<jupyter_output><empty_output><jupyter_text>Next, we want a specific input format. Word2Vec expects single sentences, each one as a list of words. In other words, the input format is a list of lists.It is not at all straightforward how to split a paragraph into sentences. There are all kinds of gotchas in natural language. English sentences can end with "?", "!", """, or ".", among other things, and spacing and capitalization are not reliable guides either. For this reason, we'll use NLTK's punkt tokenizer for sentence splitting. In order to use this, you will need to install NLTK and use nltk.download() to download the relevant training file for punkt.<jupyter_code># Download the punkt tokenizer for sentence splitting
import nltk.data
#nltk.download()   

# Load the punkt tokenizer
tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')

# Define a function to split a review into parsed sentences
def review_to_sentences( review, tokenizer, remove_stopwords=False ):
    # Function to split a review into parsed sentences. Returns a 
    # list of sentences, where each sentence is a list of words
    #
    # 1. Use the NLTK tokenizer to split the paragraph into sentences
    raw_sentences = tokenizer.tokenize(review.strip())
    #
    # 2. Loop over each sentence
    sentences = []
    for raw_sentence in raw_sentences:
        # If a sentence is empty, skip it
        if len(raw_sentence) > 0:
            # Otherwise, call review_to_wordlist to get a list of words
            sentences.append( review_to_wordlist( raw_sentence, \
              remove_stopwords ))
    #
    # Return the list of sentences (each sentence is a list of words,
    # so this returns a list of lists
    return sentences
sentences = []  # Initialize an empty list of sentences

print ("Parsing sentences from training set")
for review in train["review"]:
    sentences += review_to_sentences(review, tokenizer)

print ("Parsing sentences from unlabeled set")
for review in unlabeled_train["review"]:
    sentences += review_to_sentences(review, tokenizer)

# A minor detail to note is the difference between the "+=" and "append" when it comes to Python lists. In many applications the two are interchangeable, but here they are not. If you are appending a list of lists to another list of lists, "append" will only append the first list; you need to use "+=" in order to join all of the lists at once.
# Check how many sentences we have in total - should be around 850,000+
print (len(sentences))
print (sentences[0])
print (sentences[1])<jupyter_output>['with', 'all', 'this', 'stuff', 'going', 'down', 'at', 'the', 'moment', 'with', 'mj', 'i', 've', 'started', 'listening', 'to', 'his', 'music', 'watching', 'the', 'odd', 'documentary', 'here', 'and', 'there', 'watched', 'the', 'wiz', 'and', 'watched', 'moonwalker', 'again']
['maybe', 'i', 'just', 'want', 'to', 'get', 'a', 'certain', 'insight', 'into', 'this', 'guy', 'who', 'i', 'thought', 'was', 'really', 'cool', 'in', 'the', 'eighties', 'just', 'to', 'maybe', 'make', 'up', 'my', 'mind', 'whether', 'he', 'is', 'guilty', 'or', 'innocent']<jupyter_text>Training and Saving Your Model- **Architecture**: Architecture options are skip-gram (default) or continuous bag of words. We found that skip-gram was very slightly slower but produced better results.- Training algorithm: Hierarchical softmax (default) or negative sampling. For us, the default worked well.- Downsampling of frequent words: The Google documentation recommends values between .00001 and .001. For us, values closer 0.001 seemed to improve the accuracy of the final model.- Word vector dimensionality: More features result in longer runtimes, and often, but not always, result in better models. Reasonable values can be in the tens to hundreds; we used 300.- Context / window size: How many words of context should the training algorithm take into account? 10 seems to work well for hierarchical softmax (more is better, up to a point).- Worker threads: Number of parallel processes to run. This is computer-specific, but between 4 and 6 should work on most systems.- Minimum word count: This helps limit the size of the vocabulary to meaningful words. Any word that does not occur at least this many times across all documents is ignored. Reasonable values could be between 10 and 100. In this case, since each movie occurs 30 times, we set the minimum word count to 40, to avoid attaching too much importance to individual movie titles. This resulted in an overall vocabulary size of around 15,000 words. Higher values also help limit run time.<jupyter_code># Import the built-in logging module and configure it so that Word2Vec 
# creates nice output messages
import logging
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s',\
    level=logging.INFO)

# Set values for various parameters
num_features = 300    # Word vector dimensionality                      
min_word_count = 40   # Minimum word count                        
num_workers = 4       # Number of threads to run in parallel
context = 10          # Context window size                                                                                    
downsampling = 1e-3   # Downsample setting for frequent words

# Initialize and train the model (this will take some time)
from gensim.models import word2vec
print ("Training model...")
model = word2vec.Word2Vec(sentences, workers=num_workers, \
            size=num_features, min_count = min_word_count, \
            window = context, sample = downsampling)

# If you don't plan to train the model any further, calling 
# init_sims will make the model much more memory-efficient.
model.init_sims(replace=True)

# It can be helpful to create a meaningful model name and 
# save the model for later use. You can load it later using Word2Vec.load()
model_name = "300features_40minwords_10context"
model.save(model_name)<jupyter_output>2017-03-31 11:56:07,911 : INFO : 'pattern' package not found; tag filters are not available for English
2017-03-31 11:56:07,938 : INFO : collecting all words and their counts
2017-03-31 11:56:07,939 : INFO : PROGRESS: at sentence #0, processed 0 words, keeping 0 word types
2017-03-31 11:56:07,989 : INFO : PROGRESS: at sentence #10000, processed 225803 words, keeping 17776 word types
2017-03-31 11:56:08,046 : INFO : PROGRESS: at sentence #20000, processed 451892 words, keeping 24948 word types
2017-03-31 11:56:08,097 : INFO : PROGRESS: at sentence #30000, processed 671315 words, keeping 30034 word types<jupyter_text>If you are on a Mac or Linux system, you can use the "top" command from within Terminal (not from within Python) to see if your system is successfully parallelizing while the model is training. Type<jupyter_code>#!top -o cpu<jupyter_output><empty_output><jupyter_text>Exploring the Model ResultsCongratulations on making it successfully through everything so far! Let's take a look at the model we created out of our 75,000 training reviews.The "doesnt_match" function will try to deduce which word in a set is most dissimilar from the others:<jupyter_code>model.doesnt_match("man woman child kitchen".split())<jupyter_output><empty_output><jupyter_text>Our model is capable of distinguishing differences in meaning! It knows that men, women and children are more similar to each other than they are to kitchens. More exploration shows that the model is sensitive to more subtle differences in meaning, such as differences between countries and cities:<jupyter_code>model.doesnt_match("france england germany berlin".split())

model.doesnt_match("paris berlin london austria".split())<jupyter_output><empty_output><jupyter_text>We can also use the "most_similar" function to get insight into the model's word clusters:<jupyter_code>model.most_similar("man")
model.most_similar("queen")
model.most_similar("awful")
model.most_similar("book")
model.syn1neg.shape<jupyter_output><empty_output><jupyter_text>Part 3  Numeric Representations of WordsNow that we have a trained model with some semantic understanding of words, how should we use it? If you look beneath the hood, the Word2Vec model trained in Part 2 consists of a feature vector for each word in the vocabulary, stored in a numpy array called "syn0":The number of rows in syn0 is the number of words in the model's vocabulary, and the number of columns corresponds to the size of the feature vector, which we set in Part 2.  Setting the minimum word count to 40 gave us a total vocabulary of 16,492 words with 300 features apiece. Individual word vectors can be accessed in the following way:<jupyter_code># Load the model that we created in Part 2
from gensim.models import Word2Vec
model = Word2Vec.load("300features_40minwords_10context")


type(model.syn1neg)


model.syn1neg.shape
model["flower"]<jupyter_output><empty_output><jupyter_text>From Words To Paragraphs, Attempt 1: Vector AveragingOne challenge with the IMDB dataset is the variable-length reviews. We need to find a way to take individual word vectors and transform them into a feature set that is the same length for every review.Since each word is a vector in 300-dimensional space, we can use vector operations to combine the words in each review. One method we tried was to simply average the word vectors in a given review (for this purpose, we removed stop words, which would just add noise).The following code averages the feature vectors, building on our code from Part 2.<jupyter_code>import numpy as np  # Make sure that numpy is imported

def makeFeatureVec(words, model, num_features):
    
    #print("in makefeature vec")
    # Function to average all of the word vectors in a given
    # paragraph
    #
    # Pre-initialize an empty numpy array (for speed)
    featureVec = np.zeros((num_features,),dtype="float32")
    #
    nwords = 0
    # 
    # Index2word is a list that contains the names of the words in 
    # the model's vocabulary. Convert it to a set, for speed 
    index2word_set = set(model.wv.index2word)
    #
    # Loop over each word in the review and, if it is in the model's
    # vocaublary, add its feature vector to the total
    for word in words:
        if word in index2word_set:
#            print(word)
            nwords = nwords + 1
            featureVec = np.add(featureVec,model[word])
    # 
    # Divide the result by the number of words to get the average
    featureVec = np.divide(featureVec,nwords)
    #print("featureVec",featureVec)
    return featureVec


def getAvgFeatureVecs(reviews, model, num_features):
    # Given a set of reviews (each one a list of words), calculate 
    # the average feature vector for each one and return a 2D numpy array 
    # 
    # Initialize a counter
    counter = 0
    # 
    # Preallocate a 2D numpy array, for speed
    reviewFeatureVecs = np.zeros((len(reviews),num_features),dtype="float32")
    # (13526, 300)
    
    # Loop through the reviews
    for review in reviews:
        ##### PROBLEM HERE. SEEMS TO ONLY GO THROUGH FIRST REVIEW 
        
#        print("in review")
        
        #
        # Print a status message every 1000th review
        if counter%1000 == 0:
            print ("Review %d of %d" % (counter, len(reviews)))
        # 
        # Call the function (defined above) that makes average feature vectors
        reviewFeatureVecs[counter] = makeFeatureVec(review, model, \
           num_features)
        #
        # Increment the counter
        counter = counter + 1
        #print("counter" , counter)
        
    return reviewFeatureVecs
# train["review"]
review_to_wordlist<jupyter_output><empty_output><jupyter_text>Now, we can call these functions to create average vectors for each paragraph. The following operations will take a few minutes:<jupyter_code># ****************************************************************
# Calculate average feature vectors for training and testing sets,
# using the functions we defined above. Notice that we now use stop word
# removal.

clean_train_reviews = []
for review in train["review"]:
    clean_train_reviews.append( review_to_wordlist( review, \
        remove_stopwords=True ))

trainDataVecs = getAvgFeatureVecs( clean_train_reviews, model, num_features )

print ("Creating average feature vecs for test reviews")
clean_test_reviews = []
for review in test["review"]:
    clean_test_reviews.append( review_to_wordlist( review, \
        remove_stopwords=True ))

testDataVecs = getAvgFeatureVecs( clean_test_reviews, model, num_features )
testDataVecs.shape<jupyter_output><empty_output><jupyter_text>Next, use the average paragraph vectors to train a random forest. Note that, as in Part 1, we can only use the labeled training reviews to train the model. We found that this produced results much better than chance, but underperformed Bag of Words by a few percentage points.Since the element-wise average of the vectors didn't produce spectacular results, perhaps we could do it in a more intelligent way? A standard way of weighting word vectors is to apply "tf-idf" weights, which measure how important a given word is within a given set of documents. One way to extract tf-idf weights in Python is by using scikit-learn's TfidfVectorizer, which has an interface similar to the CountVectorizer that we used in Part 1. However, when we tried weighting our word vectors in this way, we found no substantial improvement in performance.<jupyter_code># Fit a random forest to the training data, using 100 trees
from sklearn.ensemble import RandomForestClassifier
forest = RandomForestClassifier( n_estimators = 100 )

print("Fitting a random forest to labeled training data...")
forest = forest.fit( trainDataVecs, train["sentiment"] )

# Test & extract results 
result = forest.predict( testDataVecs )

# Write the test results 
output = pd.DataFrame( data={"id":test["id"], "sentiment":result} )
output.to_csv( "Word2Vec_AverageVectors.csv", index=False, quoting=3 )

# Your submission scored 0.82836.
testDataVecs.shape
trainDataVecs.shape<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08231111111111111, 'fraction_numerical': 0.02162962962962963, 'mean_word_length': 4.167176974892835, 'pattern_counts': {'":': 3, '<': 46, '<?xml version=': 0, '>': 47, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '649717', 'license': 'MIT', 'n_tokens_mistral': 4977, 'n_tokens_neox': 4585, 'n_words': 2070, 'path': '*Deep_Learning/Word_Embedding/Word2Vec/Word2Vec Kaggle Part2.ipynb', 'repo_name': 'jasjung/Machine_Learnin', 'size': '16875'}