<jupyter_start><jupyter_text>Before you turn this problem in, make sure everything runs as expected. First, **restart the kernel** (in the menubar, select Kernel$\rightarrow$Restart) and then **run all cells** (in the menubar, select Cell$\rightarrow$Run All).Make sure you fill in any place that says `YOUR CODE HERE` or "YOUR ANSWER HERE", as well as your name and collaborators below:<jupyter_code>NAME = "<NAME>"
StudentNumber = "13413351"<jupyter_output><empty_output><jupyter_text>---  COS 802 2019 Homework 2 - Part 3 [35 points, 42 with extra credit] Exploring Word Embeddings.**You will learn how to:**- Train your own word embedding.- Use pretrained word embeddings.- Use word embeddings in simple classifiers**What this homework does not cover!!!!**- Sequential Deep Learning Models with embeddings - A challenge you have with words is that the context is defined by the sequential context. [Resource 1](https://machinelearningmastery.com/sequence-classification-lstm-recurrent-neural-networks-python-keras/) [Resource 2](https://www.coursera.org/lecture/nlp-sequence-models/why-sequence-models-0h7gT)- Other Deep learning models with embeddings [Resource 3](https://realpython.com/python-keras-text-classification/convolutional-neural-networks-cnn)**Note:** you can experiment by adding aditional cells, but they must be removed from final solution. Only the cells originally in the notebook plus the ones you have filled with your solution are required.  1 Packages Scikit-Learn for text Analysis- [sklearn](https://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html): Scikit-Learn Working With Text Data- [matplotlib](http://matplotlib.org) is a library for plotting graphs in Python.- [gensim](https://pypi.org/project/gensim/) NLP library with word-embedding functions.<jupyter_code>%pylab inline
!pip install wordcloud
!pip install nltk
!pip install gensim
import gensim
import nltk

import pandas as pd
import numpy as np

from nltk.tokenize import sent_tokenize, word_tokenize
from sklearn.model_selection import train_test_split
nltk.download()<jupyter_output>showing info https://raw.githubusercontent.com/nltk/nltk_data/gh-pages/index.xml<jupyter_text>Fetch the AGNews dataset from the FastAI repositoryFast AI repo https://course.fast.ai/datasets> 496,835 categorized news articles from >2000 news sources from the 4 largest classes from AG’s corpus of news articles, using only the title and description fields. The number of training samples for each class is 30,000 and testing 1900.<jupyter_code>import os

if not os.path.isfile('ag_news_csv.tgz'):
    !wget https://s3.amazonaws.com/fast-ai-nlp/ag_news_csv.tgz

#U ncompress the archive
if not os.path.isfile('ag_news_csv/train.csv'):
    !tar -xzf ag_news_csv.tgz
    # Lets see what is in the 
    !ls ag_news_csv/
# View categories
!cat ag_news_csv/classes.txt
df_ag_news_train  = pd.read_csv('ag_news_csv/train.csv', header = None)
df_ag_news_train.columns = ['category','title','text']
df_ag_news_train.head()<jupyter_output><empty_output><jupyter_text>Task1: Building own Word2Vec [4 points] What is word2vec*Word2vec is a group of related models that are used to produce word embeddings. These models are shallow, two-layer neural networks that are trained to reconstruct linguistic contexts of words. Word2vec takes as its input a large corpus of text and produces a vector space, typically of several hundred dimensions, with each unique word in the corpus being assigned a corresponding vector in the space.* [URL](https://en.wikipedia.org/wiki/Word2vec)**Task:**You will use gensim to be able to train both skip-gram and cbow word2vec models using the AGNews data.<jupyter_code>documents = df_ag_news_train.text.values

# Go through all the documents and tokenize them by words
data_gensim = []
for doc in documents:
    
    # We are training on sentences
    for sentence in sent_tokenize(doc):
        tokens = []
        # We want only lower case
        for word in word_tokenize(sentence):
            tokens.append(word.lower())
        data_gensim .append(tokens)

# The result
print("Original doc: ", documents[0])
print("Tokenized doc: ", data_gensim[0])
print("Num Sentences: ", len(data_gensim))<jupyter_output>Original doc:  Reuters - Short-sellers, Wall Street's dwindling\band of ultra-cynics, are seeing green again.
Tokenized doc:  ['reuters', '-', 'short-sellers', ',', 'wall', 'street', "'s", 'dwindling\\band', 'of', 'ultra-cynics', ',', 'are', 'seeing', 'green', 'again', '.']
Num Sentences:  154351<jupyter_text>Hmm, that does not seem right. Can we remove all of the special characters? Let's use the RegexTokenizer that can use a regular expression that removes special characters<jupyter_code>from nltk.tokenize import RegexpTokenizer

tokenizer = RegexpTokenizer(r'\w+') # Can you figure out what the regular expression is doing here?<jupyter_output><empty_output><jupyter_text>Let's do this again.<jupyter_code>documents = df_ag_news_train.text.values

# Go through all the documents and tokenize them by words
data_gensim = []
for doc in documents:
    # We are training on sentences
    for sentence in sent_tokenize(doc):
        tokens = []
        # We want only lower case
        for word in tokenizer.tokenize(sentence): # We are using the new tokeniser here
            tokens.append(word.lower())
        data_gensim .append(tokens)

# The result
print("Original doc: ", documents[0])
print("Tokenized doc: ", data_gensim[0])
print("Num Sentences: ", len(data_gensim))
# Word2vec properties
embedding_size = 300
window_size = 5
min_count_words = 1 # minimum times we must see a word before we make a vector for it<jupyter_output><empty_output><jupyter_text>Q1.1 Take a look at **[gensim.models.Word2Vec](https://radimrehurek.com/gensim/models/word2vec.html)** and train a **CBOW** model. The model should be saved in *model_cbow*<jupyter_code># Train CBOW model
# YOUR CODE HERE
from gensim.models import Word2Vec
model_cbow = Word2Vec( data_gensim, min_count=min_count_words,size= embedding_size, window= window_size)

# Note: This will take some time. Be patient.<jupyter_output><empty_output><jupyter_text>Lets see a vector for the word *money*<jupyter_code>model_cbow.wv['money']<jupyter_output><empty_output><jupyter_text>One of the things we can do with word embeddings we could not do with bag-of-words models is that we can now do [cosine similarity](https://www.machinelearningplus.com/nlp/cosine-similarity/) of words. 1.0 is perfect match.<jupyter_code>print("CBOW similarity: ", model_cbow.wv.similarity('money', 'cash'))
print("CBOW Most Similar to Money")
model_cbow.wv.most_similar('money')<jupyter_output>CBOW Most Similar to Money<jupyter_text>Q1.2Now, Take a look at **[gensim.models.Word2Vec](https://radimrehurek.com/gensim/models/word2vec.html)** and train a **Skip-Gram** model. The model should be saved in *model_skipgram*<jupyter_code># Train CBOW model
# YOUR CODE HERE

model_skipgram = Word2Vec( data_gensim, min_count=min_count_words,size= embedding_size, window= window_size, sg=1) 
#sg ({0, 1}, optional) – Training algorithm: 1 for skip-gram; otherwise CBOW.
# Note: This will take some time. Be patient.
print("Skip-Gram similarity: ", model_skipgram.wv.similarity('money', 'cash'))
print("SkipGram Most Similar to Money")
model_skipgram.wv.most_similar('money')<jupyter_output>SkipGram Most Similar to Money<jupyter_text>Task 2: Pretrained Embeddings. [4 Points]A great feature of embeddings is someone else can pretrain them for us. In this case we can get the original [Google News word2vec vectors](https://code.google.com/archive/p/word2vec/).<jupyter_code># If you have not downloaded this, uncomment the next 2 lines
!wget -c https://s3.amazonaws.com/dl4j-distribution/GoogleNews-vectors-negative300.bin.gz
!gunzip GoogleNews-vectors-negative300.bin.gz
from gensim import models

## Uncomment below

# model_w2v_google = models.KeyedVectors.load_word2vec_format('GoogleNews-vectors-negative300.bin', binary=True)

## Note: This will take some time. Be patient.
#the cell above is not editable!
#I was unable to access the files, instead i did manual download
model_w2v_google = models.KeyedVectors.load_word2vec_format('GoogleNews-vectors-negative300.bin', binary=True)
## Uncomment below
#  print("Google News Pretrained W2V similarity: ", model_w2v_google.wv.similarity('money', 'cash'))
## Uncomment below

print("Google News Pretrained W2V similarity: ", model_w2v_google.wv.most_similar('money'))
## Uncomment below

print("Google Pretrained W2V Most Similar to Money")
model_w2v_google.wv.most_similar('money')<jupyter_output><empty_output><jupyter_text>Questions**Q2.1:** Why do the 3 different word2vec models have differing distances between money and cash?**Q2.2:** Test other words and check if the distance makes sense to you. What do you observe?  AnswersA2.1 and A2.2 HereYOUR ANSWER HERE  Task 3: Visualizing Word Vectors with TSNE [4 points]In this section you will first reduce the large vector space (dimenstion 300) to a 2D space that you can then visualise the words and their semantic relations on a plot. We use tSNE for this.*t-distributed Stochastic Neighbor Embedding. t-SNE is a tool to visualize high-dimensional data. It converts similarities between data points to joint probabilities and tries to minimize the Kullback-Leibler divergence between the joint probabilities of the low-dimensional embedding and the high-dimensional data. t-SNE has a cost function that is not convex, i.e. with different initializations we can get different results.* [URL](https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html)<jupyter_code>from sklearn.manifold import TSNE
tsne_model = TSNE(perplexity=40, n_components=2, init='pca', n_iter=1500)<jupyter_output><empty_output><jupyter_text>Lets get data from the CBOW model so that we can then train the tSNE model.<jupyter_code>labels_viz = []
tokens_viz = []
count = 0
for word in model_cbow.wv.vocab:
    if word in model_w2v_google.wv.vocab:
        tokens_viz.append(model_cbow[word])
        labels_viz.append(word)
        if count > 10000:
            break
        count+=1<jupyter_output>c:\users\mohlatlegon\appdata\local\programs\python\python36\lib\site-packages\ipykernel_launcher.py:5: DeprecationWarning: Call to deprecated `wv` (Attribute will be removed in 4.0.0, use self instead).
  """
c:\users\mohlatlegon\appdata\local\programs\python\python36\lib\site-packages\ipykernel_launcher.py:6: DeprecationWarning: Call to deprecated `__getitem__` (Method will be removed in 4.0.0, use self.wv.__getitem__() instead).<jupyter_text>Now lets train the TSNE Model for CBOW<jupyter_code>values_cbow_tsne = tsne_model.fit_transform(tokens_viz)
# Note: This will take some time. Be patient.
import matplotlib.pyplot as plt
x = []
y = []
for value in values_cbow_tsne:
    x.append(value[0])
    y.append(value[1])
    
plt.figure(figsize=(20, 20)) 
for i in range(200):
    plt.scatter(x[i],y[i])
    plt.annotate(labels_viz[i],
                    xy=(x[i], y[i]),
                    xytext=(5, 2),
                    textcoords='offset points',
                    ha='right',
                    va='bottom')
# Get the same words in the gogogle model
tokens_viz_w2v = []
for word in labels_viz:
        tokens_viz_w2v.append( model_w2v_google[word])
values_w2v_tsne = tsne_model.fit_transform(tokens_viz_w2v)
# Note: This will take some time. Be patient.
import matplotlib.pyplot as plt
x = []
y = []
for value in values_w2v_tsne:
    x.append(value[0])
    y.append(value[1])
    
plt.figure(figsize=(20, 20)) 
for i in range(200):
    plt.scatter(x[i],y[i])
    plt.annotate(labels_viz[i],
                    xy=(x[i], y[i]),
                    xytext=(5, 2),
                    textcoords='offset points',
                    ha='right',
                    va='bottom')<jupyter_output><empty_output><jupyter_text>Question 3:What do you observe in the two visualisations? Any similarities, what are the differences. What can you say about one embedding over the other?  A3YOUR ANSWER HERE  Task 4: Classifiers and embeddings [40 points]In this task, which takes up most of this homework, you will experiment with different ways to use word embeddings to transform documents. For this section you will transform documents [multiple word embeddings] into a single vector and looking at the effect on classifier performance.* **Q1:** experiment with using a *sum* to transform the word embedding sequence to a document vector [5]* **Q2:** experiment with using a *max* to transform the word embedding sequence to a document vector [5]* **Q3:** experiment with using a 1 and 2 and also playing with the different CBOW, and SkipGram models to transform the word embedding sequence to a document vector [7]* **Q4: Discusion:** Discuss your observations on the performance results [10]* **Q5 [Extra Credit - extra 5 points]:** experiment with using approaches from [this paper](https://arxiv.org/abs/1803.01400) to transform the word embedding sequence to a document vector - Paper: *Concatenated Power Mean Word Embeddings as Universal Cross-Lingual Sentence Representations***Evalution:** Evaluate the performance via accuracy on predicting the document class. Example is shown below for a mean transform with logistic regression model.**Note** We will not cover using paragraph2vec or doc2vec but you can look at that yourself. It is available in the gensim package.<jupyter_code># Helper function to transform a document into a mean of the word vectors. 

# Note!! You should add your code in this function to implement Q1-5
def embedding_transform(documents_tokenized, w2v_model, embedding_size, transform_type='mean'):
    transformed_docs = []
    for words in documents_tokenized:
        transformed_doc = []
        for w in words:
            if w in w2v_model:
                transformed_doc.append(w2v_model[w])
            else:
                transformed_doc.append(np.zeros(embedding_size)) ## Add a default 0 vector for unknown words
        
        # How do you transform to a document? Put your answer in here. 
        if  transform_type == 'mean':
            transformed_doc = np.mean(transformed_doc, axis=0)
        elif transform_type == 'sum':
            # YOUR CODE HERE
            transformed_doc = np.sum(transformed_doc, axis=0)
        elif transform_type == 'max':
            # YOUR CODE HERE
            transformed_doc = np.max(transformed_doc, axis=0)
        elif transform_type == 'extra':
            # YOUR CODE HERE
            transformed_doc = np.power(np.mean(np.power(transformed_doc,2),axis =0),1/2)
        else:
            raise NotImplementedError()
        transformed_docs.append(transformed_doc)
    output = np.array(transformed_docs)
    return output<jupyter_output><empty_output><jupyter_text>Go through all the documents and tokenize them by words<jupyter_code>documents_tokenized = []
for doc in documents:
    tokens = []
    # We want only lower case
    for word in tokenizer.tokenize(doc): # We are using the new tokeniser here
        tokens.append(word.lower())
    documents_tokenized.append(tokens)
y = df_ag_news_train.category # Get all the document labels

# Now lets split into training and testing.
doc_train, doc_test, y_train, y_test = train_test_split(documents_tokenized,y , test_size = 0.3)<jupyter_output><empty_output><jupyter_text>Here is where we transform the data with the mean transform.<jupyter_code>X_train_mean = embedding_transform(doc_train, model_cbow.wv, embedding_size, transform_type='mean')
X_train_mean.shape<jupyter_output><empty_output><jupyter_text>Note the size of each document is now the embedding_size. As you can see, every document is now only of size 300, instead of the larger sives with Bag-of-words<jupyter_code>from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
# Train and then check score.
clf = LogisticRegression(n_jobs=-1) # Use multicore
scores = cross_val_score(clf, X_train_mean, y_train, cv=5)
scores<jupyter_output><empty_output><jupyter_text>Answers to all Task 4 questions belowDo your experimentation below to answer the questions. You can add code lines, visualisations and tests as needed and also include the a **discussion**.   A 4.1<jupyter_code># YOUR CODE HERE
X_train_sum = embedding_transform(doc_train, model_cbow.wv, embedding_size, transform_type='sum')
X_train_sum.shape<jupyter_output><empty_output><jupyter_text>A 4.2<jupyter_code># YOUR CODE HERE
X_train_max = embedding_transform(doc_train, model_cbow.wv, embedding_size, transform_type='max')
X_train_max.shape<jupyter_output><empty_output><jupyter_text>A 4.3<jupyter_code># YOUR CODE HERE
X_train_sum_sg = embedding_transform(doc_train, model_skipgram.wv, embedding_size, transform_type='sum')
X_train_sum_sg.shape<jupyter_output><empty_output><jupyter_text>A 4.5 (optional)<jupyter_code># YOUR CODE HERE
X_train_extra = embedding_transform(doc_train, model_cbow.wv, embedding_size, transform_type='extra')
X_train_extra.shape<jupyter_output><empty_output><jupyter_text>Discussion 4<jupyter_code>### Answer
# YOUR CODE HERE
The Skip-Gram algorithm performs better than the continous bag of words (CBOW) algorithm for the model.
The Skip-Gram model had a better model accuracy (validation accuracy scores) across all the embedding transformations and the models ran quicker than the
CBOW models.
The different embedding transfomations are discussed below:
Sum:
Model Average Accuracy: CBOW: 87.6% and Skip-Gram: 88.9%
Max:
Model Average Accuracy: CBOW: 81.3% and Skip-Gram: 85.3%
Mean:
Model Average Accuracy: CBOW: 86.9% and Skip-Gram: 88.8%
Power-Mean Algorthim:
Model Average Accuracy: CBOW: 86.1% and Skip-Gram: 88.1%<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09304025185518328, 'fraction_numerical': 0.015291207555655498, 'mean_word_length': 4.756957928802589, 'pattern_counts': {'":': 0, '<': 89, '<?xml version=': 0, '>': 92, 'https://': 16, 'lorem ipsum': 0, 'www.': 2, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '330916', 'license': 'CC-BY-3.0', 'n_tokens_mistral': 5737, 'n_tokens_neox': 5392, 'n_words': 1791, 'path': 'u13413351-Mohlatlego-Nakeng-hw2-02-all-about-embeddings-w2v.ipynb', 'repo_name': 'Mohlatlegonakeng/Mohlatlego-Nakeng', 'size': '17799'}