<jupyter_start><jupyter_text>Author: <NAME>This Jupyter notebook shows how to train and use word embeddings with the [Gensim](https://radimrehurek.com/gensim/) library. Some snippets have been adapted from the [Gensim's Word2Vec tutorial](https://rare-technologies.com/word2vec-tutorial/).[Word embeddings](https://ruder.io/word-embeddings-1/) are vector representations of words which are generally low-dimensional (often less than 1000 dimensions).  TRAINING WORD EMBEDDINGS (Word2Vec)---As usual, we first import the libraries that we are going to use, including Gensim.**Note:** All these libraries need to be downloaded beforehand if not using Google Colab. Check their official websites for details on how to install them.<jupyter_code>import numpy as np
import nltk
from nltk.tokenize import word_tokenize
import gensim
import requests
import random
nltk.download('punkt')<jupyter_output>[nltk_data] Downloading package punkt to /root/nltk_data...
[nltk_data]   Unzipping tokenizers/punkt.zip.<jupyter_text>To learn word embeddings, in this case Word2Vec ([Mikolov et al. 2013](https://arxiv.org/pdf/1301.3781.pdf)), we first need a sufficiently large text corpus. To this end we are going to use the [IMDb movie review corpus](https://ai.stanford.edu/~amaas/data/sentiment/), which includes a relatively large number of sentences (>25,000).**Note:** While >25,000 sentence is a decent corpus to train word embeddings, most general-domain word embeddings such as the [pre-trained model of Word2Vec](https://code.google.com/archive/p/word2vec/) are trained on corpora with billions of words.<jupyter_code>url_train="http://josecamachocollados.com/imdb_train.txt" # Containing all sentences of the imdb training set

#Load training set
response_train = requests.get(url_train)
dataset_file_train = response_train.text.split("\n")
random.shuffle(dataset_file_train) # We shuffle all sentences of our corpus<jupyter_output><empty_output><jupyter_text>Let's check what the IMDb dataset looks like<jupyter_code>for line in dataset_file_train[:5]:
  print (line)<jupyter_output>i love all types of films , especially horror . that being said , survival does not live up to any of the hype surrounding it . <br /> <br /> i ca n't give it any points on originality . there is nothing wrong with exploring the same themes , or remaking what others have done . it has just become a cop out for indie films to take us on a slasher journey through the woods , a crazed killer , and as of lately , throw in some crazy family . on those lines i have to compare it with the likes of texas chainsaw , wrong turn ( though the twist in that one is obvious ) , and others . survival falls up way short against comparable films . the plot was just not original in any way . some films can get by with a weak ( and way over-done ) plot with superb acting , special fx , or a slew of other factors . survivial does n't have any of that to bank on . if you will , note the following : the acting in the movie never took off . i do n't knock or blame the actors for that , nor the director . the [...]<jupyter_text>Once the dataset is loaded, we are going to tokenize and store our corpus into sentences.<jupyter_code>gensim_imdb_corpus=[]
for line in dataset_file_train:
  gensim_imdb_corpus.append(word_tokenize(line))
# This could take a lot of memory. If it's the case you can reduce the number of lines
print(gensim_imdb_corpus[0])<jupyter_output>['i', 'love', 'all', 'types', 'of', 'films', ',', 'especially', 'horror', '.', 'that', 'being', 'said', ',', 'survival', 'does', 'not', 'live', 'up', 'to', 'any', 'of', 'the', 'hype', 'surrounding', 'it', '.', '<', 'br', '/', '>', '<', 'br', '/', '>', 'i', 'ca', "n't", 'give', 'it', 'any', 'points', 'on', 'originality', '.', 'there', 'is', 'nothing', 'wrong', 'with', 'exploring', 'the', 'same', 'themes', ',', 'or', 'remaking', 'what', 'others', 'have', 'done', '.', 'it', 'has', 'just', 'become', 'a', 'cop', 'out', 'for', 'indie', 'films', 'to', 'take', 'us', 'on', 'a', 'slasher', 'journey', 'through', 'the', 'woods', ',', 'a', 'crazed', 'killer', ',', 'and', 'as', 'of', 'lately', ',', 'throw', 'in', 'some', 'crazy', 'family', '.', 'on', 'those', 'lines', 'i', 'have', 'to', 'compare', 'it', 'with', 'the', 'likes', 'of', 'texas', 'chainsaw', ',', 'wrong', 'turn', '(', 'though', 'the', 'twist', 'in', 'that', 'one', 'is', 'obvious', ')', ',', 'and', 'others', '.', 'survival', 'falls', 'up'[...]<jupyter_text>**Note:** The corpus may be further preprocessed if necessary (e.g. lowercased) or further cleaned. In this case the version of the IMDb corpus that we use was already lowercased. Finally, we can train our Word2vec word embedding model! For more information on training Word2Vec with gensim, you can check [here](https://radimrehurek.com/gensim/models/word2vec.html). This can take a few minutes (or hours depending on the size of the corpus).<jupyter_code>from gensim.models import Word2Vec
model = Word2Vec(gensim_imdb_corpus, size=100, window=5, min_count=3) 
# Size is the number of dimensions of the embeddings we are going to learn
# Window is the size considered for context of a target word
# Min count is the minimum number of times that a word need to occur to be learnt<jupyter_output><empty_output><jupyter_text>**Note:** You can save and load models using `model.save` and `model.load` functions. This enables you to export your models and use them anytime, or to use models training by someone else (i.e. pre-trained models).  **Exercise (Optional):** Train the same model using [FastText](https://radimrehurek.com/gensim/models/fasttext.html) instead of Word2vec. FastText is a model similar to Word2vec but takes also into account character information, which can be useful for noisy text such as the one we find in social media.  PLAYING WITH WORD2VEC---Now that our model has been trained, we can check the vectors for each word, which should have 100 dimensions.<jupyter_code>vector_movie=model['movie']
print ("Number of dimensions: "+str(len(vector_movie)))
print (vector_movie)
vector_love=model['love']
print ("Number of dimensions: "+str(len(vector_love)))
print (vector_love)<jupyter_output>Number of dimensions: 100
[ 1.3227099   1.6779795   0.8052093  -1.2148128  -0.04069256  2.2782824
  2.3685286  -0.557829    0.22121817 -0.10329057 -1.4483973   0.31201068
  0.2580814   0.34781384 -0.03476735 -0.8020109  -0.41720378  0.36654934
 -2.2042248  -0.81793946 -0.8517394  -1.499687   -2.8129475  -0.85805815
  3.1895504  -0.91617733  0.5238506  -1.1572242  -1.4115835   1.3821214
 -2.3398905   0.88816375  2.8979106  -1.8825063  -1.1497135   2.536966
  1.9810194  -1.4453561  -0.8771605  -0.17044427  0.58202636 -1.5836657
  3.0723312  -1.5278088  -1.1722782  -1.2712455   0.16938174  0.37277755
 -2.2325587  -0.59212303 -1.7500398  -2.7819529   1.3532248  -0.68875307
  0.3423171   0.43497545 -0.01049814  1.7840114   2.7322843   1.9269272
 -0.8985554  -0.69286233  1.9395846  -3.214609    2.1300218   0.10937414
 -0.84017426 -0.59636647  0.01071645 -2.6799884  -1.4849445  -2.5780046
 -1.3305414   2.2073247   1.7584609   1.3804194   0.39238128  1.2054267
  2.0669765  -0.1398903   1.88447[...]<jupyter_text>We can also check the similarity (measured by cosine similarity) between some words. Let's start with finding the most similar words to *film* or *casablanca* in our vector space. We can find the most similar words of any input word by using the `.most_similar` command.<jupyter_code>model.most_similar('movie')
model.most_similar('casablanca')<jupyter_output>/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:1: DeprecationWarning: Call to deprecated `most_similar` (Method will be removed in 4.0.0, use self.wv.most_similar() instead).
  """Entry point for launching an IPython kernel.<jupyter_text>We can also check the similarity between two given words.<jupyter_code>print(model.similarity('movie', 'film'))
print(model.similarity('movie', 'popcorn'))
print(model.similarity('movie', 'table'))<jupyter_output>0.9424787
0.10715821
-0.007065352<jupyter_text>Here we can see how words like *movie* and *film* are very close (in fact they are synonyms). Then other words like *movie* and *popcorn* are somehow related, while *movie* and *table* do not seem to be similar at all in this corpus. **Note:** In this notebook we have learned our own word embeddings in IMDb. However, please note that in many cases we are going to directly use an available pre-trained word embedding model. These are generally trained on large corpora and are therefore more complete/accurate. For example, there are pre-trained models for [Word2Vec](https://code.google.com/archive/p/word2vec/), [GloVe](https://nlp.stanford.edu/projects/glove/) or even [FastText trained on Twitter](https://github.com/pedrada88/crossembeddings-twitter). **Exercise (optional):** Choose a pre-trained model from Word2vec, GloVe or FastText (there are many available online) and load it using gensim. Check a few similarities and compare it with the word embeddings trained on IMDb.<jupyter_code>from gensim.models import Word2Vec
import gensim.downloader
print(list(gensim.downloader.info()['models'].keys()))

#pretrained_model = Word2Vec.load("word2vec.model")
glove_vectors = gensim.downloader.load('glove-twitter-50')
glove_vectors.most_similar('twitter')
print(glove_vectors.similarity('movie', 'film'))
print(glove_vectors.similarity('movie', 'popcorn'))
print(glove_vectors.similarity('movie', 'table'))<jupyter_output>0.74811
0.5045911
0.49300164<jupyter_text>**Exercise 1:** Train a Word2vec word embedding model on the IMDb corpus with 75 dimensions and a window size of 8. Then, check the most similar words of *movie* in the vector space and the similarity between *movie* and *table*. Compare the results with the previous trained model.<jupyter_code>model_ex = Word2Vec(gensim_imdb_corpus, size=75, window=8, min_count=3) 
print(model_ex.similarity('movie', 'table'))<jupyter_output>-0.08676528<jupyter_text>**Exercise (optional):** Take a corpus of your choice (any domain) and train a word embedding model using gensim. Check a few similarities of words and compare with the models trained on IMDB.<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12463314419878693, 'fraction_numerical': 0.07894736842105263, 'mean_word_length': 4.957459207459207, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 44, 'https://': 11, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '179822', 'license': 'Apache-2.0', 'n_tokens_mistral': 3830, 'n_tokens_neox': 3311, 'n_words': 1176, 'path': 'Word_Embeddings_Gensim.ipynb', 'repo_name': 'lzanellac/ALBERT', 'size': '10237'}