<jupyter_start><jupyter_text>Please check out the original [tensorflow implementation](https://github.com/uclmr/emoji2vec) of Emoji2Vec by its authors as well as the [paper](https://arxiv.org/pdf/1609.08359.pdf) for more details. This notebook is intended to provide an intuition behind Emoji2Vec and provide an idea of its features rather than serve as a in-depth analysis of the framework.<jupyter_code># External dependencies
import os
import numpy as np
import pickle as pk
import gensim.models as gsm
import torch
from tabulate import tabulate

# Internal dependencies
from model import Emoji2Vec, ModelParams
from phrase2vec import Phrase2Vec
from utils import build_kb, get_examples_from_kb, generate_embeddings, get_metrics, generate_predictions<jupyter_output><empty_output><jupyter_text>What is Emoji2Vec?Emoji2Vec representations are a form of distributed representations for Unicode emojis, trained directly from natural language descriptions of emojis. This notebook presents a high level walkthrough the process of training Emoji2Vec representations and demonstrates some experimental results. First, lets define the logical steps in training Emoji2Vec representations.1. For each emoji in a data set a number of natural language descriptions is collected.2. Each description is encoded to a fixed form vector in a high dimensional space.     - Although it can be done by an arbitrary encoding method, this implementation follows the approach presented in the [paper](https://arxiv.org/pdf/1609.08359.pdf) using 300-dimensional [Google News word2vec embeddings](https://code.google.com/archive/p/word2vec/) together with a simple phrase encoder `phrase2vec.Phrase2Vec`.3. A neural network model is trained to classify emojis from their descriptions.    - Inside the model each unique emoji has its own vector of parameters that is updated when this emoji is being classified. Through continuous incrementation in training such vectors of parameters become emoji representations.4. Neural network's parameters are extracted and used as distributed emoji representations that are embedded in the same space as the underlying word embeddings.  How to train Emoji2Vec representations?First, we need a couple of things to train Emoji2Vec representations. Most importantly we need something to encode natural language descriptions of emojis to a fixed form high dimensional vectors. For this we are going to use [Google News wor2vec embeddings](https://code.google.com/archive/p/word2vec/), so in order to continue with this notebook make sure you've downloaded those embeddings and placed a `.bin.gz` file in `data/word2vec`. Of course we also need some emojis and their natural language descriptions. Fortunately authors of the paper collected a data set that is part of this repository. Our training data set is in `data/training`. Let's add this directory to our parameters.<jupyter_code>params = {"data_folder": "data/training"}<jupyter_output><empty_output><jupyter_text>To train Emoji2Vec representations we need to put emojis in some sort of organised structure, so that after training we know which vector of parameters in the model is associated with which emoji. Of course the same needs to be done for each phrase (natural language description of emoji) in the data set so that we know which descrption describes which emoji.<jupyter_code># Build knowledge base
print('reading training data from: ' + params["data_folder"])
kb, ind2phr, ind2emoji = build_kb(params["data_folder"])<jupyter_output>reading training data from: data/training<jupyter_text>We'll need this emoji mapping later, so for now let's save it to a file.<jupyter_code>params.update({"mapping_file": "emoji_mapping_file.pkl"})
# Save the mapping from index to emoji
pk.dump(ind2emoji, open(params["mapping_file"], 'wb'))<jupyter_output><empty_output><jupyter_text>Time to encode descriptions of emojis to fixed form vectors. As mentioned before we're using 300 dimensional Google News embeddings located in `data/word2vec`. As we will need them later we're going to save the generated embeddings of descriptions to a file `phrase_embeddings.pkl`. Let's add those paths to our parameters.<jupyter_code>params.update({"word2vec_embeddings_file": "data/word2vec/GoogleNews-vectors-negative300.bin.gz",
               "phrase_embeddings_file": "phrase_embeddings.pkl"})<jupyter_output><empty_output><jupyter_text>To reitarete - phrase embeddings are embeddigns of each emoji description that will serve as input for the neural network, whereas Google News embeddings are what we use to generate phrase embeddings.<jupyter_code># Generate embeddings for each phrase in the training set
embeddings_array = generate_embeddings(ind2phr=ind2phr, kb=kb, embeddings_file=params["phrase_embeddings_file"],
                                       word2vec_file=params["word2vec_embeddings_file"])<jupyter_output>loading embeddings...<jupyter_text>So we have the training input (phrase embeddings), now let's load our training and development data sets.<jupyter_code>train_set = get_examples_from_kb(kb=kb, example_type='train')
dev_set = get_examples_from_kb(kb=kb, example_type='dev')<jupyter_output><empty_output><jupyter_text>And define training parameters.<jupyter_code>model_params = ModelParams(in_dim=300, 
                           out_dim=300, 
                           max_epochs=60, 
                           pos_ex=4, 
                           neg_ratio=1, 
                           learning_rate=0.001,
                           dropout=0.0, 
                           class_threshold=0.5)<jupyter_output><empty_output><jupyter_text>Define the model, it needs to know the training parameters, the number of emojis (size of parameter matrix depends on it) and the matrix of phrase embeddings (embeddings of emoji descriptions).<jupyter_code>model = Emoji2Vec(model_params=model_params, num_emojis=kb.dim_size(0), embeddings_array=embeddings_array)<jupyter_output><empty_output><jupyter_text>Now that the model has been defined, let's train it!<jupyter_code>model.train(kb=kb, epochs=model_params.max_epochs, learning_rate=model_params.learning_rate)<jupyter_output>/home/user/Documents/emoji2vec/e2v_env/lib/python3.6/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no predicted samples.
  'precision', 'predicted', average, warn_for)<jupyter_text>Let's save the trained PyTorch model to a file so that we can load it later if we need it.<jupyter_code>model_folder = 'example_emoji2vec'
if not os.path.isdir(model_folder):
    os.makedirs(model_folder)
    
torch.save(model.nn, model_folder + '/model.pt')<jupyter_output><empty_output><jupyter_text>Now that the model is trained and saved we need to extract emoji representations from the neural network's parameters. This can be done by a `model.Emoji2Vec` method called `create_gensim_files`, which will save the distributed representations of emojis in a format compatible with `gensim.models` allowing us to add Emoji2Vec representations to a gensim model and use them as any other word embeddings.<jupyter_code>e2v = model.create_gensim_files(model_folder=model_folder, ind2emoj=ind2emoji, out_dim=model_params.out_dim)<jupyter_output><empty_output><jupyter_text>Done.That's it, you've generated your own Emoji2Vec representations which now sit in `example_emoji2vec/` as `emoji2vec.txt` and `emoji2vec.bin`. Good job. Now is the time for the fun part, for example we can create a gensim model with our newly generated emoji embeddings.<jupyter_code>e2v = gsm.KeyedVectors.load_word2vec_format("example_emoji2vec/emoji2vec.bin", binary=True)<jupyter_output><empty_output><jupyter_text>Let's take a look at some of the emojis in the data set.<jupyter_code>vocabulary = e2v.vocab.keys()
# Sample 10 random emojis from the data set.
example_emojis = np.random.choice(list(vocabulary), 10)
print(example_emojis)<jupyter_output>['🇨🇺' '🙌🏿' '🌮' '☣' '👱🏿' '🇨🇫' '🇲🇵' '🕓' '🔪' '🐢']<jupyter_text>Each of the above emojis has its own vector representation.<jupyter_code>e2v.wv[example_emojis]<jupyter_output>/home/user/Documents/emoji2vec/e2v_env/lib/python3.6/site-packages/ipykernel_launcher.py:1: DeprecationWarning: Call to deprecated `wv` (Attribute will be removed in 4.0.0, use self instead).
  """Entry point for launching an IPython kernel.<jupyter_text>Thanks to the fact that the learnt emoji representations are compatible with `gensim.models`, we can do a lot of cool things like finding similar emojis.<jupyter_code>e2v.most_similar('🚔')
e2v.most_similar('🏊')
e2v.most_similar('🚄')
e2v.most_similar('🌩')<jupyter_output><empty_output><jupyter_text>However the real fun starts when we combine the power of emoji embeddings with word embeddings. Let's create a model that combines the two, this will allow us to measure similarity between emojis and phrases.<jupyter_code>phraseVecModel = Phrase2Vec.from_word2vec_paths(300,
                                                "data/word2vec/GoogleNews-vectors-negative300.bin.gz",
                                                "example_model/emoji2vec.bin")
# mapping from id to emoji
mapping = pk.load(open(params["mapping_file"], 'rb'))
# mapping from emoji to id
inverse_mapping = {v: k for k, v in mapping.items()}
def find_analogous_emoji(emoji_1,
                         emoji_2,
                         emoji_3,
                         top_n,
                         mapping=mapping,
                         inverse_mapping=inverse_mapping,
                         e2v=e2v,
                         model=model):
    similarities = []
    vector = e2v[emoji_1] - e2v[emoji_2] + e2v[emoji_3]
    vector = vector / np.linalg.norm(vector)
    
    for idx in range(len(mapping)):
        emoij_idx_similarity = model.nn.forward(torch.Tensor(vector.reshape(1, -1)), idx).detach().numpy()
        similarities.append(emoij_idx_similarity)
    
    similarities = np.array(similarities)
    n_most_similar_idxs = similarities.argsort(axis=0)[-top_n:][::-1].reshape(-1)
    n_most_similar_emojis = [mapping[emoji_idx] for emoji_idx in n_most_similar_idxs]
    
    str_expression = ' '.join([emoji_1, "-", emoji_2, "+", emoji_3,])
    top_score = similarities[n_most_similar_idxs[0]]
    
    return str_expression, n_most_similar_emojis
        
table = [["Expression", "Closest emojis"],
         [*find_analogous_emoji("🤴", "🚹", "🚺", 3)],
         [*find_analogous_emoji("👑", "🚹", "🚺", 3)],
         [*find_analogous_emoji("👦", "🚹", "🚺", 3)],
         [*find_analogous_emoji("💵", "🇺🇸", "🇬🇧", 3)],
         [*find_analogous_emoji("💵", "🇺🇸", "🇪🇺", 3)],
         [*find_analogous_emoji("💷", "🇬🇧", "🇪🇺", 3)]]
print(tabulate(table, headers="firstrow", tablefmt="rst"))<jupyter_output>============  ==================
Expression    Closest emojis
============  ==================
🤴 - 🚹 + 🚺  ['👸', '🚺', '🏰']
👑 - 🚹 + 🚺  ['👑', '👸', '🏰']
👦 - 🚹 + 🚺  ['🚺', '👸', '🐣']
💵 - 🇺🇸 + 🇬🇧  ['💵', '💷', '💶']
💵 - 🇺🇸 + 🇪🇺  ['💵', '💶', '💴']
💷 - 🇬🇧 + 🇪🇺  ['💷', '💶', '💵']
============  ==================<jupyter_text>Although far from perfect, similarly to the [published](https://arxiv.org/pdf/1609.08359.pdf) results an emoji considered to be "right" is usually within the top 3 examples selected by the model. Combining Emoji2Vec representations with word2vec embeddings they were trained on also allows to inspect the relationships between word embeddings and emoji embeddings. For example with the `phrase2emoji` function below we can see emojis most similar to an arbitrary phrase.<jupyter_code>def phrase2emoji(phrase, 
                 top_n,
                 phraseVecModel=phraseVecModel,
                 mapping=mapping,
                 model=model):
    
    similarities = []
    phrase_vec = phraseVecModel[phrase]
    for idx in range(len(mapping)):
        emoij_idx_similarity = model.nn.forward(torch.Tensor(phrase_vec.reshape(1, -1)), idx).detach().numpy()
        similarities.append(emoij_idx_similarity)
        
    similarities = np.array(similarities)
    n_most_similar_idxs = similarities.argsort(axis=0)[-top_n:][::-1].reshape(-1)
    n_most_similar_emojis = [mapping[emoji_idx] for emoji_idx in n_most_similar_idxs]
    
    return phrase, n_most_similar_emojis
    
table = [["Phrase", "Closest emojis"],
         [*phrase2emoji("funny", 3)],
         [*phrase2emoji("scary", 3)],
         [*phrase2emoji("okay", 3)],
         [*phrase2emoji("crazy", 3)],
         [*phrase2emoji("wild", 3)],
         [*phrase2emoji("afraid", 3)]]
print(tabulate(table, headers="firstrow", tablefmt="rst"))<jupyter_output>========  ==================
Phrase    Closest emojis
========  ==================
funny     ['😜', '👅', '😛']
scary     ['👹', '🎃', '😜']
okay      ['👌', '🆗', '🙆']
crazy     ['😜', '😝', '💭']
wild      ['🌵', '🐗', '🐆']
afraid    ['😨', '😟', '😞']
========  ==================
{'fraction_non_alphanumeric': 0.11574038087835212, 'fraction_numerical': 0.014768752429071124, 'mean_word_length': 4.292472233648704, 'pattern_counts': {'":': 4, '<': 66, '<?xml version=': 0, '>': 66, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '304993', 'license': 'MIT', 'n_tokens_mistral': 4319, 'n_tokens_neox': 4075, 'n_words': 1203, 'path': 'presentation.ipynb', 'repo_name': 'pwiercinski/emoji2vec_pytorch', 'size': '12865'}