<jupyter_start><jupyter_text>Interactive Recommendation System with Word Embeddings using Word2Vec, Plotly, and NetworkXHere, We will be exploring how to build a model that can understand words in a mathematical way, such that words with similar meaninings that share certain charactertics belong close to each other in vector space. That's a fancy way of saying that the mathemical representations for words that have some similar meaning are close to each other, and words that don't share a lot of meaning are further apart. We will show that this model can be adapted to be used in a recommendation system, or as a discovery tool (i.e. we don't have this in stock, maybe you'd like this alternative?). 1. IntroductionWe will be using a recipe dataset, to train a model to learn the interactions between different kind of ingredients and available products in a supermarket. This model can then be implemented in a number of different ways, for example,- to recommend products based on items added to cart;- to offer alternatives products based on stock;- to discover new products to create different recipes. Sample*Herb Rubbed Sirloin Tip Roast*- 1 1/4 tablespoons paprika- 1 tablespoon kosher salt- 1 teaspoon garlic powder- 1/2 teaspoon ground black pepper- 1/2 teaspoon onion powder- 1/2 teaspoon ground cayenne pepper- 1/2 teaspoon dried oregano- 1/2 teaspoon dried thyme- 2 tablespoons olive oil- 1 (3 pound) sirloin tip roastIn a small bowl, mix the paprika, kosher salt, garlic powder, black pepper, onion powder, cayenne pepper, oregano, and thyme. Stir in the olive oil, and allow the mixture to sit about 15 minutes.Preheat oven to 350 degrees F (175 degrees C). Line a baking sheet with aluminum foil.Place the roast on the prepared baking sheet, and cover on all sides with the spice mixture.Roast 1 hour in the preheated oven, or to a minimum internal temperature of 145 degrees F (63 degrees C). Let sit 15 minutes before slicing.  2. Exploratory Data Analysis and Preprocessing[This is the dataset](https://eightportions.com/datasets/Recipes/fn:1) we will be using. It is collated by <NAME>, sourced from [Food Network](https://www.foodnetwork.com/), [Epicurious](https://www.epicurious.com/), and [Allrecipes](https://www.allrecipes.com/).<jupyter_code>from gensim.parsing.preprocessing import remove_stopwords
from tqdm import tqdm
import pandas as pd
import pickle
import string
import json
recipe_sources = ['ar', 'epi', 'fn']
df = pd.DataFrame()
sources, titles, ingredients, instructions = [], [], [], []
for recipe_source in recipe_sources:
    data = json.load(open(f'Data/Dataset/recipes_raw_nosource_{recipe_source}.json', 'r'))
    for _, recipe in data.items():
        if ('title' in recipe) and ('ingredients' in recipe) and ('instructions' in recipe):
            sources.append(recipe_source)
            titles.append(recipe['title'])
            ingredients.append([str(ingredient).replace('ADVERTISEMENT', '') for ingredient in recipe['ingredients']])
            instructions.append(str(recipe['instructions']).replace('ADVERTISEMENT', '').replace('\n', ' '))
            
df['source'] = sources
df['title'] = titles
df['ingredients'] = ingredients
df['instructions'] = instructions
to_remove = [
    'tablespoon',
    'tablespoons',
    'teaspoon',
    'teaspoons',
    'tsp',
    'tsps',
    'tbsp',
    'tbsps',
    'pound',
    'pounds',
    'grams',
    'mg',
    'ounce'
    'ounces',
    'kg',
    'crushed',
    'chopped',
    'finely',
    'softened',
    'cups',
    'cup'
]

translation_table = str.maketrans('', '', string.punctuation+string.digits)
def preprocess(items):
    res = []
    for i, item in enumerate(tqdm(items)):
        temp = item.lower().replace('-', ' ')
        temp = temp.translate(translation_table)
        temp = remove_stopwords(temp)
        for stop_word in to_remove:
            temp = temp.replace(stop_word, '')
        res.append(temp.split())
    return res
instructions = df.instructions.values.tolist()
ingredients = [', '.join(x) for x in df.ingredients.values]
train_data = preprocess(instructions+ingredients)
with open('Data/train_data.pkl', 'wb') as f:
    pickle.dump(train_data, f)<jupyter_output><empty_output><jupyter_text>3. Word2Vec with GensimWord2Vec original papers can be found [here](https://arxiv.org/pdf/1301.3781.pdf) and [here](https://arxiv.org/pdf/1310.4546.pdf), while the documentation for the Gensim model can be found [here](https://radimrehurek.com/gensim/models/word2vec.html).![Word2Vec architecture](Data/word2vec.jpeg)<jupyter_code>from gensim.models.word2vec import Word2Vec
with open('Data/train_data.pkl', 'rb') as f:
    train_data = pickle.load(f)
model = Word2Vec()
model?
model.build_vocab(train_data)
%%time
model.train(train_data, total_examples=model.corpus_count, epochs=model.epochs)
res = model.wv.most_similar('lemon chicken garlic onion'.split(), topn=20)
res
model.save('Data/w2v.model')<jupyter_output><empty_output><jupyter_text>4. Exploring Results<jupyter_code>from sklearn.metrics.pairwise import cosine_similarity
from gensim.models.word2vec import Word2Vec
import numpy as np

import matplotlib
%matplotlib inline
%config InlineBackend.figure_format = 'svg'
#matplotlib.rcParams['figure.dpi'] = 800
import matplotlib.pyplot as plt
plt.style.use('ggplot')

import random
random.seed(3)
model = Word2Vec.load('Data/w2v.model')
inferred_vector = model.wv.most_similar('lemon chicken garlic onion'.split(), topn=20)
inferred_vector
with open('Data/items_for_sale.txt') as f:
    items_for_sale = f.read().split('\n')
words, vectors = [], []
for item in items_for_sale:
    try:
        words.append(item)
        vectors.append(model.wv.get_vector(item))
    except KeyError:
        print(f'Word {item} not found in model!')
sims = cosine_similarity(vectors, vectors)
for i in range(len(vectors)):
    for j in range(len(vectors)):
        if i<=j:
            sims[i, j] = False
k = 20
indices = list(range(len(vectors)))
random.shuffle(indices)
indices = indices[120:120+k]

small_vectors = [vectors[i] for i in indices]
small_words = [words[i] for i in indices]

small_sims = cosine_similarity(small_vectors, small_vectors)

for x in range(len(small_sims)):
    small_sims[x, x] = 0
# adapted from https://matplotlib.org/3.1.1/gallery/images_contours_and_fields/image_annotated_heatmap.html

fig, ax = plt.subplots()
im = ax.imshow(small_sims)

ax.set_xticks(np.arange(len(small_vectors)))
ax.set_yticks(np.arange(len(small_vectors)))

ax.set_xticklabels(small_words)
ax.set_yticklabels(small_words)
ax.grid(False)

plt.setp(ax.get_xticklabels(), rotation=90)

fig.tight_layout()
plt.savefig('plot2.png', dpi=300)
plt.show()
np.savetxt('Data/words.txt', words, fmt='%s')
np.save('Data/vectors.npy', vectors)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1163716814159292, 'fraction_numerical': 0.014011799410029498, 'mean_word_length': 4.599504541701074, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 14, 'https://': 8, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '272341', 'license': 'MIT', 'n_tokens_mistral': 2259, 'n_tokens_neox': 2117, 'n_words': 653, 'path': 'Interactive Word Embeddings using Word2Vec and Plotly/1. Notebook.ipynb', 'repo_name': 'padam56/Data-Analysis-and-Visualization-Projects', 'size': '6782'}