<jupyter_start><jupyter_text>Visualizing Embedding<jupyter_code>!pip install -Uqq fastbook
import fastbook
fastbook.setup_book()
#hide
from fastbook import *
# from kaggle import api
from pandas.api.types import is_string_dtype, is_numeric_dtype, is_categorical_dtype
from fastai.tabular.all import *
from sklearn.ensemble import RandomForestRegressor, RandomForestClassifier
from sklearn.tree import DecisionTreeRegressor
# from dtreeviz.trees import *
from IPython.display import Image, display_svg, SVG

pd.options.display.max_rows = 20
pd.options.display.max_columns = 8
from sklearn.model_selection import ParameterGrid
from keras.models import Sequential
from keras.models import Model as KerasModel
from keras.layers import Input, Dense, Activation, Reshape
from keras.layers import Concatenate
from keras.layers import Dense, Flatten, Dropout, Input, Embedding, Reshape
from keras.layers import concatenate
from sklearn.ensemble import RandomForestRegressor
from sklearn import linear_model
import pickle
import csv
from datetime import datetime
from sklearn import preprocessing
from keras.callbacks import ModelCheckpoint
import xgboost as xgb
from sklearn.preprocessing import OneHotEncoder
%matplotlib inline

from sklearn.model_selection import train_test_split
import math
import pandas as pd

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OrdinalEncoder<jupyter_output><empty_output><jupyter_text>What we could do here is visualize the entity embedding, we might not able to gather anything huge, but somethings like Asin_embedding and hour embedding might give us some info. And also this is something I can use somewhere else too. The beauty of embedding that it starts as an assumption or random numbers,creates a meaning btw the numbers, which is awesome, which means it might be able to show relations that maybe even we miss.  The reason is whenever I looked at certain codes, everything seems confusing to me. but that is the fun of this from confusion of where to work we focused of visualizing the embeddings, and from there it might be again a confusing road but i think that is okay, rather than trying to visualize lets break down and see what is going to happen. We can put out a theory which can help us on how to proceed.the theory is , we take one embedding like asin_embedding this will have a rows of list of numbers, say 10 columns of list, with as many rows as the number of the data we started with, but we have given embeddings not the same size of unique asins?, should we give the size of the same , but i didnt observed that anywhere else, but if we have not given the same size, what will they represent, the chance is they would represent, maybe like groups of asin, but then how would you know which asins are which. but i think one of the input we have given is unique asin number, so there would be some data which represents that, and we can use that.What i found out is two things1) that we will have unqiue number of list(embeddings), each embeddings is a list of numbers and this number of embeddings = number of unique asins. And then we use PCA analysis to understand what it is. exciting news is we will also learn about PCA.2) we can try this embedding in fastai version of the learner, when we create our own module, i think it wont be that confusing, and it is simple(not easy), where we create a class from module, and it that we pass in the embedding details and also at the end, we can get this by calling something like learner.embs etc..One thing we can try later as well is using embeddings of fastai, use the collab chapter one, there there is only two variables, they have considered it both cat variables.  Lets look  embeddings in detail<jupyter_code>infile = open('/content/gdrive/MyDrive/bluebook/df_main.pkl','rb')
df_main = pickle.load(infile)
df_main = add_datepart(df_main,'date')
condition = (df_main.Year < 2020) | (df_main.Month < 8)
train_idx = np.where(condition)[0]
valid_idx = np.where(~condition)[0]
cat = ['Asin',
 'Is_month_end',
 'Is_month_start',
 'Is_quarter_end',
 'Is_quarter_start',
 'Is_year_end',
 'Is_year_start',
 'hour','Month','Week', 'Day','Dayofweek','Dayofyear']
enc = OrdinalEncoder()
enc.fit(df_main[cat])
df_main[cat]= enc.transform(df_main[cat])
X =df_main.drop(['Quantity'],axis=1)
saved_embeddings_fname = '/content/gdrive/MyDrive/bluebook/embeddings.pickle'
def embed_features(X, saved_embeddings_fname):
    f_embeddings = open(saved_embeddings_fname, "rb")
    embeddings = pickle.load(f_embeddings)

    index_embedding_mapping = {0: 0, 3: 1, 9: 2, 6: 3, 8: 4, 7: 5}
    X_embedded = []

    (num_records, num_features) = X.shape
    for record in X: 
        embedded_features = []
        for i, feat in enumerate(record):
            feat = int(feat)
            if i not in index_embedding_mapping.keys():
                embedded_features += [feat]
            else:
                embedding_index = index_embedding_mapping[i]
                embedded_features += embeddings[embedding_index][feat].tolist()

        X_embedded.append(embedded_features)

    return np.array(X_embedded)
X_emb = embed_features(X.values,saved_embeddings_fname)
X_train = list(itemgetter(*train_idx)(X_emb)) 
X_valid = list(itemgetter(*valid_idx)(X_emb)) 

# or we can use list comprehension too for ex, test = [X_emb[i] for i in train_idx ]
y_train = df_main['Quantity'].iloc[train_idx]

y_valid = df_main['Quantity'].iloc[valid_idx]
X_emb.shape
array(X_train).shape
df_main.shape<jupyter_output><empty_output><jupyter_text>The difference in column sizes can be attributed of X_embed and df_main , to the embedding sizes of 6 columns which is 6+4+3+7+5+10 =35, this 35 new columns minus 6 columns of which we applied embedding and 1 column of quanity gives us the extra 28 columns<jupyter_code>X_emb[0].shape

f_embeddings = open(saved_embeddings_fname, "rb")
embeddings = pickle.load(f_embeddings)
array(embeddings).shape
array(embeddings[0]).shape #Asin<jupyter_output><empty_output><jupyter_text>When we look at the saved embeddings what we notice is that, espiecally the shapes of the embedding, there are 6. This 6 are the embeddings we created for the 6 categories.When we look inside each of the 6 embedding what we can see is that, each embedding has the shape specified by us in the begining, for example, unique Asins and the number of ouput we specified. Now what happened in the function above is that we replace for example asin one, with the values in the embedding[0] (which is asin) and within that the asin one, will be a list of numbers of size we defined earlier. In the asin case it is 46.<jupyter_code>embeddings[0][0] # for example this is the list we can use to replace the asin one<jupyter_output><empty_output><jupyter_text>In papers code they have given github links for projects they have used entity embeddings, this is one such project, which we can refer to create viz for embeddings.-https://github.com/entron/entity-embedding-rossmann/blob/master/plot_embeddings.ipynb  Entity Embedding Visualization<jupyter_code>embeddings[0].shape # For asin<jupyter_output><empty_output><jupyter_text>We need to know the order in which the embeddings are created, because we have 46 asins and 46 embeddings for these asins, assuming that embeddings are in the same order as that for unique we are proceeding. We will come back to this later.<jupyter_code>df_main['Asin'].unique()
df_main['Asin'].unique()
import pickle
from sklearn import manifold
import matplotlib.pyplot as plt
import numpy as np
%matplotlib inline
Asin_names = df_main['Asin'].unique()
asin_embedding = embeddings[0]


tsne = manifold.TSNE(init='pca', random_state=0, method='exact', perplexity=5, learning_rate=100)
Y = tsne.fit_transform(asin_embedding)
plt.figure(figsize=(15,15))
plt.scatter(-Y[:, 0], -Y[:, 1])
for i, txt in enumerate(Asin_names):
    plt.annotate(txt, (-Y[i, 0],-Y[i, 1]), xytext = (-20, 8), textcoords = 'offset points')
plt.savefig('asin_embedding.pdf')


df_main['Asin'].unique()
B08332221J - Shower head filter
B08ZDFSC56 - Nemora Dec light-05
B07ZVX96TY - Wooden frame<jupyter_output><empty_output><jupyter_text>B0897LPBHR - 10STAGEB08ZDC735Z - Wooden light  Hour<jupyter_code>hours = df_main['hour'].unique()
hours
hour_embedding = embeddings[1]
hour_embedding.shape


tsne = manifold.TSNE(init='pca', random_state=0, method='exact', perplexity=5, learning_rate=100)
Y = tsne.fit_transform(hour_embedding)
plt.figure(figsize=(15,15))
plt.scatter(-Y[:, 0], -Y[:, 1])
for i, txt in enumerate(hours):
    plt.annotate(txt, (-Y[i, 0],-Y[i, 1]), xytext = (-20, 8), textcoords = 'offset points')
plt.savefig('hours.pdf')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0796918123275069, 'fraction_numerical': 0.014029438822447102, 'mean_word_length': 5.176846590909091, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 29, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '541007', 'license': 'MIT', 'n_tokens_mistral': 2753, 'n_tokens_neox': 2606, 'n_words': 1031, 'path': 'project_01_05_visualizing_embedding.ipynb', 'repo_name': 'akshaypt7/Cohort-Analysis', 'size': '8696'}