<jupyter_start><jupyter_text>Create Adjacency MatricesWe create a series of adjacency matrices in which the nodes are the different ingredients found in the Recipes1M+ Database. We generate 3 different ingredient lists:1. Full ingredients found in the Recipes 1M+ database2. First words of ingredients in the Recipes 1M+ database3. Full ingredients in the Recipes 1M+ database that can be cross-referenced with ingredients in the USDA nutrition facts databaseWe then use each of these ingredient lists to create a variety of different adjacency matrices which we subsequently use for exploration, processing, graph filtering, ingredient suggestion, ingredient replacement, and more!  Part 1: Load data<jupyter_code>%matplotlib inline
import os, sys
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import json
from pandas.io.json import json_normalize
import itertools
from scipy.spatial.distance import pdist, squareform
from graphProcessingFunctions import *
#Load Recipes 1M+ Database
with open('../data/recipes_with_nutritional_info_fixed_qty.json') as json_file:
    data = json.load(json_file)
df = pd.DataFrame(data)
df.head()
print("The number of recipes in the Recipes 1M+ Database is {}.".format(len(df)))
#Load USDA nutrition facts database
usda = pd.read_excel('../data/USDA_Database.xlsx')
usda = usda[["SR description","Nutrient description", "Nutrient value"]]
usda.head()<jupyter_output><empty_output><jupyter_text>Part 2: Build ingredient lists from Recipes 1M+ Database<jupyter_code>all_ingredients_full = []
raw_ingredients = df[['ingredients']].values
for ri in raw_ingredients:
    for i in ri[0]:
        all_ingredients_full += [i['text']]
all_ingredients_full_unique = np.unique(all_ingredients_full)
print("There are {} unique ingredients.".format(len(all_ingredients_full_unique)))
print("Example ingredients: ")
print(all_ingredients_full_unique[10:15])
all_ingredients_firstword = []
for ing in all_ingredients_full:
    all_ingredients_firstword.append(ing.split(',')[0])
all_ingredients_unique = np.unique(all_ingredients_firstword,return_index=False)
print("There are {} unique ingredients (first words only).".format(len(all_ingredients_unique)))
print('Example ingredients: ')
print(all_ingredients_unique[10:15])
#Save ingredient lists
np.save('../data/Ingredient_List.npy', all_ingredients_full_unique)
np.save('../data/Ingredient_List_First_Word.npy', all_ingredients_unique)<jupyter_output><empty_output><jupyter_text>Adjacency Matrix No. 1Weights between ingredient nodes are equal to the number of recipes in which the two ingredients appear together in the Recipe1M+ dataset<jupyter_code>#Create adjacency matrix with weights equal to the number of recipes in which two ingredients appear together
Adj_mat_1 = np.zeros([len(all_ingredients_full_unique),len(all_ingredients_full_unique)])
ingredients_df = df['ingredients']
for recipe in ingredients_df.values:
    ingredients = []
    for ingredient in recipe:
        ingredients += [ingredient['text']]
    indices = np.where(np.in1d(all_ingredients_full_unique,ingredients))
    combos = list(itertools.combinations(indices[0],2))
    for combo in combos:
        Adj_mat_1[combo[0],combo[1]] += 1
        Adj_mat_1[combo[1],combo[0]] += 1
#Visualize and save
plt.spy(Adj_mat_1)
print(np.max(Adj_mat_1))
np.save('../data/Adjacency_Matrix_1.npy',Adj_mat_1)
am1 = np.unravel_index(np.argmax(Adj_mat_1, axis=None), Adj_mat_1.shape)
print("The ingredients most commonly used together are {0} and {1}.".format(all_ingredients_full_unique[am1[0]], all_ingredients_full_unique[am1[1]]))<jupyter_output>The ingredients most commonly used together are salt, table and wheat flour, white, all-purpose, unenriched.<jupyter_text>Adjacency Matrix No. 2Same as adjacency matrix 1 but using only the first word of each ingredient<jupyter_code>Adj_mat_2 = np.zeros([len(all_ingredients_unique),len(all_ingredients_unique)])
ingredients_df = df['ingredients']
for recipe in ingredients_df.values:
    ingredients = []
    for ingredient in recipe:
        ingredients += [ingredient['text'].split(',')[0]]
    indices = np.where(np.in1d(all_ingredients_unique,ingredients))
    combos = list(itertools.combinations(indices[0],2))
    for combo in combos:
        Adj_mat_2[combo[0],combo[1]] += 1
        Adj_mat_2[combo[1],combo[0]] += 1
plt.spy(Adj_mat_2)
print(np.max(Adj_mat_2))
np.save('../data/Adjacency_Matrix_2.npy',Adj_mat_2)<jupyter_output>10666.0<jupyter_text>Adjacency Matrix No. 3Build an adjacency matrix Cora-style (like we did in the first assignment): Have each recipe represent a "feature" for each ingredient. The feature equals 1 if the recipe includes that ingredient and 0 otherwise. Then use an RBF kernel to compute the distances between ingredients based on their feature vectors.<jupyter_code>num_recipes = len(df)
feature = np.zeros([len(all_ingredients_full_unique),num_recipes])
ingredients_df = df['ingredients']
recipe_counter = 0
for recipe in ingredients_df.values:
    ingredients = []
    for ingredient in recipe:
        ingredients += [ingredient['text']]
    indices = np.where(np.in1d(all_ingredients_full_unique,ingredients))
    feature[indices,recipe_counter] = 1
    recipe_counter = recipe_counter + 1
#Plot distances between nodes
distances = pdist(feature, metric='euclidean')
plt.hist(distances)
mean_dist = np.mean(distances)
print(mean_dist)
def epsilon_similarity_graph(X: np.ndarray, sigma=1, epsilon=0):
    """ X (n x d): coordinates of the n data points in R^d.
        sigma (float): width of the kernel
        epsilon (float): threshold
        Return:
        adjacency (n x n ndarray): adjacency matrix of the graph.
    """
    dist = squareform(pdist(X))
    adjacency = np.exp(- dist ** 2 / (2 * sigma ** 2))
    adjacency[adjacency < epsilon] = 0
    np.fill_diagonal(adjacency, 0)
    return adjacency
sigma_coeff = 0.8
sigma = sigma_coeff*mean_dist
adjacency = epsilon_similarity_graph(feature, sigma, epsilon=0.2)
plt.spy(adjacency)
plt.show()
#Save the matrix
np.save('../data/Adjacency_Matrix_3.npy',adjacency)<jupyter_output><empty_output><jupyter_text>Side noteTesting the credibility of nutrition information for each ingredient from the Recipes 1M+ database<jupyter_code>length = range(len(df))
for i in length:
    rec_ingr = df.iloc[i]["ingredients"]
    for ingredient_ndx in range(len(rec_ingr)):
        if rec_ingr[ingredient_ndx]['text'] == 'broccoli, raw':
            quantity = df.iloc[i]["quantity"][ingredient_ndx]['text']
            unit = df.iloc[i]["unit"][ingredient_ndx]['text']
            nutr = df.iloc[i]["nutr_per_ingredient"][ingredient_ndx]
            print("Recipe {0}: {1} {2} of broccoli nutrition facts: {3}".format(i,quantity,unit,nutr))<jupyter_output>Recipe 10: 1/2 cup of broccoli nutrition facts: {'fat': 0.17, 'nrg': 15.5, 'pro': 1.285, 'sat': 0.0175, 'sod': 15.0, 'sug': 0.775}
Recipe 11: 2 cup of broccoli nutrition facts: {'fat': 0.68, 'nrg': 62.0, 'pro': 5.14, 'sat': 0.07, 'sod': 60.0, 'sug': 3.1}
Recipe 14: 10 ounce of broccoli nutrition facts: {'fat': 1.0489315000000001, 'nrg': 96.38830000000002, 'pro': 7.994559, 'sat': 0.11056305, 'sod': 93.55335000000001, 'sug': 4.819415}
Recipe 67: 10 ounce of broccoli nutrition facts: {'fat': 1.0489315000000001, 'nrg': 96.38830000000002, 'pro': 7.994559, 'sat': 0.11056305, 'sod': 93.55335000000001, 'sug': 4.819415}
Recipe 187: 2 cup of broccoli nutrition facts: {'fat': 0.68, 'nrg': 62.0, 'pro': 5.14, 'sat': 0.07, 'sod': 60.0, 'sug': 3.1}
Recipe 239: 1 1/2 pound of broccoli nutrition facts: {'fat': 2.5174355999999998, 'nrg': 231.33192, 'pro': 19.186941599999997, 'sat': 0.26535131999999995, 'sod': 224.52803999999998, 'sug': 11.566595999999999}
Recipe 310: 2 cup of broccoli nutrition facts: {[...]<jupyter_text>We can conclude that there is a bug in the "nutrients per ingredient" section, because this is scaled to the quantity of the ingredient but is not consistent across units. For example, 10 oz of broccoli is less than 2 cups but has 34% more fat. Furthermore, some recipes contain erroneous quantities, such as 16 pounds of broccoli. Therefore, we use the USDA Nutrition Information Database to get nutrition statistics for each ingredient.  Adjacency Matrix No. 4The features of each ingredient are ten commonly-found nutrients parsed from the USDA database. If the full ingredient in the Recipes 1M+ database does not exist in the USDA database, match only the first word of the ingredient. Then, use an RBF kernel to create an adjacency matrix from the nutrition information.<jupyter_code>#Match first two words of unique ingredients to ingredients in USDA database
usda_ingr_names = usda["SR description"]
#save first two words
usda_ingr_first2words = []
usda_ingr_firstword = []
for ingr in usda_ingr_names:
    word_vec = ingr.split(',')
    if len(word_vec)>2:
        usda_ingr_first2words.append(word_vec[0].lower() + ',' + word_vec[1])
    else:
        usda_ingr_first2words.append(ingr.lower())
    if len(word_vec)>1:
        usda_ingr_firstword.append(word_vec[0].lower())
    else:
        usda_ingr_firstword.append(ingr.lower())
usda_ingr_first2words_unique = np.unique(usda_ingr_first2words)
usda_ingr_firstword_unique = np.unique(usda_ingr_firstword)
print(len(usda_ingr_first2words))
print(len(usda_ingr_firstword))
print(len(usda_ingr_first2words_unique))
print(len(usda_ingr_firstword_unique))
ingredients_matched = []; #List of all ingredients that are found in both the USDA and Recipes 1M+ datasets
ingredients_matched_mask = [];
count = 0;
for ingr in all_ingredients_full_unique:
    vec = ingr.split(',')
    if len(vec) > 2:
        ingr_name = vec[0] + ',' + vec[1]
    else:
        ingr_name = ingr
    if ingr_name in usda_ingr_first2words_unique:
        ingredients_matched.append(ingr)
        ingredients_matched_mask.append(True)
    else: #match only first word
        if vec[0] in usda_ingr_firstword_unique:
            ingredients_matched.append(ingr)
            ingredients_matched_mask.append(True)
        else:
            ingredients_matched_mask.append(False)
def getIngredientNutritionInfo(ingredient,stringType):
    if stringType == "firstWord":
        vecToUse = usda_ingr_firstword
    else:
        vecToUse = usda_ingr_first2words
    nutr_info = np.zeros([1,10])
    index_of_ingr = vecToUse.index(ingredient)
    fragment = usda.iloc[index_of_ingr:index_of_ingr+40]
    descriptions = fragment["Nutrient description"].values
    values = fragment["Nutrient value"].values
    for d in range(len(descriptions)):
        desc = descriptions[d]
        if (desc == "Protein"):
            nutr_info[:,0] = values[d]
        elif (desc == "Total Fat"):
            nutr_info[:,1] = values[d]
        elif desc == "Carbohydrate":
            nutr_info[:,2] = values[d]
        elif desc == "Energy":
            nutr_info[:,3] = values[d]
        elif desc == "Calcium":
            nutr_info[:,4] = values[d]
        elif desc == "Iron":
            nutr_info[:,5] = values[d]
        elif desc == "Sugars, total":
            nutr_info[:,6] = values[d]
        elif desc == "Sodium":
            nutr_info[:,7] = values[d]
        elif desc == "Vitamin B-12":
            nutr_info[:,8] = values[d]
        elif desc == "Fiber, total dietary":
            nutr_info[:,9] = values[d]
    return nutr_info
ingredient_nutrition_facts = np.zeros([len(ingredients_matched),10])
for i in range(len(ingredients_matched)):
    ingr = ingredients_matched[i]
    vec = ingr.split(',')
    if len(vec) > 2:
        ingr_name = vec[0] + ',' + vec[1]
    else:
        ingr_name = ingr
    if ingr_name in usda_ingr_first2words_unique:
        nutr = getIngredientNutritionInfo(ingr_name,"firstTwoWords")
        ingredient_nutrition_facts[i,:] = nutr
    else: #match only first word
        ingr_name = vec[0]
        if ingr_name in usda_ingr_firstword_unique:
            nutr = getIngredientNutritionInfo(ingr_name,"firstWord")
            ingredient_nutrition_facts[i,:] = nutr
distances = pdist(ingredient_nutrition_facts, metric='euclidean')
plt.hist(distances)
mean_dist = np.median(distances)
print(mean_dist)
sigma_coeff = 0.8
sigma = sigma_coeff*mean_dist
adjacency = epsilon_similarity_graph(ingredient_nutrition_facts, sigma, epsilon=0.2)
plt.spy(adjacency)
plt.show()
plt.hist(np.ravel(adjacency),bins=20)
np.save('../data/Ingredient_List_USDA_Nutrition_Info.npy', ingredients_matched)
np.save('../data/Adjacency_Matrix_4.npy', adjacency)<jupyter_output><empty_output><jupyter_text>Adjacency Matrix No. 5Normalize the nutrition facts to the mean nutrition fact so that some nutrients are not weighted more than others<jupyter_code>ingredient_nutrition_facts_normalized = ingredient_nutrition_facts / np.mean(ingredient_nutrition_facts, axis=0)
distances = pdist(ingredient_nutrition_facts_normalized, metric='euclidean')
plt.hist(distances)
mean_dist = np.median(distances)
print(mean_dist)
sigma_coeff = 0.8
sigma = sigma_coeff*mean_dist
adjacency = epsilon_similarity_graph(ingredient_nutrition_facts_normalized, sigma, epsilon=0.2)
plt.spy(adjacency)
plt.show()
np.save('../data/Adjacency_Matrix_5.npy', adjacency)<jupyter_output><empty_output><jupyter_text>Adjacency Matrix No. 6Same as 5 but sparsified for vizualization and community detection purposes<jupyter_code>sigma_coeff = 0.4
sigma = sigma_coeff*mean_dist
adjacency = epsilon_similarity_graph(ingredient_nutrition_facts_normalized, sigma, epsilon=0.1)
plt.spy(adjacency)
plt.show()
plt.hist(np.ravel(adjacency),bins=20)
np.save('../data/Adjacency_Matrix_6.npy', adjacency)
#Save nutrition facts matrix and nutrient labels for future reference
np.save('../data/Nutrition_Facts_Per_Ingredient.npy', ingredient_nutrition_facts)
nutrition_fact_column_labels = ["Protein", "Total Fat","Carbohydrate","Energy","Calcium","Iron","Sugars, total","Sodium","Vitamin B-12","Fiber, total dietary"]
np.save('../data/Nutrition_Facts_Column_Labels.npy', nutrition_fact_column_labels)<jupyter_output><empty_output><jupyter_text>Find vegan ingredientsVitamin B-12 is only found in animal products. Therefore, we find all vegan ingredients by thresholding the ingredient list by their B-12 quantity.<jupyter_code>vegan_ingredient_mask = ingredient_nutrition_facts[:,8]  < 0.05
vegan_ingredients = np.array(ingredients_matched)[vegan_ingredient_mask]
problems = ['animal fat, bacon grease', 'fat, goose', 'lard', 'soup, chicken broth or bouillon, dry', 'soup, cream of chicken, canned, condensed','soup, swanson chicken broth 99% fat free','margarine-like, margarine-butter blend, soybean oil and butter']
indices = []
for problem in problems:
    index = vegan_ingredients.tolist().index(problem)
    indices.append(index)
vegan_ingredients = np.delete(vegan_ingredients,indices)
vegan_ingredient_mask[indices] = False
print('Some vegan ingredients:')
print(vegan_ingredients[20:25])
print("There are {} vegan ingredients".format(len(vegan_ingredients)))
np.save('../data/Vegan_Ingredient_Mask_USDA_Nutrition_Info.npy', vegan_ingredient_mask)<jupyter_output><empty_output><jupyter_text>Adjacency Matrix No. 7Concatenate recipe fetures and nutrition featues and then apply an RBF kernel. Normalize to the feature mean.Potential problem: There are more recipe features than nutrition features, so these may be weighted more heavily<jupyter_code>print(ingredient_nutrition_facts.shape)
feature_matched = feature[ingredients_matched_mask]
print(feature_matched.shape)
#Try concatenating both
combined_feature_matrix = np.zeros([ingredient_nutrition_facts.shape[0], ingredient_nutrition_facts.shape[1] + feature_matched.shape[1]])
combined_feature_matrix[:,0:(ingredient_nutrition_facts.shape[1])] = ingredient_nutrition_facts
combined_feature_matrix[:,ingredient_nutrition_facts.shape[1]:(ingredient_nutrition_facts.shape[1]+ feature_matched.shape[1])] = feature_matched
#Normalize features based on mean value
#Possible problem: there are many more recipe features than nutrition ones
feature_means = np.mean(combined_feature_matrix,axis=0)
combined_feature_matrix = combined_feature_matrix[:, feature_means != 0]
feature_means = feature_means[feature_means != 0]
combined_feature_matrix_normalized = combined_feature_matrix / feature_means;
print(combined_feature_matrix_normalized.shape)
distances = pdist(combined_feature_matrix_normalized, metric='euclidean')
plt.hist(distances)
mean_dist = np.median(distances)
print(mean_dist)
sigma_coeff = 0.8
sigma = sigma_coeff*mean_dist
adjacency = epsilon_similarity_graph(combined_feature_matrix_normalized, sigma, epsilon=0.2)
plt.spy(adjacency)
plt.show()
np.save('../data/Adjacency_Matrix_7.npy', adjacency)<jupyter_output><empty_output><jupyter_text>Adjacency Matrix 8Same as 7 but without feature normalization (to try to weigh nutrition information more than recipe features)<jupyter_code>distances = pdist(combined_feature_matrix, metric='euclidean')
plt.hist(distances)
mean_dist = np.median(distances)
print(mean_dist)
sigma_coeff = 0.8
sigma = sigma_coeff*mean_dist
adjacency = epsilon_similarity_graph(combined_feature_matrix, sigma, epsilon=0.2)
plt.spy(adjacency)
plt.show()
laplacian = compute_laplacian(adjacency,True)
lamb_norm, U_norm = spectral_decomposition(laplacian)
compute_number_connected_components(lamb_norm,threshold=1e-12)
np.save('../data/Adjacency_Matrix_8.npy', adjacency)<jupyter_output><empty_output><jupyter_text>Adjacency Matrix 9:Try to combine the ingredient coocurrence adjacency matrix (1) with the nutrition adjacency matrix (5) by adding them. Make sure that the coocurrence weights are not weighted more heavily than the nutrition weights by dividing the cooccurrence matrix by its maximum value.<jupyter_code>distances = pdist(ingredient_nutrition_facts_normalized, metric='euclidean')
mean_dist = np.median(distances)
sigma_coeff = 0.8
sigma = sigma_coeff*mean_dist
adjacency = epsilon_similarity_graph(ingredient_nutrition_facts_normalized, sigma, epsilon=0.2)
normalized_recipe_adj_mat = Adj_mat_1/np.max(Adj_mat_1)
normalized_recipe_adj_mat = normalized_recipe_adj_mat[ingredients_matched_mask,:]
normalized_recipe_adj_mat = normalized_recipe_adj_mat[:,ingredients_matched_mask]
combined_mat = normalized_recipe_adj_mat + adjacency
plt.spy(combined_mat)
plt.show()
np.save('../data/Adjacency_Matrix_9.npy', combined_mat)<jupyter_output><empty_output><jupyter_text>Adjacency Matrix 10: Same as 9 but divide the recipes matrix by its mean value instead of the max to increase its weight<jupyter_code>normalized_recipe_adj_mat = Adj_mat_1/np.mean(Adj_mat_1)
normalized_recipe_adj_mat = normalized_recipe_adj_mat[ingredients_matched_mask,:]
normalized_recipe_adj_mat = normalized_recipe_adj_mat[:,ingredients_matched_mask]
combined_mat = normalized_recipe_adj_mat + adjacency
plt.spy(combined_mat)
plt.show()
np.save('../data/Adjacency_Matrix_10.npy', combined_mat)<jupyter_output><empty_output><jupyter_text>Adjacency Matrix 11Combine the two matrices by multiplyting them<jupyter_code>normalized_recipe_adj_mat = Adj_mat_1/np.max(Adj_mat_1)
normalized_recipe_adj_mat = normalized_recipe_adj_mat[ingredients_matched_mask,:]
normalized_recipe_adj_mat = normalized_recipe_adj_mat[:,ingredients_matched_mask]
combined_mat = np.multiply(normalized_recipe_adj_mat, adjacency)
plt.spy(combined_mat)
plt.show()
np.save('../data/Adjacency_Matrix_11.npy', combined_mat)<jupyter_output><empty_output><jupyter_text>Adjacency Matrix No. 12Add the nutrition and recipe adjacency matrices made using RBF kernels (3 and 5)<jupyter_code>mat_3 = np.load('../data/Adjacency_Matrix_3.npy')
mat_3 = mat_3[ingredients_matched_mask,:]
mat_3 = mat_3[:,ingredients_matched_mask]
mat_5 = np.load('../data/Adjacency_Matrix_5.npy')
combined_mat = mat_3 + mat_5
plt.spy(combined_mat)
plt.show()
np.save('../data/Adjacency_Matrix_12.npy', combined_mat)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1071302417116617, 'fraction_numerical': 0.029267800373416763, 'mean_word_length': 5.836150396688513, 'pattern_counts': {'":': 9, '<': 64, '<?xml version=': 0, '>': 66, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 11, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '229830', 'license': 'MIT', 'n_tokens_mistral': 7292, 'n_tokens_neox': 6594, 'n_words': 1536, 'path': 'notebooks/01_CreateAdjacencyMatrices.ipynb', 'repo_name': 'ValentinViennot/ntds_kitchen_science', 'size': '19817'}