<jupyter_start><jupyter_text>The Ultimate Food Finder The Ultimate Food Finder is an app that helps you find dishes that are not only good for your health but also delicious and easy to make by your standards. Almost 4 in 10 US adults are obese according to Centers for Disease Control and Prevention, costing the country $147 billion in medical bills in 2008 (Overweight & Obesity, 2018). One of the main reasons is believed to be eating behaviors, and it is not always lack of responsibility on our part that we do not eat in a healthy way. Pew Research Center did a national survey of 2,250 adults and found that 73% of those who eat junk food do so because of convenience. The main assumption of this proposal is that people do not eat unhealthy food by choice but necessity. Therefore, a tool for them to discover food with the optimal balance of health, taste and convenience will lead to a more healthy lifestyle and minimize healthcare spendings.We sift through over 240,000 recipes on allrecipes.com to bring you the right recipe based on nutritional facts, quantity and complexity of instructions, time taken to prepare the dish, prevalence of ingredients, ratings and reviews of the recipe and from user accounts. As a proof of concept, we successfully scraped about 1% of the dataset and created plots showing tradeoffs between conveience and nutritional values as well as deliciousness.<jupyter_code>import re
import numpy as np
import pandas as pd
from collections import Counter
import matplotlib.pyplot as plt
import seaborn as sns
from datetime import datetime
import warnings
warnings.filterwarnings("ignore")

DATA_PATH='data/'
food_df = pd.read_csv(f'{DATA_PATH}food_df.csv')<jupyter_output><empty_output><jupyter_text>Figure 1: The Convenience-Health Tradeoff Figure 1 is a scatterplot of recipies with highest vitamins as percentage of daily intake on one axis and time used to prepare and cook the meal on the other. We also did a clustering on the recipes to determine groups of similar dishes. This allows us to choose to make the group of dishes that take less time and are more healthy.   Minutes of Ready Time We will use minutes of ready time as a proxy for convenience. Here we show the distribution of ready time, as well as top 10 fastest and slowest food to cook.<jupyter_code>def unit_cov(t):
    if t=='d':
        return(60*24)
    elif t=='h':
        return(60)
    else:
        return(1)

def text_to_min(ready_time):
    if type(ready_time)!=str: return(0)
    dur = 0
    time_list = ready_time.split()
    for i in range(len(time_list)-1):
        if time_list[i].isdigit():
            dur+=int(time_list[i])*unit_cov(time_list[i+1])
    return(dur)

food_df['ready_min'] = food_df['ready_time'].apply(text_to_min)
nonzero_mins = food_df.query('ready_min>0')['ready_min']
g = sns.distplot(nonzero_mins,bins=100)
g.set(xlim=(0, 1000))
nonzero_mins.describe()
#top 10 fastest food to cook
food_df.query('ready_min>0').sort_values('ready_min')[['recipe_name','ready_min']].head(10)
#top 10 slowest food to cook
food_df.query('ready_min>0').sort_values('ready_min',ascending=False)[['recipe_name','ready_min']].head(10)<jupyter_output><empty_output><jupyter_text>Vitamin Score We use the amount of vitamins per serving as proportion of recommended daily intake as an indicator of a healthy meal. The vitamin proportions are weighted equally namely:* Vitamin A* Vitamin C* Vitamin B6* Thiamin* Niacin* FolateDaily amount of vitamin intake is taken from Institute of Medicine (1970) for Males over 19 years old.<jupyter_code>DAILY_VITAMIN_A = 3000 #IU
DAILY_VITAMIN_C = 90 #mg
DAILY_VITAMIN_B6 = 1.3 #mg
DAILY_THIAMIN = 1.2 #mg
DAILY_NIACIN = 16 #mg
DAILY_FOLATE = 400 #mcg

food_df['multi_vitamin'] = food_df.vitamin_a / DAILY_VITAMIN_A + \
    food_df.vitamin_c / DAILY_VITAMIN_C + food_df.vitamin_b6 / DAILY_VITAMIN_B6 +\
    food_df.thiamin / DAILY_THIAMIN + food_df.niacin / DAILY_NIACIN +\
    food_df.folate / DAILY_FOLATE

sns.distplot(food_df['multi_vitamin'].dropna())
food_df['multi_vitamin'].describe()
#top 10 food with highest amount of vitamins
food_df.sort_values('multi_vitamin',ascending=False)[['recipe_name','multi_vitamin']].head(10)
#top 10 food with lowest amount of vitamins
food_df.sort_values('multi_vitamin',ascending=True)[['recipe_name','multi_vitamin']].head(10)<jupyter_output><empty_output><jupyter_text>Convenience-Health Clusters We use k-means clustering to group all recipes into arbitrary groups so we can explore in the next section which food groups are the best in terms of convenience-health tradeoff. We remove ready time and vitamin score outliers using interquartile range (remove all recipes above `Q3 + 1.5×IQR`), then mean center and normalize both numbers by standard deviation before performing the clustering.<jupyter_code>#filter by IQR
plot1_df = food_df.query('ready_min>0')[['ready_min','multi_vitamin','nb_calories','recipe_name']]
plot1_df.set_index('recipe_name',inplace=True)
ready_min_stats = plot1_df['ready_min'].describe()
ready_min_thres = ready_min_stats[6]+1.5*(ready_min_stats[6]-ready_min_stats[4])
multi_vitamin_stats = plot1_df['multi_vitamin'].describe()
multi_vitamin_thres = multi_vitamin_stats[6]+1.5*(multi_vitamin_stats[6]-multi_vitamin_stats[4])
plot1_df = plot1_df.query(f'ready_min<{ready_min_thres} & multi_vitamin < {multi_vitamin_thres}')
#standardize
from sklearn.preprocessing import scale
scaled1 = scale(plot1_df)
#determine number of clusters
from sklearn.cluster import KMeans  
ks = range(1, 10)
fits = [KMeans(n_clusters=k).fit(scaled1) for k in ks]
inertia = [fits[i].inertia_ for i in range(len(fits))]
plt.plot(inertia)
#optimal fit
k=3
fit = KMeans(n_clusters=k).fit(scaled1)
plot1_df['cluster'] = fit.labels_
plot1_df['cluster'] = plot1_df['cluster']
Counter(plot1_df['cluster'])
#first cluster
plot1_df[plot1_df.cluster==0].head()
#second cluster
plot1_df[plot1_df.cluster==1].head()
#first cluster
plot1_df[plot1_df.cluster==2].head()<jupyter_output><empty_output><jupyter_text>The Healthiest Food Can Take Less Time When we look at the scatterplot, we can see that the cluster of food with high vitamin scores can also take about the same amount of time as one cluster and significnatly less than the other. This invalidates the claim (excuse?) that it takes more time to cook healthy food.<jupyter_code>sns.set_style(style='white')
g = sns.relplot(x='ready_min', y='multi_vitamin',
             size='nb_calories', hue = 'cluster',
             alpha=0.5, height = 6, palette='Accent',
             data=plot1_df)
g.set(xlabel='Minutes to Prepare and Cook', ylabel='Vitamin Score',
      title='Healthy Food Can Take Less Time')<jupyter_output><empty_output><jupyter_text>Figure 2: The Convenience-Taste Tradeoff Figure 2 is another scatterplot of recipies with ratings (normalized by number of reviews) on one axis and number of ingredients on the other (each ingredient is weighted using inverse document frequency to give more weights to rare ingredients). We can look at this plot and choose the dish that is delicious (by popular demands) and easy to make.  Number of (Weighted) Ingredients We use the number of ingredients mentioned in the recipe as another measure of convenience. We also weight them using inverse document frequency in order to give more weights to ingredients that rarely appear (and thus are less convenient to find).<jupyter_code>ingre_list = []
for i in range(len(food_df)):
    ingre_list += food_df.iloc[i,food_df.columns.get_loc('ingre_ids')].split('|')
c = Counter(ingre_list)
print(f'There are {len(c)} ingredients.')
ingre_cols = list(c.keys())
#get idf (because term frequency is always 1; all ingredient ids appear only once for each recipe)
from sklearn.feature_extraction.text import TfidfVectorizer
tokenize = lambda x: x.split('|')
tfidf = TfidfVectorizer(min_df=0, use_idf=True, tokenizer=tokenize)
ingredient_mat = tfidf.fit_transform(food_df['ingre_ids'])
#sum to get (weighted) number of ingredients
food_df['nb_ingredients'] = np.sum(ingredient_mat,axis=1)
sns.distplot(food_df['nb_ingredients'])
food_df['nb_ingredients'].describe()<jupyter_output><empty_output><jupyter_text>Ratings People are very generous when it comes to rating others' recipies as you can see from the fact that the median ratings is above 4. We remove outliers using the same method we used earlier. We also consider only those with 20 or more reviews to prevent bias with small sample size.<jupyter_code>sns.distplot(food_df['ratings'])
food_df['ratings'].describe()
ratings_stats = food_df['ratings'].describe()
ratings_thres = ratings_stats[4]-1.5*(ratings_stats[6]-ratings_stats[4])
plot2_df = food_df[['ratings','nb_ingredients','nb_reviews','recipe_name']]
plot2_df.set_index('recipe_name',inplace=True)
plot2_df = plot2_df.query(f'ratings > {ratings_thres} & nb_reviews > 20')
sns.distplot(plot2_df['ratings'])
plot2_df['ratings'].describe()<jupyter_output><empty_output><jupyter_text>Convenience-Taste Clusters We perform the clustering again so that we can choose which group of food has the best tradeoff in terms of convenience (according to number of ingredients) and deliciousness (as rated by people).<jupyter_code>scaled2 = scale(plot2_df[['ratings','nb_ingredients']])
ks = range(1, 10)
fits = [KMeans(n_clusters=k).fit(scaled2) for k in ks]
inertia = [fits[i].inertia_ for i in range(len(fits))]
plt.plot(inertia)
#optimal fit
k=3
fit = KMeans(n_clusters=k).fit(scaled2)
plot2_df['cluster'] = fit.labels_
plot2_df['cluster'] = plot2_df['cluster']
Counter(plot2_df['cluster'])
#first cluster
plot2_df[plot2_df.cluster==0].head()
#second cluster
plot2_df[plot2_df.cluster==1].head()
#third cluster
plot2_df[plot2_df.cluster==2].head()<jupyter_output><empty_output><jupyter_text>Delicious Food (by Popular Demands) Are Easier to CookWe can see from the plot that recipes with higher ratings are also easier to cook (causality unclear). This proves that taste and convenience do not necessarily have to come at expense of each other.<jupyter_code>sns.set_style(style='white')
g = sns.relplot(x='nb_ingredients', y='ratings',
             size='nb_reviews', hue = 'cluster',
             alpha=0.5, height = 6, palette='Accent',
             data=plot2_df)
g.set(xlabel='(Weighted) Number of Ingredients', ylabel='Ratings',
      title='Delicious Food (by Popular Demands) Are Easier to Cook')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09575594264267154, 'fraction_numerical': 0.01780386873255702, 'mean_word_length': 5.909574468085107, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 43, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '588557', 'license': 'Apache-2.0', 'n_tokens_mistral': 3486, 'n_tokens_neox': 3260, 'n_words': 1097, 'path': 'proposal.ipynb', 'repo_name': 'cstorm125/food', 'size': '10391'}