<jupyter_start><jupyter_text>Market Basket  An ingredient-based recommender systemThis is an exmample of a recommender system using some recipes data and the Apriori algorithm. These data can be obtained from Kaggle:- https://www.kaggle.com/kaggle/recipe-ingredients-datasetand are packaged with the assignment in the following directory:- `./data/train.json` To start, load the recipe data from `json` format and print the first 5 recipes.<jupyter_code>import json
from pprint import pprint

with open('./data/train.json', 'r') as train:
    recipe = json.load(train)

pprint(recipe [:4])<jupyter_output>[{'cuisine': 'greek',
  'id': 10259,
  'ingredients': ['romaine lettuce',
                  'black olives',
                  'grape tomatoes',
                  'garlic',
                  'pepper',
                  'purple onion',
                  'seasoning',
                  'garbanzo beans',
                  'feta cheese crumbles']},
 {'cuisine': 'southern_us',
  'id': 25693,
  'ingredients': ['plain flour',
                  'ground pepper',
                  'salt',
                  'tomatoes',
                  'ground black pepper',
                  'thyme',
                  'eggs',
                  'green tomatoes',
                  'yellow corn meal',
                  'milk',
                  'vegetable oil']},
 {'cuisine': 'filipino',
  'id': 20130,
  'ingredients': ['eggs',
                  'pepper',
                  'salt',
                  'mayonaise',
                  'cooking oil',
                  'green chilies',
                  'grilled chicken bre[...]<jupyter_text>Next, I wrote a function called `count_items(recipes)` that counts up the number of recipes that include each `ingredient`, storing each in the counter as a single-element tuple (for downstream convienience), i.e., incrementing like `counts[tuple([ingredient])] +=1`. When complete, exhibit this functions utility in application to the `recipes` and prints the number of 'candidates' in the output.<jupyter_code>from collections import Counter

def count_items(recipes):
    counts = Counter()
    for r in recipes: 
        for ingredient in r['ingredients']:
            counts[tuple([ingredient])] +=1
    return counts
i = count_items(recipe)
pprint(i.most_common(10))<jupyter_output>[(('salt',), 18049),
 (('onions',), 7972),
 (('olive oil',), 7972),
 (('water',), 7457),
 (('garlic',), 7380),
 (('sugar',), 6434),
 (('garlic cloves',), 6237),
 (('butter',), 4848),
 (('ground black pepper',), 4785),
 (('all-purpose flour',), 4632)]<jupyter_text>Then, I wrote a function called `store_frequent(candidates, threshold = 25)`, which accepts a `Counter` of `candidates`, i.e., item or itemset counts, and stores only those with count above the determined `threshold` value in a separate counter called `frequent`, which is `return`ed at the end of the function.<jupyter_code>def store_frequent(candidates, threshold = 25):
    frequent=[]
    for c, k in candidates.items():
        if k >= threshold:
            frequent.append([c,k])
    frequent = Counter(dict(frequent)) 
    '''coercing the list i made back into a counter as frequent works best as counter not as a dict'''
    return frequent
frequent = store_frequent(count_items(recipe),25) 

pprint(frequent.most_common(5)) # the top 5 ingredients to check that it worked 
pprint(len(frequent)) #how many ingredients are higher than 25 in count<jupyter_output>[(('salt',), 18049),
 (('onions',), 7972),
 (('olive oil',), 7972),
 (('water',), 7457),
 (('garlic',), 7380)]
1487<jupyter_text>Then I wrote a function called `get_next(recipes, frequent, threshold = 25)` that accepts the `frequent` items output from the `store_frequent()` function. With these inputs, your function should:1. create a new `Counter` called `next_candidates`2. compute the `size` of the itemsets for `next_candidates` from a single key in `frequent`3. `for` any `recipe` with _at least_ as many ingredients as `size`:    1. loop over all itemsets of size `size` (see combinations note below)    2. utilize the apriori principle and subsets of itemsets to count up potentially-frequent candidate itemsets in `next_candidates`4. `return(next_candidates)`  This is generating fewer candidates but more frequent as it increases the `size`. This follows the image below: ![lattice](images/lattice.png)As this gets larger, it has to make combinations of ingredients from the recipes and then it has to add in all other combinations to the item set before the items less than the threshold is cut off.<jupyter_code>from itertools import combinations
import random

def get_next(recipe, frequent, threshold = 25):
    next_candidate=Counter()
    nonviable = set()
    size = len(list(frequent)[0])+1
    for re in recipe:
        if len(re['ingredients']) >= size:
            items = re['ingredients']
            for candidate in combinations(items, size):
                if candidate in next_candidate:
                    next_candidate[candidate] += 1
                elif candidate not in nonviable:
                    for subitemset in combinations(candidate, size-1):
                        if subitemset not in frequent:
                            nonviable.add(candidate)
                            break
                    else:
                        next_candidate[candidate] += 1     
    return next_candidate
    
next_candidate2 = get_next(recipe, frequent, threshold=25)
print(next_candidate2.most_common(50))
next_candidate3 = get_next(recipe, next_candidate2, threshold=25)
print(next_candidate3.most_common(50))<jupyter_output>[(('pepper', 'salt', 'onions'), 397), (('garlic', 'onions', 'salt'), 394), (('water', 'salt', 'onions'), 370), (('salt', 'garlic', 'onions'), 369), (('olive oil', 'salt', 'onions'), 360), (('salt', 'onions', 'garlic'), 339), (('olive oil', 'salt', 'garlic cloves'), 293), (('salt', 'pepper', 'garlic'), 289), (('salt', 'olive oil', 'garlic'), 276), (('garlic', 'salt', 'onions'), 269), (('pepper', 'olive oil', 'salt'), 261), (('salt', 'butter', 'all-purpose flour'), 260), (('olive oil', 'garlic', 'onions'), 255), (('garlic', 'pepper', 'salt'), 253), (('olive oil', 'garlic', 'salt'), 248), (('garlic', 'olive oil', 'salt'), 245), (('olive oil', 'salt', 'pepper'), 244), (('salt', 'sugar', 'all-purpose flour'), 243), (('pepper', 'salt', 'garlic'), 243), (('salt', 'baking powder', 'all-purpose flour'), 243), (('garlic cloves', 'olive oil', 'salt'), 240), (('pepper', 'onions', 'salt'), 239), (('olive oil', 'pepper', 'salt'), 234), (('water', 'garlic', 'onions'), 230), (('olive oil', 'salt', 'ga[...]<jupyter_text>With this I have the pieces to run Apriori/collect frequent itemsets, collecting all frequent itemsets up to a particular `size`. To do this, I wrote a function called `train(recipes, size = 4)`, which:1. initializes two empty dictionaries, `candidates`, and `frequent`;2. runs the `count_items` and `store_frequent` function, storing output in the `candidates`, and `frequent` dictionaries using the integer `1` as a key;3. loops over sizes: 2, 3, .., `size` to compute and store the subsequent sizes candidates and frequent itemsets in the same structure as (2), but now utilizing the `get_next` function, instead of `count_items`; and4. `return`s the `candidates` and `frequent` itemsets.<jupyter_code>def train(recipes, size = 4):
    candidates=dict()
    frequent=dict()
    i = 1
    candidates[i] = count_items(recipes)
    frequent[i] = store_frequent(candidates[i], threshold=25)
    while i <= (size-1):
        i=i+1
        candidates[i]= get_next(recipes, candidates[i-1], threshold = 25)
        frequent[i] = store_frequent(candidates[i], threshold=25) #these return the same thing...
    return candidates, frequent
candidates, frequent=train(random.sample(recipe,8000),size=4)
candidates, frequent=train(recipe,size=4)
pprint(candidates[3].most_common(30))
pprint(frequent[3].most_common(30))<jupyter_output>[(('pepper', 'salt', 'onions'), 397),
 (('garlic', 'onions', 'salt'), 394),
 (('water', 'salt', 'onions'), 370),
 (('salt', 'garlic', 'onions'), 369),
 (('olive oil', 'salt', 'onions'), 360),
 (('salt', 'onions', 'garlic'), 339),
 (('olive oil', 'salt', 'garlic cloves'), 293),
 (('salt', 'pepper', 'garlic'), 289),
 (('salt', 'olive oil', 'garlic'), 276),
 (('garlic', 'salt', 'onions'), 269),
 (('pepper', 'olive oil', 'salt'), 261),
 (('salt', 'butter', 'all-purpose flour'), 260),
 (('olive oil', 'garlic', 'onions'), 255),
 (('garlic', 'pepper', 'salt'), 253),
 (('olive oil', 'garlic', 'salt'), 248),
 (('garlic', 'olive oil', 'salt'), 245),
 (('olive oil', 'salt', 'pepper'), 244),
 (('salt', 'sugar', 'all-purpose flour'), 243),
 (('pepper', 'salt', 'garlic'), 243),
 (('salt', 'baking powder', 'all-purpose flour'), 243),
 (('garlic cloves', 'olive oil', 'salt'), 240),
 (('pepper', 'onions', 'salt'), 239),
 (('olive oil', 'pepper', 'salt'), 234),
 (('water', 'garlic', 'onions'), 230),
 (([...]<jupyter_text>With the `frequent` itemsets up to `size`, I can utilize them to recommend missing ingredients from ingredient 'baskets' of at most `size - 1`. To do this, I wrote a function called `recommend(basket, frequent)` that does the following: 1. initializes an empty `recommendations` list2. loops over all frequent `itemset`s of `size 1 greater than the `basket`    - if there's one item left from the `itemset` when the `basket` removed, append the remaining item to the `recommendations` list in a tuple, with the number of ocurrences of the itemset in the second position4. `return` `recommendations`, but sorted from high to low by itemset ocurrence.With this code complete, I have reported the top 10 recommended items to buy for recipe flexibility in the following scenarios:- `basket = tuple(['butter', 'flour'])`- `basket = tuple(['soy sauce', 'green onions'])`- `basket = tuple(['avocado', 'garlic', 'salt'])` The function works well in pointing out additional items to add, however due to the fact that the trained data is sensitive to order in the items it comes up with the same items multiple times with different counts. This is because they used to be [(flour, butter, salt), ] and [(butter, flour, salt), ] and the result shows just salt with different numbers. However, it is performing within the specified limits of the question. The data does not have a suggestion for something that goes with avocado, garlic and salt, but appropriately suggest other goods needed for baking when supplied with butter and flour, which are hallmarks of baking recipes.<jupyter_code>import operator
def recommend(basket, frequent):
    recommendations = []
    size = len(frequent)
    if (len(basket)-1) < size:
        for i in frequent:
            if (len(basket)+1) == i:

                for l,v in frequent[i].items():
                    toup = set(basket + l)
                    
                    if len(toup) == i:

                        ple = [s for s in l if s not in basket]
                        recommendations.append(tuple([ple, v]))
    else:
        print("Cannot compute recommended next item in basket, trained itemset not large enough")
    recommendations.sort(key = operator.itemgetter(1), reverse = True)
    return recommendations
basket = tuple(['butter', 'flour'])

recommendations = recommend(basket, frequent)

print(recommendations[:9])
#there is some overlap that occurs with the items as the combinations stored in the first D5 code is sensitive to order, so (salt,butter,flour) and (butter,flour,salt) are separate items and have different counts.


basket = tuple(['soy sauce', 'green onions'])
recommendations = recommend(basket, frequent)

print(recommendations[:9])
basket = tuple(['avocado', 'garlic', 'salt'])
recommendations = recommend(basket, frequent)

print(recommendations[:9])<jupyter_output>[]
{'fraction_non_alphanumeric': 0.14712351945854485, 'fraction_numerical': 0.0266497461928934, 'mean_word_length': 3.7284, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '665447', 'license': 'Unlicense', 'n_tokens_mistral': 3928, 'n_tokens_neox': 3485, 'n_words': 1271, 'path': 'Python/Market Basket/Market Basket Example.ipynb', 'repo_name': 'jbochenek/jbochenek.github.io', 'size': '11820'}