<jupyter_start><jupyter_text>Recommender Systems Deep-Dive LabThere are many different approaches that we can take when creating recommender systems. In the Intro to Recommender Systems lesson and lab, we put together a user similarity based recommender that first calculated the similarities between users and then leveraged a rank-based item recommender within each group of similar customers. In other words, for a given user, our recommender found the top 5 customers who were the most similar to them, aggregated and ranked the purchases of those 5 customers, and then recommended the top 5 most popular products among that group of similar users to the customer.In this lab, we are going to start out with the same data set, but we are going to dive deeper into the analysis of customers and products and look at an alternative way to generate recommendations.We will begin by importing everything we will need for this lab (libraries, data set, etc.).<jupyter_code>import pandas as pd
from scipy.spatial.distance import pdist, squareform
data = pd.read_csv('./data/customer_product_sales.csv')<jupyter_output><empty_output><jupyter_text>Data PreparationWe will then put together the foundational transformations of the data that we will need to eventually produce recommendations. The steps in this section should be familiar to you, as you would have had to tranform the data in this manner to create the user similarity based recommender in the Intro to Recommender Systems lab.First, we will create a data frame that contains the total quantity of each product purchased by each customer.<jupyter_code>customer_products = data.groupby(['CustomerID', 'ProductName']).agg({'Quantity':'sum'}).reset_index()<jupyter_output><empty_output><jupyter_text>Then, we want to create a matrix that has customers on one axis, products on the other, and the quantity purchased as the values. There will be many instances where a customer has not purchased a product, which by default will be expressed with a null value. We will want to replace those nulls with zeros by appending `.fillna(0)` to our pivot table.<jupyter_code>prod_cust_pivot = customer_products.pivot_table(values='Quantity', 
                                                columns='CustomerID', 
                                                index='ProductName', 
                                                aggfunc='sum').fillna(0)

prod_cust_pivot.head()<jupyter_output><empty_output><jupyter_text>In the pivot table we created, the rows represent the products and the columns represent the customers. Depending on what need to do with the matrix, we may instead need to transpose it so that the rows represent customers and the columns to represent products. We can do this easily by appending `.T` to our product customer matrix.<jupyter_code>cust_prod_pivot = prod_cust_pivot.T
cust_prod_pivot.head()<jupyter_output><empty_output><jupyter_text>Another thing we may want to do is normalize the values across rows or columns of the matrix so that all the values are between 0 and 1. Doing this for customers would help us identify customers that may have purcahsed a similar mix of products even though some of those customers may have purchased large quantities while other may have purchased smaller quantities. Doing this for products would help us better identify products that have been purchased by similar groups of customers regardless of the quantities purchased.We can normalize across rows for each matrix as follows.<jupyter_code>prod_cust_pivot = prod_cust_pivot.div(prod_cust_pivot.sum(axis=1), axis=0)
cust_prod_pivot = cust_prod_pivot.div(cust_prod_pivot.sum(axis=1), axis=0)<jupyter_output><empty_output><jupyter_text>User Similarity Based Recommendations The next step in creating recommendations is calculating similarities. For our user similarity based recommender, we calculated them between customers.<jupyter_code>cust_dist = pd.DataFrame(1/(1 + squareform(pdist(cust_prod_pivot, 'euclidean'))),
                         index=cust_prod_pivot.index, columns=cust_prod_pivot.index)

cust_dist.head()<jupyter_output><empty_output><jupyter_text>Once we had our similarity matrix, then we could produce recommendations for each user and package all the recommendations into a data frame.<jupyter_code>recommendations = {}
customers = list(customer_products['CustomerID'].unique())

for customer in customers:
    similar_cust = list(cust_dist[customer].sort_values(ascending=False)[1:].head().index)
    sim_cust_prod = customer_products[customer_products['CustomerID'].isin(similar_cust)]
    grouped = sim_cust_prod.groupby('ProductName').agg({'Quantity':'sum'})
    ranked_products = grouped.sort_values('Quantity', ascending=False).reset_index()
    
    merged = pd.merge(ranked_products, pd.DataFrame(cust_prod_pivot.T[customer]), on='ProductName')
    merged.columns = ['ProductName', 'Quantity', 'Purchased']
    recs = merged[merged['Purchased']==0].head()
    recommendations[customer] = list(recs['ProductName'])

user_recs = pd.DataFrame.from_dict(recommendations, orient='index').reset_index()
user_recs.columns = ['CustomerID', 'Rec1', 'Rec2', 'Rec3', 'Rec4', 'Rec5']
user_recs.head()<jupyter_output><empty_output><jupyter_text>Deeper Dive Into Our User Similarity RecommendationsLet's deconstruct what we've done and take a deeper dive into how we put this together. Doing this will equip us with the knowledge to be able to put together an item-based similarity recommender in the next section. After creating an empty dictionary to store our recommendations and getting a unique list of customer IDs to iterate through, we are first identifying the top 5 similar customers to the customer we are trying to generate recommendations for. Let's plug in customer ID 33 and see what results we get.<jupyter_code>similar_cust = list(cust_dist[33].sort_values(ascending=False)[1:].head().index)
similar_cust<jupyter_output><empty_output><jupyter_text>What we get is a list containing the 5 customer IDs of the customers whose purchase behavior is most similar to customer 33. We then go back to our customer_products data frame and select just the purchases where the customer ID is in our list of similar customers. We aggregate on product name, summing up the total quantity purchased of each product by all 5 similar customers, and then we rank them by sorting in descending order by the total quantity.<jupyter_code>sim_cust_prod = customer_products[customer_products['CustomerID'].isin(similar_cust)]
grouped = sim_cust_prod.groupby('ProductName').agg({'Quantity':'sum'})
ranked_products = grouped.sort_values('Quantity', ascending=False).reset_index()
ranked_products.head()<jupyter_output><empty_output><jupyter_text>We now have a ranked list of products that similar customers have purchased, but we haven't taken into consideration yet whether our target customer already purchases any of those items. We want to recommend them items that they might like but haven't purchased before. So we will merge the list of ranked products with our target customer's purchase list and keep only the records for items that the customer has not purchased. These will be the items that we recommend to the customer.<jupyter_code>merged = pd.merge(ranked_products, pd.DataFrame(cust_prod_pivot.T[33]), on='ProductName')
merged.columns = ['ProductName', 'Quantity', 'Purchased']
recs = merged[merged['Purchased']==0].head()
recs<jupyter_output><empty_output><jupyter_text>Item Similarity Based RecommendationsIn this section, you will create an item similarity based recommender system in a step-by-step fashion. Whereas our user similarity based recommender leveraged similarities between customers, this recommender will utilize similarities between products. You already have all the tools in your toolbox, so follow each of the steps below to complete this lab.  Step 1: Create a product distance matrix.  Step 2: Get the products purchased for a specific customer of your choice.  Step 3: For each product the customer purchased, get a list of the top 5 similar products. Package the lists into a nested list, flatten the list, and then filter out any products the customer has already purchased.  Step 4: Count the number of times each similar product occurs in your filtered list. Sort and return a list containing the top 5 items.  Step 5: Now that we have generated product recommendations for a single user, put the pieces together and iterate over a list of all CustomerIDs.- Create an empty dictionary that will hold the recommendations for all customers.- Create a list of unique CustomerIDs to iterate over.- Iterate over the customer list performing steps 2 through 4 for each and appending the results of each iteration to the dictionary you created.  Step 6: Store the results in a Pandas data frame. The data frame should a column for Customer ID and then a column for each of the 5 product recommendations for each customer.  Recommending Items to a New CustomerSuppose we get a new customer and on their first visit, they purchase the following items and quantities.<jupyter_code>new_customer = {'Cookies - Assorted':3,
                'Flavouring - Orange':3,
                'Fenngreek Seed':1,
                'Wine - White Cab Sauv.on':1,
                'Bandage - Flexible Neon':3,
                'Oil - Shortening - All - Purpose':2,
                'Beef - Montreal Smoked Brisket':4,
                'French Pastry - Mini Chocolate':4,
                'Snapple Lemon Tea':5,
                'Pepper - White, Ground':2,
                'Spinach - Baby':5,
                'Sole - Dover, Whole, Fresh':4}<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0644927536231884, 'fraction_numerical': 0.005797101449275362, 'mean_word_length': 4.699705014749263, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '201132', 'license': 'Unlicense', 'n_tokens_mistral': 2545, 'n_tokens_neox': 2408, 'n_words': 1129, 'path': 'module-2/lab-recommender-deepdive/your-code/main.ipynb', 'repo_name': 'Almuglezsalcedo/datamad0119', 'size': '9660'}