<jupyter_start><jupyter_text>Machine Learning Engineer Nanodegree Capstone Project - InstaCart Market Basket Analysis Pre-processing and feature engineeringThis notebook handles cleaning up the data and creating engineered features to tease out general features about the user and product. We will use the hdf files we created during exploration for fast read access to the raw data and save the features we create and some interim data that is computational expensive so we don't need to repeat those calculations between notebook sessions and across notebooks User-product features_raw features _ - `place_in_cart` : rank in which product was placed in cart _engineered features_ - `up_times` :  times user purchased this product - `up_reord_prob` : (conditional) given user bought product, probability that they reordered it? - `up_oprob` : probability user purchases this product in each order - `up_avg_days` : average number of days between product in order/basket - `ip_last_reord` : whether product was reordered the last time it was in a user's basket - `u_aprob` : probablity (weighted) user bought a product from this aisle in an order - `u_a_reord` : probablity that a product from this aisle was reordered - `up_cart_rank` : average rank in cart - `up_dow` : average day of week order was placed - `up_tod` : average hour (24 hr) of day the order was placed - `u_a_pcount` : average  of products purchased by user - `u_a_reord` : (conditional) probability that products from this aisle were reordered - `ua_ocount` : number of orders with products from this aisle  User features   - `uo_count` :  orders placed by user - `ubasket_avg` : average basket size for user - `top-1` - `top-10` : aisle_id of 10 most frequently ordered products aggregated by aisle Basket features_raw features_ impute NaN days_since_prior_order as mean    `outlier` distribute days_since_prior_order=30 randomly into 20 bins between 30 and 50 - `30plus` : flag to indicate this was likely an aggregated piece of information<jupyter_code>%%time
## Most of the calculation used for preprocessed are in the caps-utils.py module
%run cap-utils.py

import warnings
warnings.filterwarnings('ignore')
# Import libraries necessary for this project
#import numpy as np
#import pandas as pd
import os

_datapath = './'
os.chdir(_datapath)

# RAW_STORE = 'instacart_raw.hdf5'
# HDF_STORE = 'instacart.hdf5'
# FVARS_STORE = 'features.hdf5'
# STATS_STORE = 'stats.hdf5'
# BASKETS = 'baskets.hdf5'<jupyter_output>Loaded numpy and pandas libraries
Wall time: 1.62 s<jupyter_text>Note on computational resourcesSeveral of the preprocessing steps take significant computational resources. I ran this on a AMD Phenom II quad-processer (quite old) but with 24 GB RAM. Expanding the `priors` table or calculating the user-product features takes up more than 8GB during run time. Jupyter will 'crash' with a memory error if the machine does not have sufficient memory. During testing and developing the python code, I used the DataFrame `sample` function liberally. Augment the orders and products tables1. We will handle missing values in orders and also add a column containing the cumulative sum of days between orders. This will be used to calculate the days between when a particular product is ordered2. We will also add the depart_id and aisle_id to the priors table. We will do this to the entrire priors table<jupyter_code>%%time
d = get_from_hdf(['orders', 'priors', 'products'], RAW_STORE)
orders = d['orders']
priors = d['priors']
products = d['products']
%%time
preprocess_orders(orders)
print orders.head()
print orders.columns
%%time
priors = add_product_groups(priors, products)
print priors.columns<jupyter_output>Index([u'order_id', u'product_id', u'add_to_cart_order', u'reordered',
       u'department_id', u'aisle_id'],
      dtype='object')
Wall time: 42.4 s<jupyter_text>We need to expand the products table with `user_id` and calculate the days elapsed between orders for a specific user and productThis calculation turned out to be very costly using a lambda expression in a DataFrame `aggregate` function. I implemented this using a python loop manipulating the data as a numpy matrix. Even with the speedup from the vectorized logic, calculating `ds_last` was one of the three slowest steps in the pre-processing stage.<jupyter_code>%%time
# This code block takes a long time (and a lot of memory). For testing, we can take samples instead
# option 1 - get_hist_chunk(orders, priors, all=True) - the whole dataset 
#               (90 s. Check the next step though)
# option 2 - get_hist_chunk(orders, priors, frac=0.001) - randomly selects 0.1% of the users (45 s)
# 
ps=get_hist_chunk(orders, priors, all=True)
print "{} product orders".format(len(ps))
print ps.columns
%%time
# The routine does this sort - showing here for reference
#      ps.sort_values(by=['user_id', 'product_id', 'order_number'])
g = user_product_ds_last(ps)
#print ps[['user_id','product_id','order_number']].head(10)
print len(g.columns), g.columns
%%time
ps = ps.merge(g, on=['user_id', 'product_id', 'order_number'])
del g
print len(ps), len(ps.columns), ps.columns<jupyter_output>32434489 16 Index([u'order_id', u'user_id', u'eval_set', u'order_number', u'order_dow',
       u'order_hour_of_day', u'days_since_prior_order', u'30plus_days',
       u'csum_ds', u'log_ds', u'product_id', u'add_to_cart_order',
       u'reordered', u'department_id', u'aisle_id', u'ds_last'],
      dtype='object')
Wall time: 1min 59s<jupyter_text>We calculated `ds_last` for the entire `orders_products` (aka `priors`) table _ONCE_ and saved it in hdf format.<jupyter_code>ps.to_hdf(HDF_STORE,'aug-priors')
#ps=pd.read_hdf(HDF_STORE, 'aug-priors')<jupyter_output><empty_output><jupyter_text>Calculate user-product features from order history We calculate the features from the historical data, i.e., exclude order from the training and test sets.This step is computational expensive.<jupyter_code>%%time
#phist = ps[ps.eval_set == 0]
up_feat = user_product_features(phist)
up_feat['aisle_id'] = up_feat['aisle_id'].astype('int16')
print up_feat.info()
up_feat['uo_count']=up_feat['uo_count'].astype('int16')
up_feat['ubasket_avg']=up_feat['ubasket_avg'].astype('float16')
up_feat['up_dow']=up_feat['up_dow'].astype('float16')
up_feat['up_tod']=up_feat['up_tod'].astype('float16')
up_feat['ua_ocount']=up_feat['ua_ocount'].astype('int16')
up_feat['up_30_avg']=up_feat['up_30_avg'].astype('float16')
up_feat.to_hdf(FVARS_STORE, "up_feat")
try:
    del up_feat
except:
    pass
gc.collect()
try:
    del products, priors
except:
    pass<jupyter_output><empty_output><jupyter_text>Calculate frequently purchased products (grouped in aisles)  from order history We calculate the number of products from each aisle in each user's purchase history. We then identified 10 aisles with the most number of products for each user and placed the aisle id in 10 columns `top-1, top-2, ... top-10`, in rank order.This step is computational expensive.<jupyter_code>%%time
ps.sort_values(by=["user_id", 'order_id'], inplace=True)
ga=build_user_topn_aisles(ps, 10)
print ga.info()
ga.to_hdf(FVARS_STORE, "ua_top10")
try:
    del ps, phist, ga, b_feat
except:
    pass
gc.collect()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09423289183222958, 'fraction_numerical': 0.01283112582781457, 'mean_word_length': 5.5483288166215, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 22, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '292831', 'license': 'FSFAP', 'n_tokens_mistral': 2345, 'n_tokens_neox': 2232, 'n_words': 820, 'path': 'Instacart/Instacart - Preprocessing.ipynb', 'repo_name': 'zkhundkar/VbtmcPORkTqFuOmcLiwpgkmdIOqZQT', 'size': '7248'}