<jupyter_start><jupyter_text>Predicting a customer's next purchase using automated feature engineering    **As customers use your product, they leave behind a trail of behaviors that indicate how they will act in the future. Through automated feature engineering we can identify the predictive patterns in granular customer behavioral data that can be used to improve the customer's experience and generate additional revenue for your business.**In this tutorial, we show how [Featuretools](https://github.com/FeatureLabs/featuretools) can be used to perform feature engineering on a multi-table dataset of 3 million online grocery orders provided by Instacart. We will generate a feature matrix that can be used to train a machine learning model to predict what product a customer buys next.*Note: This notebook requires a dataset from Instacart. You can download the dataset [here](https://www.kaggle.com/c/instacart-market-basket-analysis/data). Once you have downloaded the data, be sure to place the CSV files contained in the archive in a directory called `data`. If you use a different directory name, you will need to update the code below to point to the proper location.* Highlights* We automatically generate features using Deep Feature Synthesis and select the 20 most important features for predictive modeling* We demonstrate how to generate features in a scalable manner using [Dask](http://dask.pydata.org/en/latest/)* We automatically generate label times using [Compose](https://github.com/FeatureLabs/compose) which can be reused for numerous prediction problems* We develop a model for predicting what a customer will buy next, starting with a sample of data and then scaling to the full dataset  You must have Featuretools version 0.16.0 or greater installed to run this notebook<jupyter_code>import os
import composeml as cp
import featuretools as ft
import dask.dataframe as dd
import numpy as np
import pandas as pd
import utils
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score

from dask.distributed import Client
ft.__version__<jupyter_output><empty_output><jupyter_text>Step 1. Load and preprocess dataFirst, we will create a Dask distributed client so we can track the progress of our computation on the Dask dashboard that is created when the client is initialized.<jupyter_code>client = Client(n_workers=2)
client<jupyter_output><empty_output><jupyter_text>Next, we will specify our input and output directories and set the blocksize we will be using to read the raw CSV files into Dask dataframes. When running on a machine with 16GB of memory available for two Dask workers, a `100MB` blocksize has worked well. This number may need to be adjusted based on your specific environment. Refer to the [Dask documentation](https://docs.dask.org/en/latest/best-practices.html) for additional info.<jupyter_code>data_dir = os.path.join("data")
output_dir = os.path.join("data", "dask_data")
blocksize = "100MB"<jupyter_output><empty_output><jupyter_text>Now we will read our data into Dask dataframes. This operation will complete quite fast as we are not actually bringing the data into memory at this stage.<jupyter_code>order_products = dd.concat([dd.read_csv(os.path.join(data_dir, "order_products__prior.csv"), blocksize=blocksize),
                            dd.read_csv(os.path.join(data_dir, "order_products__train.csv"), blocksize=blocksize)])
orders = dd.read_csv(os.path.join(data_dir, "orders.csv"), blocksize=blocksize)
departments = dd.read_csv(os.path.join(data_dir, "departments.csv"), blocksize=blocksize)
products = dd.read_csv(os.path.join(data_dir, "products.csv"), blocksize=blocksize)<jupyter_output><empty_output><jupyter_text>In the next few cells, we will perform some required preprocessing to clean up our data. We will merge together some of the raw dataframes and add absolute order time information from the relative times used in the raw data. This will allow us to use cutoff times as part of the Deep Feature Synthesis process.<jupyter_code>order_products = order_products.merge(products).merge(departments)
def add_time(df):
    df.reset_index(drop=True)
    df["order_time"] = np.nan
    days_since = df.columns.tolist().index("days_since_prior_order")
    hour_of_day = df.columns.tolist().index("order_hour_of_day")
    order_time = df.columns.tolist().index("order_time")

    df.iloc[0, order_time] = pd.Timestamp('Jan 1, 2015') +  pd.Timedelta(df.iloc[0, hour_of_day], "h")
    for i in range(1, df.shape[0]):
        df.iloc[i, order_time] = df.iloc[i - 1, order_time] \
            + pd.Timedelta(df.iloc[i, days_since], "d") \
                                    + pd.Timedelta(df.iloc[i, hour_of_day], "h")

    to_drop = ["order_number", "order_dow", "order_hour_of_day", "days_since_prior_order", "eval_set"]
    df.drop(to_drop, axis=1, inplace=True)

    return df
orders = orders.groupby("user_id").apply(add_time, meta={
    "order_id": int,
    "user_id": int,
    "order_time": np.datetime64(0, 'ns').dtype})
order_products = order_products.merge(orders[["order_id", "order_time"]])
order_products["order_product_id"] = order_products["order_id"] * 1000 + order_products["add_to_cart_order"]
order_products = order_products.drop(["product_id", "department_id", "add_to_cart_order"], axis=1)<jupyter_output>c:\users\ajkramer\appdata\local\continuum\anaconda3\envs\best_purchase\lib\site-packages\dask\dataframe\multi.py:445: UserWarning: Merging dataframes with merge column data type mismatches: 
+--------------------------+------------+-------------+
| Merge columns            | left dtype | right dtype |
+--------------------------+------------+-------------+
| ('order_id', 'order_id') | int64      | int32       |
+--------------------------+------------+-------------+
Cast dtypes explicitly to avoid unexpected results.
  warnings.warn(<jupyter_text>Now that the preprocessing work is complete, we will save the results to disk. This will allow us to start from this point in the process in the future, without having to repeat all of the preprocessing steps. If you have already saved the results to disk previously, you can skip the cell below. Note: The process of saving to CSV is computationally intensive and may take 45 minutes or more, depending on the system you are using. You can use the Dask dashboard to monitor the progress.<jupyter_code># Save preprocessed data to disk
orders.to_csv(os.path.join(output_dir, "orders-*.csv"), index=False)
order_products.to_csv(os.path.join(output_dir, "order_products-*.csv"), index=False)<jupyter_output><empty_output><jupyter_text>If you have already performed the preprocessing steps and saved the processed files to disk, you can read them in with the commands in the following cell.<jupyter_code># Read preprocessed data from disk
orders = dd.read_csv(os.path.join(output_dir, "orders-*.csv"), blocksize=blocksize)
order_products = dd.read_csv(os.path.join(output_dir, "order_products-*.csv"), blocksize=blocksize)
orders.head()
order_products.head()<jupyter_output><empty_output><jupyter_text>Step 2: Create a Featuretools entitysetWhen using Dask dataframes to create an entityset, variable type inference is not performed as it is with entitysets created from pandas dataframes. As a result, users must specify the Featuretools variable types for all of the columns in the dataframes that make up the entityset when using Dask. In the following cell we define the data types for the `order_products` and `orders` entities.<jupyter_code>order_products_vtypes = {
    "order_id": ft.variable_types.Id,
    "reordered": ft.variable_types.Boolean,
    "product_name": ft.variable_types.Categorical,
    "aisle_id": ft.variable_types.Categorical,
    "department": ft.variable_types.Categorical,
    "order_time": ft.variable_types.Datetime,
    "order_product_id": ft.variable_types.Index,
}

order_vtypes = {
    "order_id": ft.variable_types.Index,
    "user_id": ft.variable_types.Id,
    "order_time": ft.variable_types.DatetimeTimeIndex,
}<jupyter_output><empty_output><jupyter_text>Now that we have defined the data types, we can create the entityset and establish the relationship between the two entities. For our initial run we will use a sample of the full data to determine what features are the best predictors. Once we have the feature importances established we will rerun on the full dataset using only the most important features.First we will sample our data - grabbing the orders for 1000 different customers. Because we cannot pass a Dask series to `.isin()` we must call `.compute()` on the ids to convert this into a pandas series.<jupyter_code>orders_sample = orders[orders['user_id'].isin(range(1001,2001))]<jupyter_output><empty_output><jupyter_text>Next we will get the order products associated with the orders we sampled.<jupyter_code>order_products_sample = order_products[order_products['order_id'].isin(orders_sample['order_id'].compute())]<jupyter_output><empty_output><jupyter_text>Now that we have sampled our data, we can create an entityset from these sampled dataframes.<jupyter_code>es = ft.EntitySet("instacart_sample")
es.entity_from_dataframe(entity_id="order_products",
                         dataframe=order_products_sample,
                         index="order_product_id",
                         variable_types=order_products_vtypes,
                         time_index="order_time")

es.entity_from_dataframe(entity_id="orders",
                         dataframe=orders_sample,
                         index="order_id",
                         variable_types=order_vtypes,
                         time_index="order_time")

es.add_relationship(ft.Relationship(es["orders"]["order_id"], es["order_products"]["order_id"]))<jupyter_output><empty_output><jupyter_text>Next, we will normalize the `orders` entity to create a new `users` entity that we will later use as the target entity during the deep feature synthesis process.<jupyter_code>es.normalize_entity(base_entity_id="orders", new_entity_id="users", index="user_id")
es.plot()<jupyter_output>c:\users\ajkramer\appdata\local\continuum\anaconda3\envs\best_purchase\lib\site-packages\featuretools\entityset\entityset.py:248: UserWarning: Not adding duplicate relationship: <Relationship: orders.user_id -> users.user_id>
  warnings.warn(<jupyter_text>To finish up creation of the entity set we will add last time indexes and set some interesting values.<jupyter_code>es.add_last_time_indexes()
es["order_products"]["department"].interesting_values = ['produce', 'dairy eggs', 'snacks', 'beverages', 'frozen', 'pantry', 'bakery', 'canned goods', 'deli', 'dry goods pasta']
es["order_products"]["product_name"].interesting_values = ['Banana', 'Bag of Organic Bananas', 'Organic Baby Spinach', 'Organic Strawberries', 'Organic Hass Avocado', 'Organic Avocado', 'Large Lemon', 'Limes', 'Strawberries', 'Organic Whole Milk']<jupyter_output><empty_output><jupyter_text>Step 3. Use Compose to generate our cutoff times dataframeIn the cells that follow we will demonstrate how [Compose](https://github.com/FeatureLabs/compose) can be used to generate the label times dataframe that will be used as cutoff times for deep feature synthesis.First we define a labeling function to add a label if a user has bought a specific product or not, and then we will create our `LabelMaker` using this function.<jupyter_code>def bought_product(df, product_name):
    purchased = df.product_name.str.contains(product_name).any()
    return purchased
lm = cp.LabelMaker(
    target_entity='user_id',
    time_index='order_time',
    labeling_function=bought_product,
    window_size='4w',
)
def denormalize(es):
    df = es['order_products'].df.merge(es['orders'].df).merge(es['users'].df)
    return df<jupyter_output><empty_output><jupyter_text>Compose does not currently work on Dask dataframes, so we must first run `.compute()` on the denormalized entityset to switch to pandas.<jupyter_code>df = denormalize(es).compute()<jupyter_output><empty_output><jupyter_text>Now we can create our labels, indicating whether or not a user has purchased Bananas.<jupyter_code>label_times = lm.search(
    df.sort_values('order_time'),
    minimum_data='2015-03-15',
    num_examples_per_instance=2,
    product_name='Banana',
    verbose=True,
)
label_times.head()<jupyter_output><empty_output><jupyter_text>We can see above the our training examples contain three pieces of information: a user id, the last time we can use data before feature engineering (called the "cutoff time"), and the label to predict. These are called our "label times". We can use `describe` to print out the distribution with the settings and transforms that were used to make these labels. This is useful as a reference for understanding how the labels were generated from raw data. Also, the label distribution is helpful for determining if we have imbalanced labels.<jupyter_code>label_times.describe()<jupyter_output>Label Distribution
------------------
False      951
True       499
Total:    1450


Settings
--------
gap                                    None
minimum_data                     2015-03-15
num_examples_per_instance                 2
target_column                bought_product
target_entity                       user_id
target_type                        discrete
window_size                              4w


Transforms
----------
No transforms applied<jupyter_text>Step 4. Run Deep Feature SynthesisWith our label times created, we are ready to run deep feature synthesis to generate our feature matrix. This will execute quickly and the resulting feature matrix will be returned as a Dask dataframe. This process does not cause the feature matrix to be computed or brought into memory.When we use DFS, we specify- `target_entity` - the table to build features for - `users` in this case- `cutoff_time` - the point in time to calculate the featuresA good way to think of the `cutoff_time` is that it let's us "pretend" we are at an earlier point in time when generating our features so we can simulate making predictions. We get this time for each customer from the label times we generated above.For this initial run we will not specify any primitives, which will result in all the default primitives being used to create features.<jupyter_code>feature_matrix, features = ft.dfs(target_entity="users",
                                  cutoff_time=label_times,
                                  entityset=es,
                                  verbose=True)<jupyter_output>Built 59 features
Elapsed: 00:14 | Progress: 100%|███████████████████████████████████████████████████████████████████████████████████████<jupyter_text>Next, let's compute the feature matrix and take a look at what we created.<jupyter_code>fm = feature_matrix.compute()
fm.head()<jupyter_output><empty_output><jupyter_text>Before we use this feature matrix to build a predictive model, we will first encode any categorical features using `ft.encode_features()`. Note, at this time `ft.encode_features()` does not work with a Dask feature matrix, so we will use the pandas version we computed above.<jupyter_code>fm_encoded, features_encoded = ft.encode_features(fm,
                                                  features)

print("Number of features %s" % len(features_encoded))
fm_encoded.head()<jupyter_output>Number of features 63<jupyter_text>Step 5. Machine LearningUsing the default parameters, we generated dozens of potential features for our prediction problem. With a few simple commands, this feature matrix can be used for machine learning<jupyter_code>X = fm_encoded.drop(["user_id"], axis=1)
X = X.fillna(0)
y = X.pop("bought_product").astype('bool')
X.head()
y.head()<jupyter_output><empty_output><jupyter_text>Let's train a Random Forest and validate using 3-fold cross validation<jupyter_code>clf = RandomForestClassifier(n_estimators=400, n_jobs=-1)
scores = cross_val_score(estimator=clf,X=X, y=y, cv=3,
                         scoring="roc_auc", verbose=True)

"AUC %.2f +/- %.2f" % (scores.mean(), scores.std())<jupyter_output>[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.
[Parallel(n_jobs=1)]: Done   3 out of   3 | elapsed:   11.6s finished<jupyter_text>As you can see this model predicts the next purchase better than guessing. Next we will identify the top 20 features so we can use them to later perform machine learning on the whole dataset.<jupyter_code>clf.fit(X, y)
top_features = utils.feature_importances(clf, features_encoded, n=20)<jupyter_output>1: Feature: COUNT(order_products WHERE product_name = Banana), 0.077
2: Feature: COUNT(order_products WHERE product_name = Bag of Organic Bananas), 0.066
3: Feature: COUNT(order_products WHERE department = produce), 0.044
4: Feature: MEAN(orders.NUM_UNIQUE(order_products.product_name)), 0.030
5: Feature: MEAN(orders.COUNT(order_products)), 0.028
6: Feature: COUNT(order_products WHERE department = dairy eggs), 0.028
7: Feature: STD(orders.PERCENT_TRUE(order_products.reordered)), 0.027
8: Feature: COUNT(order_products), 0.026
9: Feature: STD(orders.NUM_UNIQUE(order_products.department)), 0.025
10: Feature: STD(orders.COUNT(order_products)), 0.025
11: Feature: SUM(orders.NUM_UNIQUE(order_products.product_name)), 0.024
12: Feature: MEAN(orders.NUM_UNIQUE(order_products.aisle_id)), 0.024
13: Feature: NUM_UNIQUE(order_products.product_name), 0.024
14: Feature: STD(orders.NUM_UNIQUE(order_products.product_name)), 0.024
15: Feature: PERCENT_TRUE(order_products.reordered), 0.024
16: Feature: ME[...]<jupyter_text>To persist these features, we can save them to disk.<jupyter_code>ft.save_features(top_features, os.path.join(data_dir, "top_features.txt"))<jupyter_output><empty_output><jupyter_text>Auto ModelingInstead of just training a Random Forest, we can use [EvalML](https://evalml.alteryx.com/en/stable/) from Alteryx to build multiple models, compare them, and identify the best one<jupyter_code>import evalml
from evalml import AutoMLSearch
automl = AutoMLSearch(problem_type="binary", objective="auc")
automl.search(X, y, data_checks='disabled')<jupyter_output>Using default limit of max_batches=1.

`X` passed was not a DataTable. EvalML will try to convert the input as a Woodwork DataTable and types will be inferred. To control this behavior, please pass in a Woodwork DataTable instead.
`y` passed was not a DataColumn. EvalML will try to convert the input as a Woodwork DataTable and types will be inferred. To control this behavior, please pass in a Woodwork DataTable instead.
Generating pipelines to search over...
*****************************
* Beginning pipeline search *
*****************************

Optimizing for AUC. 
Greater score is better.

Searching up to 1 batches for a total of 9 pipelines. 
Allowed model families: extra_trees, linear_model, lightgbm, random_forest, xgboost, catboost, decision_tree<jupyter_text>Notice how with EvalML, we are able to get an AUC of .82, higher than the average of .8 for the Python Models. We can view the main features below<jupyter_code>pipeline = automl.best_pipeline
pipeline.fit(X, y)

pipeline.feature_importance[0:10]<jupyter_output><empty_output><jupyter_text>Understanding feature engineering in FeaturetoolsBefore moving forward, take a look at the features we created. You will see that they are more than just simple transformations of columns in our raw data. Instead, they perform aggregations (and sometimes stacking of aggregations) across the relationships in our dataset. If you're curious how this works, learn about the Deep Feature Synthesis algorithm in our documentation [here](https://featuretools.alteryx.com/en/stable/).DFS is so powerful because with no manual work, the library figured out that historical purchases of bananas are important for predicting future purchases. Additionally, it surfaces that purchasing dairy or eggs and reordering behavior are important features.Even though these features are intuitive, Deep Feature Synthesis will automatically adapt as we change the prediction problem, saving us the time of manually brainstorming and implementing these data transformation.  Step 6. Scale to Full DatasetNow that we have established the most important features, we will repeat the process of creating a feature matrix, using only these features, and then make predictions using our full dataset.To start, we will create a new entityset containing our full dataset.<jupyter_code>es = ft.EntitySet("instacart_full")
es.entity_from_dataframe(entity_id="order_products",
                         dataframe=order_products,
                         index="order_product_id",
                         variable_types=order_products_vtypes,
                         time_index="order_time")

es.entity_from_dataframe(entity_id="orders",
                         dataframe=orders,
                         index="order_id",
                         variable_types=order_vtypes,
                         time_index="order_time")

es.add_relationship(ft.Relationship(es["orders"]["order_id"], es["order_products"]["order_id"]))
es.normalize_entity(base_entity_id="orders", new_entity_id="users", index="user_id")
es.add_last_time_indexes()
es["order_products"]["department"].interesting_values = ['produce', 'dairy eggs', 'snacks', 'beverages', 'frozen', 'pantry', 'bakery', 'canned goods', 'deli', 'dry goods pasta']
es["order_products"]["product_name"].interesting_values = ['Banana', 'Bag of Organic Bananas', 'Organic Baby Spinach', 'Organic Strawberries', 'Organic Hass Avocado', 'Organic Avocado', 'Large Lemon', 'Limes', 'Strawberries', 'Organic Whole Milk']<jupyter_output><empty_output><jupyter_text>We will use our previously defined Compose label maker to create cutoff times for the full dataset.<jupyter_code>df = denormalize(es).compute()
label_times = lm.search(
    df.sort_values('order_time'),
    minimum_data='2015-03-15',
    num_examples_per_instance=2,
    product_name='Banana',
    verbose=True,
)
label_times.head()<jupyter_output><empty_output><jupyter_text>Next we will read in the top 20 features we identified previously and calculate a feature matrix using only these features.<jupyter_code>top_features= ft.load_features(os.path.join(data_dir, "top_features.txt"))
top_features<jupyter_output><empty_output><jupyter_text>Having read in the top features we want to use, we can now create our feature matrix on the full dataset with a call to `ft.calculate_feature_matrix()`.<jupyter_code>fm = ft.calculate_feature_matrix(top_features, entityset=es, cutoff_time=label_times, verbose=True)<jupyter_output>Elapsed: 00:03 | Progress: 100%|███████████████████████████████████████████████████████████████████████████████████████<jupyter_text>Next, we will compute our feature matrix to bring the results into memory, allowing us to encode categorical features and make our predictions.<jupyter_code>fm = fm.compute()
fm_encoded, features_encoded = ft.encode_features(fm, top_features)

print("Number of features %s" % len(features_encoded))
fm_encoded.head()
X = fm_encoded.drop(["user_id"], axis=1)
X = X.fillna(0)
y = X.pop("bought_product").astype('bool')
clf = RandomForestClassifier(n_estimators=400, n_jobs=-1)
scores = cross_val_score(estimator=clf,X=X, y=y, cv=3,
                         scoring="roc_auc", verbose=True)

"AUC %.2f +/- %.2f" % (scores.mean(), scores.std())
clf.fit(X, y)
top_features = utils.feature_importances(clf, top_features, n=20)<jupyter_output><empty_output><jupyter_text>Now that we have finished, we can close our Dask client.<jupyter_code>client.close()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10502031832035219, 'fraction_numerical': 0.010624788350829665, 'mean_word_length': 4.757981964416281, 'pattern_counts': {'":': 13, '<': 121, '<?xml version=': 0, '>': 122, 'https://': 7, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '32', 'dir': 'jupyter-structured-clean-dedup', 'id': '405637', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 7137, 'n_tokens_neox': 6689, 'n_words': 2281, 'path': 'archived-notebooks/predict-next-purchase/Tutorial.ipynb', 'repo_name': 'Featuretools/predict_next_purchase', 'size': '23624'}