<jupyter_start><jupyter_text>Preliminary Preprocessing Read and Process E-Commerce data In this notebook, we are going to use a subset of a publicly available [eCommerce dataset](https://www.kaggle.com/mkechinov/ecommerce-behavior-data-from-multi-category-store). The full dataset contains 7 months data (from October 2019 to April 2020) from a large multi-category online store. Each row in the file represents an event. All events are related to products and users. Each event is like many-to-many relation between products and users.Data collected by Open CDP project and the source of the dataset is [REES46 Marketing Platform](https://rees46.com/). We use only `2019-Oct.csv` file for training our models, so you can visit this site and download the csv file: https://www.kaggle.com/mkechinov/ecommerce-behavior-data-from-multi-category-store.  Import the required libraries<jupyter_code>import os
import numpy as np 
import gc
import shutil
import glob

import cudf
import nvtabular as nvt<jupyter_output><empty_output><jupyter_text>Read Data via cuDF from CSV At this point we expect that you have already downloaded the `2019-Oct.csv` dataset and stored it in the `INPUT_DATA_DIR` as defined below. It is worth mentioning that the raw dataset is ~ 6 GB, therefore a single GPU with 16 GB or less memory might run out of memory. To avoid that, you can directly start from the second notebook, `02-ETL_with_NVTabular`, using `'Oct-2019.parquet` provided in [here](https://drive.google.com/drive/folders/1GjNKerPMvEtQHt9Z37ncF1zFedDXL_RJ).<jupyter_code># define some information about where to get our data
INPUT_DATA_DIR = os.environ.get("INPUT_DATA_DIR", "/workspace/data/")
%%time
raw_df = cudf.read_csv(os.path.join(INPUT_DATA_DIR, '2019-Oct.csv')) 
raw_df.head()
raw_df.shape<jupyter_output><empty_output><jupyter_text>Convert timestamp from datetime<jupyter_code>raw_df['event_time_dt'] = raw_df['event_time'].astype('datetime64[s]')
raw_df['event_time_ts']= raw_df['event_time_dt'].astype('int')
raw_df.head()
# check out the columns with nulls
raw_df.isnull().any()
# Remove rows where `user_session` is null.
raw_df = raw_df[raw_df['user_session'].isnull()==False]
len(raw_df)<jupyter_output><empty_output><jupyter_text>We no longer need `event_time` column.<jupyter_code>raw_df = raw_df.drop(['event_time'],  axis=1)<jupyter_output><empty_output><jupyter_text>Categorify `user_session` columnAlthough `user_session` is not used as an input feature for the model, it is useful to convert those raw long string to int values to avoid potential failures when grouping interactions by `user_session` in the next notebook.<jupyter_code>cols = list(raw_df.columns)
cols.remove('user_session')
cols
# load data 
df_event = nvt.Dataset(raw_df) 

# categorify user_session 
cat_feats = ['user_session'] >> nvt.ops.Categorify()

workflow = nvt.Workflow(cols + cat_feats)
workflow.fit(df_event)
df = workflow.transform(df_event).to_ddf().compute()
df.head()
raw_df = None
del(raw_df)
gc.collect()<jupyter_output><empty_output><jupyter_text>Removing consecutive repeated (user, item) interactions We keep repeated interactions on the same items, removing only consecutive interactions, because it might be due to browser tab refreshes or different interaction types (e.g. click, add-to-card, purchase)<jupyter_code>%%time
df = df.sort_values(['user_session', 'event_time_ts']).reset_index(drop=True)

print("Count with in-session repeated interactions: {}".format(len(df)))
# Sorts the dataframe by session and timestamp, to remove consecutive repetitions
df['product_id_past'] = df['product_id'].shift(1).fillna(0)
df['session_id_past'] = df['user_session'].shift(1).fillna(0)
#Keeping only no consecutive repeated in session interactions
df = df[~((df['user_session'] == df['session_id_past']) & \
             (df['product_id'] == df['product_id_past']))]
print("Count after removed in-session repeated interactions: {}".format(len(df)))
del(df['product_id_past'])
del(df['session_id_past'])

gc.collect()<jupyter_output>Count with in-session repeated interactions: 42448762
Count after removed in-session repeated interactions: 30733301
CPU times: user 789 ms, sys: 120 ms, total: 909 ms
Wall time: 1.16 s<jupyter_text>Include the item first time seen feature (for recency calculation) We create `prod_first_event_time_ts` column which indicates the timestamp that an item was seen first time.<jupyter_code>item_first_interaction_df = df.groupby('product_id').agg({'event_time_ts': 'min'}) \
            .reset_index().rename(columns={'event_time_ts': 'prod_first_event_time_ts'})
item_first_interaction_df.head()
gc.collect()
df = df.merge(item_first_interaction_df, on=['product_id'], how='left').reset_index(drop=True)
df.head()
del(item_first_interaction_df)
item_first_interaction_df=None
gc.collect()<jupyter_output><empty_output><jupyter_text>In this tutorial, we only use one week of data from Oct 2019 dataset.<jupyter_code># check the min date
df['event_time_dt'].min()
# Filters only the first week of the data.
df = df[df['event_time_dt'] < np.datetime64('2019-10-08')].reset_index(drop=True)<jupyter_output><empty_output><jupyter_text>We verify that we only have the first week of Oct-2019 dataset.<jupyter_code>df['event_time_dt'].max()<jupyter_output><empty_output><jupyter_text>We drop `event_time_dt` column as it will not be used anymore.<jupyter_code>df = df.drop(['event_time_dt'],  axis=1)
df.head()<jupyter_output><empty_output><jupyter_text>Save the data as a single parquet file to be used in the ETL notebook.<jupyter_code># save df as parquet files on disk
df.to_parquet(os.path.join(INPUT_DATA_DIR, 'Oct-2019.parquet'))<jupyter_output><empty_output><jupyter_text>- Shut down the kernel<jupyter_code>import IPython

app = IPython.Application.instance()
app.kernel.do_shutdown(True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11122518822724162, 'fraction_numerical': 0.016598220396988363, 'mean_word_length': 6.762284196547145, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 50, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '127063', 'license': 'Apache-2.0', 'n_tokens_mistral': 1983, 'n_tokens_neox': 1893, 'n_words': 534, 'path': 'docs/source/examples/tutorial/01-preprocess.ipynb', 'repo_name': 'Jwmc999/Transformers4Rec', 'size': '5844'}