<jupyter_start><jupyter_text>Problem StatementThe TaskGiven a sequence of click events performed by some user during a typical session in an e-commerce website, the goal is to predict whether the user is going to buy something or not, and if he is buying, what would be the items he is going to buy. The task could therefore be divided into two sub goals:Is the user going to buy items in this session? Yes|NoIf yes, what are the items that are going to be bought?The DataDownload the DataTraining Data FilesThe training data comprises two different files:yoochoose-clicks.dat - Click events. Each record/line in the file has the following fields:- Session ID – the id of the session. In one session there are one or many clicks.- Timestamp – the time when the click occurred.- Item ID – the unique identifier of the item.- Category – the category of the item.yoochoose-buys.dat - Buy events. Each record/line in the file has the following fields:- Session ID - the id of the session. In one session there are one or many buying events.- Timestamp - the time when the buy occurred.- Item ID – the unique identifier of item.- Price – the price of the item.- Quantity – how many of this item were bought.The Session ID in yoochoose-buys.dat will always exist in the yoochoose-clicks.dat file – the records with the same Session ID together form the sequence of click events of a certain user during the session. The session could be short (few minutes) or very long (few hours), it could have one click or hundreds of clicks. All depends on the activity of the user.Test FileThe Test data is one file:yoochoose-test.dat - identically structured as the yoochoose-clicks.dat of the training dataSession IDTimestampItem IDCategorySolution fileThe task is to predict for each session in the test file, whether there is going to be a buying event in this session, and if there is, what are the items that will be bought. No need to predict quantities.The solution file, that has to be submitted, comprises records that have exactly two fields:solution.datSession IDComma separated list of Item IDs that have been bought in this sessionAll the Session IDs that exist in the solution file are coming from the test file. In the solution file there will be all the Session IDs that the challenger predicts to be ending with at least one buying event. The second field in each record/line will comprise a list of one or many Item IDs separated by “,” character. The field delimiter between Session ID and the list of Item IDs is “;”. No spaces are required in the file. An example of a solution file could be found here.If a Session ID exists in the test file but does not exist in the solution file, it means that this is a session that the challenger predicts not to end up with a buying event. Since approximately 95% of the sessions end without a buying event, the challenge is hence twofold, first - recognize the exact sessions that end with a buying event, and second - what are the items that will be bought in these sessions.Read the full problem statement and evaluation criteria here: https://2015.recsyschallenge.com/challenge.html<jupyter_code>import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)
from sklearn.model_selection import train_test_split
from sklearn import metrics
from pandas import DataFrame

# pip3 install --no-deps fastai
# Any results you write to the current directory are saved as output.
PATH = '/Users/abasar/data/recsys-challenge-2015' 
import os
for dirname, _, filenames in os.walk(PATH):
    for filename in filenames:
        print(os.path.join(dirname, filename))
# https://github.com/fastai/fastai/blob/master/fastai/tabular/transform.py#L43
import re
import calendar
from functools import partial


def add_cyclic_datepart(df:DataFrame, field_name:str, prefix:str=None, drop:bool=True, time:bool=False, add_linear:bool=False):
    "Helper function that adds trigonometric date/time features to a date in the column `field_name` of `df`."
    make_date(df, field_name)
    field = df[field_name]
    prefix = ifnone(prefix, re.sub('[Dd]ate$', '', field_name))
    series = field.apply(partial(cyclic_dt_features, time=time, add_linear=add_linear))
    columns = [prefix + c for c in cyclic_dt_feat_names(time, add_linear)]
    df_feats = pd.DataFrame([item for item in series], columns=columns, index=series.index)
    for column in columns: df[column] = df_feats[column]
    if drop: df.drop(field_name, axis=1, inplace=True)
    return df

def add_datepart(df:DataFrame, field_name:str, prefix:str=None, drop:bool=True, time:bool=False):
    "Helper function that adds columns relevant to a date in the column `field_name` of `df`."
    make_date(df, field_name)
    field = df[field_name]
    prefix = ifnone(prefix, re.sub('[Dd]ate$', '', field_name))
    attr = ['Year', 'Month', 'Week', 'Day', 'Dayofweek', 'Dayofyear', 'Is_month_end', 'Is_month_start', 
            'Is_quarter_end', 'Is_quarter_start', 'Is_year_end', 'Is_year_start']
    if time: attr = attr + ['Hour', 'Minute', 'Second']
    for n in attr: df[prefix + n] = getattr(field.dt, n.lower())
    df[prefix + 'Elapsed'] = field.astype(np.int64) // 10 ** 9
    if drop: df.drop(field_name, axis=1, inplace=True)
    return df

def make_date(df:DataFrame, date_field:str):
    "Make sure `df[field_name]` is of the right date type."
    field_dtype = df[date_field].dtype
    if isinstance(field_dtype, pd.core.dtypes.dtypes.DatetimeTZDtype):
        field_dtype = np.datetime64
    if not np.issubdtype(field_dtype, np.datetime64):
        df[date_field] = pd.to_datetime(df[date_field], infer_datetime_format=True)
        
        
def ifnone(a,b):
    "`a` if `a` is not None, otherwise `b`."
    return b if a is None else a



def cyclic_dt_features(d, time:bool=True, add_linear:bool=False):
    "Calculate the cos and sin of date/time cycles."
    tt,fs = d.timetuple(), [np.cos, np.sin]
    day_year,days_month = tt.tm_yday, calendar.monthrange(d.year, d.month)[1]
    days_year = 366 if calendar.isleap(d.year) else 365
    rs = d.weekday()/7, (d.day-1)/days_month, (d.month-1)/12, (day_year-1)/days_year
    feats = [f(r * 2 * np.pi) for r in rs for f in fs]
    if time and isinstance(d, datetime) and type(d) != date:
        rs = tt.tm_hour/24, tt.tm_hour%12/12, tt.tm_min/60, tt.tm_sec/60
        feats += [f(r * 2 * np.pi) for r in rs for f in fs]
    if add_linear:
        if type(d) == date: feats.append(d.year + rs[-1])
        else:
            secs_in_year = (datetime(d.year+1, 1, 1) - datetime(d.year, 1, 1)).total_seconds()
            feats.append(d.year + ((d - datetime(d.year, 1, 1)).total_seconds() / secs_in_year))
    return feats

def cyclic_dt_feat_names(time:bool=True, add_linear:bool=False):
    "Return feature names of date/time cycles as produced by `cyclic_dt_features`."
    fs = ['cos','sin']
    attr = [f'{r}_{f}' for r in 'weekday day_month month_year day_year'.split() for f in fs]
    if time: attr += [f'{r}_{f}' for r in 'hour clock min sec'.split() for f in fs]
    if add_linear: attr.append('year_lin')
    return attr


limit = 100000
def read_buys(limit=None):
    print("Loading buys")
    buys = pd.read_csv(f"{PATH}/yoochoose-buys.dat",
                    names=["session", "timestamp", "item", "price", "qty"],
                    parse_dates=["timestamp"], 
                    nrows = limit)
    buys = buys.sort_values(by=["timestamp", "session"])
    if limit:
        buys = buys.iloc[:limit]
    return buys

buys = read_buys(limit)
buys
def read_clicks(limit=None):
    print("Loading clicks")
    filename = f"{PATH}/yoochoose-clicks.dat"
    df = pd.read_csv(filename,
                     names=["session", "timestamp", "item", "category"],
                     parse_dates=["timestamp"],
                     converters={"category": lambda c: -1 if c == "S" else c}, 
                     nrows = limit)
    df = df.sort_values(by=["timestamp", "session"])
    if limit:
        df = df.iloc[:limit]
    print("Clicks shape %s %s" % df.shape)
    return df

clicks = read_clicks(limit)
clicks
def process_clicks(clicks, rolling_days=15):
    # Compute dwell time for each click
    print("Processing clicks")
    clicks['prev_ts'] = clicks.groupby('session')['timestamp'].transform(lambda x: x.shift())
    clicks['diff_prev'] = clicks["timestamp"] - clicks["prev_ts"] # in minutes
    clicks["dwell"] = clicks.groupby('session')['diff_prev'].transform(lambda x: x.shift(-1)).dt.seconds/60
    clicks = clicks.sort_values(by=["session", "timestamp"])
    print("Processed clicks shape %s %s" % clicks.shape)
    return clicks

clicks_proc = process_clicks(clicks)
clicks_proc
def process_buys(limit=None):
    # Group into sessions, compute nr of items bought and set label column
    buys = read_buys(limit=limit)
    print("Processing buys")
    print("Buys from %s to %s" % (buys.timestamp.min(), buys.timestamp.max()))
    grouped = buys.groupby("session")
    buys_g = pd.DataFrame(index=grouped.groups.keys())
    buys_g["items_bought"] = grouped.item.count() # quantity may be zero which is weird so dont use it
    buys_g["is_buy"] = 1 # for easier merge later on
    buys_g.index.name = "session"
    print("Buys grouped by session %s %s" % buys_g.shape)
    return buys_g

buys_proc = process_buys(limit)
buys_proc
def get_items_cats_percent(clicks, limit=None):
    buys = read_buys(limit=limit)
    # percent bought
    item_id_bought_pct = buys.item.value_counts(normalize=True)
    cat_id_viewed_pct = clicks.category.value_counts(normalize=True)
    item_id_viewed_pct = clicks.item.value_counts(normalize=True)

    return dict(views=dict(item=item_id_viewed_pct, cat=cat_id_viewed_pct), buys=item_id_bought_pct)

items_cats_percent = get_items_cats_percent(clicks, limit)
items_cats_percent
def process_sessions(processed_clicks, limit=None):
    print("Preprocessing - Grouping clicks into sessions")
    clicks = processed_clicks
    
    # Group clicks by session
    grouped = clicks.groupby("session")
    sessions = pd.DataFrame(index=grouped.groups.keys())
    
    # Session counters
    sessions["total_clicks"] = grouped.item.count()
    sessions["total_items"] = grouped.item.unique().apply(lambda x: len(x))
    sessions["total_cats"] = grouped.category.unique().apply(lambda x: len(x))
    print("Computed counters")
    
    # Session duration
    sessions["max_dwell"] = grouped.dwell.max()
    sessions["mean_dwell"] = grouped.dwell.mean()
    sessions["start_ts"] = grouped.timestamp.min()
    sessions["end_ts"] = grouped.timestamp.max()
    sessions["total_duration"] = (sessions["end_ts"] - sessions["start_ts"]).dt.seconds / 60
    print("Computed dwell and duration")
    
    # Click rate
    sessions["total_duration_secs"] = (sessions["end_ts"] - sessions["start_ts"]).dt.seconds
    sessions["click_rate"] = sessions["total_clicks"] / sessions["total_duration_secs"]
    sessions.click_rate = sessions.click_rate.replace(np.inf, np.nan)
    sessions.click_rate = sessions.click_rate.fillna(0)
    del sessions["total_duration_secs"]
    print("Computed click rate")
    
    # Dates
    #sessions = add_datepart(sessions, "start_ts", drop=False)
    #sessions = add_datepart(sessions, "end_ts", drop=False)
    sessions = add_cyclic_datepart(sessions, "start_ts", drop=False)
    sessions = add_cyclic_datepart(sessions, "end_ts", drop=False)
    print("Computed cyclic date parts")
    
    # What is the item and cat most viewed in this session?
    # How many times were they viewed?
    sessions["cat_most_viewed_n_times"] = grouped.category.value_counts().unstack().max(axis=1)
    sessions["cat_most_viewed"] = grouped.category.value_counts().unstack().idxmax(axis=1)
    sessions["item_most_viewed_n_times"] = grouped.item.value_counts().unstack().max(axis=1)
    sessions["item_most_viewed"] = grouped.item.value_counts().unstack().idxmax(axis=1)
    print("Computed most viewed item/cat per session")

    # For the item most viewed in each session, what is its global buy/view frequency?
    freqs = get_items_cats_percent(clicks, limit=limit)
    cat_views = pd.DataFrame(freqs["views"]["cat"])
    cat_views.columns = ["cat_views_freqs"]
    sessions = sessions.merge(cat_views, how="left", left_on="cat_most_viewed", right_index=True)
    sessions.cat_views_freqs = sessions.cat_views_freqs.fillna(0)
    item_views = pd.DataFrame(freqs["views"]["item"])
    item_views.columns = ["item_views_freqs"]
    sessions = sessions.merge(item_views, how="left", left_on="item_most_viewed", right_index=True)
    sessions.item_views_freqs = sessions.item_views_freqs.fillna(0)
    item_buys = pd.DataFrame(freqs["buys"])
    item_buys.columns = ["item_buys_freqs"]
    sessions = sessions.merge(item_buys, how="left", left_on="item_most_viewed", right_index=True)
    sessions.item_buys_freqs = sessions.item_buys_freqs.fillna(0)
    print("Computed most viewed/bought freqs")
    
    # Sorting sessions
    sessions = sessions.sort_values(by=["start_ts"])
    sessions.index.name = "session"
    
    print("Sessions shape %s %s" % sessions.shape)
    print("Sessions columns %s " % sessions.columns)
    print("Sessions from %s to %s" % (sessions.start_ts.min(), sessions.start_ts.max()))
    return sessions


sessions = process_sessions(clicks_proc,limit)
sessions
def prep(limit=None):
    print("Prepping data for classification")
    buys = process_buys(limit=limit)
    clicks = read_clicks(limit=limit)
    processed_clicks = process_clicks(clicks)
    sessions = process_sessions(clicks, limit=limit)
    
    print("Merging clicks and buys")
    X = pd.merge(sessions, buys, how="left", left_index=True, right_index=True)
    
    X = X.sort_values(by=["start_ts"])
    y = X["is_buy"]
    y = y.fillna(0)
    
    X["cat_most_viewed"] = X["cat_most_viewed"].astype("float64")
    
    # Delete label
    del X["is_buy"]
    
    # Delete time columns (cant be used as is and we already have the cyclic date parts)
    del X["start_ts"]
    del X["end_ts"]
    
    return X, y

X,  y  = prep(limit)
X
y
from sklearn import linear_model

def classify(X, y):
    
    print("Splitting into train and test")
    X = X.fillna(X.median())
    X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1, test_size=0.25, shuffle=False)

    print("Building datasets for lightgbm")
    
    cats = ["cat_most_viewed", "item_most_viewed"]
    evals_results = {}
    
    print("Starting classification")
    model = linear_model.LogisticRegression()
    model.fit(X_train, y_train)
    
    y_pred = model.predict(X_test)
    y_pred = np.round_(y_pred, 0)
    print('The accuracy of prediction is:', metrics.accuracy_score(y_test, y_pred))
    print('The roc_auc_score of prediction is:', metrics.roc_auc_score(y_test, y_pred))
    print('The null acccuracy is:', max(y_test.mean(), 1 - y_test.mean()))
    
    return model

classify(X,  y)
X, y = prep(limit=100)
model = classify(X, y)
vs = model.feature_importance()
ks = X.columns
d = dict(zip(ks, vs))
sorted(d.items(), key=lambda x: x[1], reverse=True)[:15]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11470102411628676, 'fraction_numerical': 0.006276841757515692, 'mean_word_length': 3.998678996036988, 'pattern_counts': {'":': 1, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '286391', 'license': 'Apache-2.0', 'n_tokens_mistral': 4991, 'n_tokens_neox': 4836, 'n_words': 1448, 'path': 'yoochoose- Retail Prediction. 2015.ipynb', 'repo_name': 'ybshankar010/machine-learning', 'size': '15135'}